Jack Vanlightly

I've started a new adhoc series on Apache Kafka performance. I'll release as and when I have benchmark results that are interesting to share. In issue #1, I ran a benchmark of Apache Kafka 3.7.2 against 4.3.0 with Dimster and saw much better latencies with 4.3.0. Initially I suspected a broker-side version-to-version performance change but it turned out to be caused by a client config default value change in AK 4.0 which affected producer batching. jack-vanlightly.com/blog/2026/7/7/…

@mitchellh Like basic research. You never know how important the results might be. Research into bat echo location yielded better designs for radar, as an example.

Mind boggling to me that I can make a thing faster and there's always people that ask "but why?" What kind of mentality is that? The pursuit of excellence does not need justification. Also, I find in so many cases, we can't know the impact of an improvement until we do it. For example, one I've talked about before: Ghostty's high IO throughput has enabled terminal program (emulator and TUI) fuzzing at a speed thats incomparably fast to prior solutions. This has resulted in upstream patches to resolve issues in popular projects like btop, tmux, and more. Speed enabled that anecdotally example that lifted the tides of adjacent communities that don't rely on Ghostty technology at all. I didn't predict this. Make things better because they can be better and let the results naturally play out.

@DavidSHolz @ShumingHu I think if it could answer instantly, you could stay in the flow.

@ShumingHu this is a nice insight! I wonder how it could 🤔🤔🤔🤔🤔

my friends are all feeling extremely productive and also extremely drained with the latest coding models. this makes me feel like something is wrong, and also that there might be a big opportunity. does anyone have any strategies they use to make it feel better day-to-day?

(from The Log is the Agent by Yohei Nakajima) arxiv.org/abs/2605.21997

I'm re-reading the Bobiverse books right now. This sentence really jumped out at me as I imagined a Bob-agent cloning itself via a fork to help solve a problem by divide-and-conquer. The first Bob in each system (session) is in charge.

@nikitabase I agree, Lakebase looks great, and Neon's original architecture made this tight integration with the lakehouse a natural extension. No arguments there.

The copies are everywhere of course. And we can calm it write amplification. The only ground truth is WAL. The important bit is that when you design your storage you can choose your data representation and you can make it usable for other systems. If it’s all part of your storage you can guarantee consistency and don’t leave as exercise for the user. Our design offers both low write amplification, high performance and access to older data in parquet. This is what makes it quite special

What's with the obsession with "a single copy" of the data? With LTAP, it might store one durable copy of the data (WAL keepers + Parquet on S3), but pageservers also maintain a copy. Sure, its cached, but it's still a copy. I don't want to think about the cost of serving a Postgres read from Parquet. You can size the Postgres instance keep more data locally, to avoid a local cache miss, so then you have another cached copy. Plus, "single copy" but with time travel? There are in fact multiple copies of the data to service time travel. Also, CDC? The change feed is a copy. Copies are everywhere!

Yes, it's about one **logical** dataset that can span a transactional and analytical system. One consistent logical dataset. But "one copy" does not mean one logical dataset, and a lot of marketing in this space is pushing the "one physical" copy narrative because it sounds good. In the case of OLTP -> OLAP, data is organized differently in each system. One is row-based and the other columnar. One is generally an B-tree index (sorted by primary key), the other might be clustered around columns for analytical queries (like Z-Order). If the OLAP storage serves the OLTP system for long-term storage, then the OLTP side will find the storage format and file organization does not meet its needs at all. So we're back to long-term, reliable "caching" for OLTP. And this cache will most likely be a full copy, one that must remain coherent with the source of truth. The alternative is to ask OLTP queries to wait while data pages are reconstituted from OLAP organized columnar data back into row-based data pages ordered by primary key! So one-logical-dataset is really far away from one-copy.

The lines between OLTP, OLAP, and HTAP are getting incredibly blurry. In his latest piece, @vanlightly tackles this head-on and proposes a brilliant, unified taxonomy for modern storage/workload architectures. Let's break down the 5 main categories 👇 🧵

Yesterday I published benchmarks on Hardwood (a new Java library for reading Parquet files) on my Threadripper 9980X. Today I ran the One Billion Row Challenge (1BRC) to see how it compared to the EPYC 7502P that @gunnarmorling used for the official benchmark results. I am pleased to announce that the best time was 140 ms running the thomaswue submission on all 64 cores. Does anyone want to try and beat that? If so, you've been nerdsniped 😄 jack-vanlightly.com/blog/2026/7/1/…

I took Hardwood (the new minimal dependency Parquet Java library) for a spin on my Threadripper 9980X with 64 cores / 128 threads. I ran the same benchmarks as Gunnar Morling's release blog post, plus a few more to test out how well Hardwood can utilize multiple decoder threads. Here's the results: jack-vanlightly.com/blog/2026/6/30…

Single-core Hardwood doing pretty well on the Threadripper. Threadripper does 4x the throughput of m7i.2xlarge (single-core test), and reaches 75% of all-core m7i.2xlarge. Maybe the AVX-512?

I'm running @gunnarmorling 's Hardwood benchmarks on my Threadripper 9980X, I'm curious to see how it scales across cores and also compare the Threadripper results to the m7i.2xlarge results in the 1.0 release blog. morling.dev/blog/hardwood-…

Today is the day: #Hardwood reaches 1.0 🚀! A from-scratch #Parquet library for the JVM, reading (and soon writing) Apache Parquet fast: multi-threaded at its core, with no mandatory dependencies. Java 21+, Apache 2.0, production-ready. Happy Parquet parsing!

Run all cores at 100% on my Threadripper 9980X, it purrs along. Run 1 core at 100% and it sounds like it's about to take off.

@_TylerHillery That's why I love LINQ in C#, so easy to slice and dice, group by etc. Java Streams, which came later, was no-way near as intuitive.

I learned SQL before I learned a general purpose programming language. It has corrupted my way of thinking. Every problem becomes: "How do I fit this into a table to make for an easy query?"

@rmoff Load-bearing? What's wrong with that? Great linked article btw.

never mind the em-dash. if I see this in a blog post, i know who wrote it for you. and I don't have positive thoughts about it. (see also writethatblog.substack.com/p/dev-reaction…)

The last of my Kafka share group benchmark and performance analysis posts is out. In this post I cover a pathological edge case with share.acquire.mode=record_limit, which needs more than one other condition to occur, but nonetheless could trip up some people. Enjoy! jack-vanlightly.com/blog/2026/6/24…

@almoggavra It used to be per day, I think it's a bit more complicated than that now.

@almoggavra You get 5-60 minutes of burst per day, depending on the instance, then it applies the baseline (like a sledgehammer). I use `aws ec2 describe-instance-types` to see what the baseline is, so when I run benchmarks, I size the throughput to not exceed the baseline.

somebody needs to create an interactive picker for aws instance types. like "get me the cheapest node with >25Gbps network".