Michael McCandless

OK in case anyone else has also wondered/struggled over the years why on earth they are called "daemons" in the UNIX world... I asked Claude. I hope it is not hallucinating; I like this origin story: claude.ai/share/923351b7…

The 2025 Community Over Code NA Apache conference in Minneapolis was awesome! I recorded most of the talks from the Search track, here: youtube.com/playlist?list=… These are 360 videos! So you can slide/rotate your device to pick which direction to look.

I spent some time looking at the Vespa source code to see how it compares with Lucene jpountz.github.io/2025/07/25/mor…

This small change yielded a ~5% speedup on several queries of Lucene's nightly benchmarks (see last data point at benchmarks.mikemccandless.com/OrStopWords.ht…). Can you guess why?

Lucene is getting an increasing number of high-quality contributions from ByteDance employees, especially around performance. Good to see that this project keeps attracting contributors from all around the world.

Another common point I did not expect: Vespa's strict vs. unstrict iterators is quite similar to Lucene's two-phase iteration. And both projects use this feature to effectively combine dynamic pruning with filtering (a hard and underappreciated problem IMO).

There has been a big regression in Lucene's nightly benchmarks recently after a kernel upgrade. @mikemccand and @rcmuir found that it was caused by a change in the Linux scheduler configuration. #issuecomment-2889212582" target="_blank" rel="nofollow noopener">github.com/apache/lucene/…

I wanted to share what I learned from Tantivy's "Search Benchmark, the Game", so I set up GitHub pages and wrote two blogs, on general observations on the benchmark jpountz.github.io/2025/05/12/ana… and how it helped drive performance improvements in Lucene jpountz.github.io/2025/04/12/why…

Yelp's nrtSearch was just upgraded to Lucene 10. Also switched from persistent storage to object storage as a source of truth, and plans on doing NRT replication via object storage instead of over the network. Very similar to Elasticsearch Serverless.

The search library benchmark from the Tantivy folks was just updated with Lucene 10.2 tantivy-search.github.io/bench. Lucene now performs much better at the COUNT collection type, a bit better at TOP_K. Still somewhat slow at TOP_100_COUNT and phrase queries across all collection types.

This xkcd was published 18 years ago, mostly irrelevant for like a decade as computers got faster, and is now very much back.

Important moment for @ProjectKuiper as we just confirmed our first 27 production satellites are operating as expected in low Earth orbit. While this is the first step in a much longer journey to launch the rest of our low Earth orbit constellation, it represents an incredible amount of invention and hard work. Am really proud of the collective team.

Guo Feng contributed a 2.5x (!) speedup to #Lucene's numeric range queries by using vectorization. HZ sped up query evaluation, ID sped up decoding data from the index. Lots of great performance improvements coming in Lucene 10.2.

This is now live on nightly benchmarks, with a 32% speedup on primary-key lookups benchmarks.mikemccandless.com/PKLookup.html and a 5% speedup on fuzzy queries. The main benefit that Lucene users will notice is likely faster indexing with explicity document IDs

Lucene's histogram collector is becoming more sophisticated, it can now take advantage of points indexes when the query fully matches a segment, which can give a multiple fold performance boost. github.com/apache/lucene/…

#Apache #Lucene will soon have a faster and smaller terms index! This is a complex part of Lucene, and a major hotspot for terms heavy use cases like (primary) key/value store (~34% speedup, but results are preliminary!). Lucene's pluggable Codec API makes experimentation like this possible. #issue-2899660205" target="_blank" rel="nofollow noopener">github.com/apache/lucene/…

I'm starting a new journey! Read more about it here: "The Solopreneur Sabbatical" coryzue.com/writing/the-so…

I must call out this unsung hero in #Apache #Lucene's benchmarking toolkit: Created by @rcmuir (thank you!), it tests performance impact of any git branch (or PR) against many (10 currently?) different CPU architectures/models/revisions via AWS's diverse instances so we can catch accidental regressions in our SIMD performance. You just run a single "make" command. It's because of passionate developers and modern cloud-native tools like these that #Lucene keeps improving so quickly! And that's so hard in this modern brittle world of tapping into vectorized (SIMD) instructions in modern CPUs from all the way up the clouds of pure avaland. github.com/apache/lucene/…

The number of improvements in Lucene here are crazy. Pretty much every count and boolean query gets a nice boost and some of the count improvements are hilarious 🚀🚀🚀

The next Apache #Lucene major release (11.0) already has some compelling performance gains over 10.x, now encoding dense postings blocks as a bitset, and using vectorized (SIMD) CPU instructions for postings int[] decoding! Thanks to the pluggable Codec API in Lucene, such experimentation and improvement is (relatively) simple! #issuecomment-2589415510" target="_blank" rel="nofollow noopener">github.com/apache/lucene/…