Kelly Sommers

You asked for longer rants, so here are longer rants! After almost 10 years I'm back to blogging. Thanks for the encouragement. Link below in the thread 🔻

Daniel Lemire, "How many branches can your CPU predict?," in Daniel Lemire's blog, March 18, 2026, lemire.me/blog/2026/03/1….

@aaronbronsteter Huge bummer @PFLMMA

Some fun fights on the PFL card tomorrow. Unfortunately for Canadians, it will (once again) not be available to watch in our country.

March Madness day 1 is one of my favourite days of the year. I have every game on a 75” TV above my work monitor in a grid. Can’t wait to see what school surprises this year.

This is why you avoid direct connections to Postgres. Benchmarked PG running on a r8g.2xlarge (8 vCPU + 64GB ram) with connections ranging from 8 → 2048. Clearly a sweet spot at 64 with degrading perf thereafter. Apps often need 1000s of connections. Scale with a proxy!

I wrote a Hybrid Logical Clock visualizer app using claude code. (Ok, claude did the work, I just PMed claude.) You can create a send event by drag and drop, a local event by double click, and take a snapshot at T-1, by pressing snapshot button. Pretty neat.

What if you could write DataFrame logic once and run it on any SQL database? Many data workflows begin with pandas for quick experimentation, while production pipelines might run on databases like PostgreSQL or BigQuery. Moving from prototype to production usually means rewriting the same transformation logic in SQL. That translation takes time and can easily introduce errors. Ibis solves this by letting you define transformations once in Python and compiling them into native SQL for 25+ backends automatically. --- 🚀 Tools for Portable DataFrames in Python: bit.ly/4cPYEUD #Python #DataScience #SQL #DataEngineer

@KhuyenTran16 I’ve been thinking a lot about this lately. I’ve also been curious about database local execution of DataFrame code. Like stored procs.

On the flip side, it’s really interesting how far some systems can go while limited to a single global writer node.

One thing I think about a lot about is software scalability and reliability. Some organizations build hyper specific and optimized building blocks. But some orgs benefit tremendously from a simpler unit they can scale as demand grows even if it’s not the fastest.

When a girl says, "5 mins", think about it like there are five minutes left in the 4th quarter and both teams have all their timeouts.

@eonem Thank you for chiming in. I love this app and how people building bigger systems than I have built (my biggest is 2,000 nodes) can chime in and educate us all.

@kellabyte Control planes often rely on intentional friction, so that over-admission can’t create instability while dealing with the resulting control plane event processing.

It’s a bit ironic to me that after a couple decades the chosen cloud control plane adopted industry wide (Kubernetes) depends on a data store technology that cannot scale writes beyond a single node.

@eonem Yeah agreed but I also feel K8 has many of the building blocks in place and is close to enabling people to carve out consistency boundaries between namespaces and workloads. It’s not necessary to treat every piece of metadata as system wide state.

@ykaas This is great information thank you!

@kellabyte For large-scale clusters (up to 65,000 nodes), GKE replaced etcd with Google Cloud Spanner as the backend state store, while still exposing the etcd API for compatibility. Spanner gives horizontal scalability, global distribution, and low-latency consistency without etcd’s limits

@teardropinocean Yes but that’s too simplistic of a view. As pods grow, as worker nodes grow, cluster metadata grows. Everything in scaling consistency is about consistency boundaries. Treating an entire cluster managing many workers and workloads as 1 consistent state is super limiting.

@kellabyte Maybe it is still like this because CAP theorem is a thing? There always has to be some write to a disk somewhere, no?

@berenddeboer I’ve seen some organizations using Kubernetes there as well. What do you see people using? Just curious!

@kellabyte In the AWS world people are definitely not using Kubernetes :-)

@kellabyte @gregyoung This is true. Like on AWS you have choice between Elastic Block Store (EBS) volumes and Elastic File System (EFS) volumes when creating your PVCs, each with their own trade offs.

@gregyoung Also I _think_ PVCs are implemented by the cloud provider underneath. Not sure if I’m right about that but that may mean reliability and guarantees could slightly differ from one vendor to another.

@kellabyte Wasn't this at least part of why PVCs were introduced?