Marek Siarkowicz

@Alfieri_Craig @GoogleOSS @AntithesisHQ I'd be thrilled to collaborate on putting etcd to the test. Let's connect and discuss how we can make this happen!

@danielepolencic I would appreciate proper research. While it is not always easy to get accurate information, open source maintainers don't bite. At least not all of them. Each community has its own chat and public meetings, where you can ask questions. They would appreciate feedback too.

@serathius Marek I replied to you on Reddit, I’m more than welcome to amend my articles and writing. My intent is to help folks. But I’d appreciate a more compassionate tone from you

etcd is the database behind every Kubernetes cluster. You can run one for years without thinking about it. Then your cluster gets big enough, and etcd is all you think about. A thread on why. 🧵

@danielepolencic As a person that worked on 30k node GKE clusters based on etcd, I don't know what this refers to. If you are interested in scalability topics by people working on the topic watch m.youtube.com/watch?v=SdLLOc… We are also working on 15GB resource size on etcd github.com/kubernetes/kub…

@danielepolencic Again, you didn't understand the motivation. It's not project A vs project B. It's the fact that AWS journal and Spanner use Atomic clocks for concensus. That allows for clean horizontal scaling of distributed system, don't expect something like this in open source.

K3s ships Kine, a shim that speaks the etcd API but stores data in SQLite or PostgreSQL. AWS replaced Raft with a journal service. Google swapped in Spanner. All kept the etcd API.

@danielepolencic Have you ever run etcd in production? Or are you collecting random complaints from the internet from people trying to use etcd with NFS? Etcd is not perfect, but the fact that some people can struggle with 3 node K8s, while others run 30k without a problem is interesting.

What breaks: Quota alarms: database fills up, control plane freezes Compaction lag: revisions grow faster than cleanup Snapshot pressure: multi-GiB transfers choke the leader Slow watches: event fan-out saturates CPU and network

@danielepolencic What have watch events to the leader? Totally unrelated. Watch events are distributed totally outside raft concensus. Also API server demuxes the watch request so that there is just one watch per resource. So number of controllers has no impact on etcd. github.com/etcd-io/etcd/i…

Controllers don't poll. They open long-lived watch connections and etcd streams events on every change. At scale the leader spends more time distributing events than processing writes.

@danielepolencic Here we go again... 8GB is an old limit that was never revisited or updated. Most cloud providers run etcd with 20GB-30GB without much problem. Usually it's Kubernetes that breaks when you put a lot of data in a single resource or namespace. Read github.com/kubernetes/kub…

The database is a single file on disk, capped at 8 GiB. Every mutation adds a new revision and etcd never overwrites the old one. 10,000 pods updated per minute = 10,000 revisions piling up.

@xu_paco Repo you linked was used as PoC, the etcd antithesis testing lives in github.com/etcd-io/etcd/t…

KubeCon Keynote Day 2: etcd + Antithesis github.com/antithesishq/e…

Learn how the etcd open source project overcame a significant knowledge drain and reliability issues by implementing deterministic simulation testing. By @ha_joslyn, thanks to @AntithesisHQ, feat. @serathius thenewstack.io/how-etcd-solve…

@indygupta Pretty sure this fact was well established over 7 years ago in multiple Reddit posts. Fact that you can doesn't automatically means that you should set limits for all your containers. Maybe we should give the discussion a rest and just put it in official documentation? @thockin

Stop using CPU Limits! (Yes, it's ok, you can say it aloud now...we have data to back it up...) SoCC 2025 talk by my student Chirag Shetty, with Saurabh Jha and other IBMers. Paper: arxiv.org/abs/2510.10747

I haven't been to an office in 2 years. I work in my pajamas making multiple six figures. Thank you, Kubernetes.

The future is really collaborative and community driven

Today our CEO, Will Wilson, announced a partnership with @CloudNativeFdn to provide free reliability testing for Graduated and Incubating projects. Stop by booth 457 to learn more. @etcdio maintainer @serathius is presenting at 12 today in the Project Pavilion about the work we did together to help advance etcd testing!

How do you make a foundational project like etcd truly robust? 🤔 @ha_joslyn sits down with @serathius (@Google) at #KubeCon to discuss how the etcd maintainer community used the challenge of DST to drive better, more rigorous testing.

@cra @CloudNativeFdn @jbryce @Google @janet_kuo @TerryTangYuan @ritazzhang @mfahlandt @reylejano .@serathius and Will Wilson of Antithesis talk about their partnership on running verification and supporting etcd!

Świętujemy 10-lecie @CloudNativeFdn 🎉 Spotkajmy się na jubileuszowej edycji Cloud Native Warsaw meetup #cTENcf, gdzie poza🎂czekają na Was porywające sesje, w tym powrót do korzeni Kubernetes, panele dyskusyjne, networking i więcej. Szczegóły: goo.gle/3LsBaJo

Autonomous Testing of etcd’s Robustness via @CloudNativeFdn, by @serathius cncf.io/blog/2025/09/2…

Autonomous Testing of etcd's Robustness - etcd.io/blog/2025/auto… #Kubernetes

We're incredibly proud that the @CloudNativeFdn relies on Antithesis to ensure that the cloud is built on solid foundations. @serathius and the maintainers of @etcdio wrote about their use of Antithesis on the CNCF blog today, and we look forward to working with CNCF to secure their other open source projects. Code is infrastructure. Link below.

Kubernetes v1.34: Snapshottable API server cache - kubernetes.io/blog/2025/09/0… #Kubernetes