Sunny Bains @TiDB

Agentic Scale for databases is for real. I think the architecture where you spin up separate instances because a single instance can’t handle millions of tables has a short shelf life.

@redixhumayun “not every” and not “not sure every”

@redixhumayun Yes, not sure very WAL write in InnoDB has to be synced. It’s very different from the Postgres implementation.

I'm surprised to learn that both Postgres and InnoDB use buffered I/O for WAL paths. Is this a common design decision? Postgres This isn't all that surprising after I learnt that Postgres still does buffered I/O for all paths. Direct I/O looks like it's still behind a debug flag. InnoDB This was the big surprise because InnoDB enables direct I/O. I found an old blog post from Mark Callaghan talking about the redo log using buffered I/O and the reason for that being allowing the OS to coalesce multiple small writes. Confusingly, InnoDB docs talk about innodb_flush_method which sets the value of whether to use buffered or direct IO for data and log files. But, digging through the code, I can see that this isn't really enabled for OS log files which is odd.

Don’t fetishize 100% correctness. Especially if you are just getting started.

Both, I get the model to write most tests. My general mode is: Review auto or manual, I get the LLM to first reproduce the problem. Only after it is reproduced, sometimes it requires manual tweaking, do I guide the LLM to write the fix. It’s an iterative process, just faster. The LLM has a tendency to solve issues using shortcuts. I monitor that part very carefully. Guide it to make architectural changes that are generic and extensible instead of ad hoc changes to make things work.

@sunbains Do you ever let the model write it's own tests? Or do you do that yourself?

I’ve been through this experiment with Codex far too many times I’ll work on a spec doc and keep trying to find new issues with codex so that they can be addressed in the doc and implementation can start? The problem? Codex is never done finding issues. Literally 10 messages of me asking “okay, we addressed that last set. Are there any more issues” and codex will always find something.

AI agents are scaling fast—but most databases can’t keep up. At #AGENTICON Bengaluru, learn why monolithic DBs break under agentic workloads + how Manus AI scaled with TiDB after migrating from Aurora MySQL in just 2 weeks. Session: ow.ly/7VUK50YwrH6 #AgenticAI #TiDB #DistributedSQL

@redixhumayun The waterfall model never works. Make it work, figure out what’s wrong and what’s working, write the tests and then improve the code, design and architecture, like a feedback loop.

@sunbains Yeah, it seems like I'm ironically not holding AI's hand enough at this point. I figured that more design work up front means I can let AI spin for more hours but perhaps we're not quite there yet.

@aximox_cc @ahmetb Scalability and efficiency

@sunbains @ahmetb Problem of storage cost?

kinda crazy how VictoriaMetrics/VictoriaLogs came out of nowhere and got popular. I have no idea where they rank as an APM. what are primary reasons people are choosing this solution? is there a better scalability story?

@acodechef Not yet.

@sunbains Would be fun to read the code. Is it open source?

This is getting better by the day 🙂. Made the insert … select parallel. The server now runs the generated query graph on a purpose built VM. This improved the ops/s significantly 🙂 I was using Tokio initially for scheduling but its context switching overhead stymied the performance. Ended up implementing a custom scheduler for the VM, one that is more focused on the CPU side of things and the query graph execution. This also allowed me to add plugins for controlling and allocating resources at a very fine level.

@sloppyquorum The context switch overhead was very high ~39%. This is a tighter loop and the vm threads do the context switching based on the scheduling and the query graph shape and resource settings. It’s not general purpose.

@sunbains What was the change in scheduler ?

@acodechef Yes and no (mostly yes). The no part is that it has pluggable policies that track cpu, net, mem usage and IO. In these tests it’s a single DML so it ends up as an event loop because the detail is set as burstable.

@sunbains Interesting stuff. Is your VM some kind of event loop?

Switching context frequently kills mental and software performance.

Your SSD is a cache. S3 is the real database. That sounds wrong until you see the math: • S3: $0.023/GB/month • EBS gp3: $0.08/GB/month • 3x replication on EBS: $0.24/GB/month At 10TB, that's $230 vs $2,400/month. The trick is making S3 feel like local disk. That's what disaggregated storage engines do — hot path stays fast, cold path stays cheap. The future of databases isn't faster disks. It's smarter caching on infinite storage. The next challenging question is how to build a latency-sensitive OLTP database based on S3. #CloudNative #S3 #ObjectStorage #DatabaseEngineering #DistributedSystems #TiDB #DataInfra

@muratdemirbas Cool stuff!

[new blog post] Claude Code experiment: Visualizing Hybrid Logical Clocks muratbuffalo.blogspot.com/2026/03/claude…

@ahmetb TiDB had to change to VictoriaMetrics for a big deployment because VictoriaMetrics solved the problem.

@sunbains i wanna hear real-world stories of how many billion time series, how many storage nodes setup etc. but I'm not sure if any uber-scale companies took it up yet. just saw somewhere openai uses it.

I wrote a Hybrid Logical Clock visualizer app using claude code. (Ok, claude did the work, I just PMed claude.) You can create a send event by drag and drop, a local event by double click, and take a snapshot at T-1, by pressing snapshot button. Pretty neat.