Vinoth Chandar

#apachespark can be great and painful at the same time. Shuffle partition tuning still remains a key lever for spark performance.

Walmart ran Apache Hudi across 600,000+ Spark cores. They picked it via weighted scorecard against Iceberg and Delta. Two real workloads tested ↓

The Hudi file group is the primitive that makes Lance fragments, RFC-80 column families, and wide AI tables all click into place. One record key → one file group. Indexed, versioned, concurrency-controlled. Two AI workloads land on it ↓

Claude Fable (plus some custom skills) just solved the biggest time sink of my oss maintainer life. complex git rebases. E.g this 1600 file mega @apachehudi refactor github.com/apache/hudi/pu… Genuinely, one of the happier days of my life.

I think AI is going to rapidly erode premium charges for "serverless" solutions - for sake of better devx and tighter integrations - ground users to the basic economics that running compute yourself is obviously cheaper.

The fact that surprised me the most in observing #dataengineering for 10 years. Small files are still a big problem

It still is. Companies use it outside of onehouse, azure also. Uniform is what databricks still uses to support Iceberg on their platform. But market narrative has been: everything is converging to Iceberg, as common denominator. This article is talking about the core open-source lakehouse storage tech, which is evolving, faster than common denominator. Eg hudi has indexes, rest don’t at this point.

@byte_array Um yeah but what about @apachextable ? Wasn’t that supposed to be the grand unifying layer for all this?

Rare: a table-formats article that doesn't tell you to rip-and-replace everything for Iceberg. If you work near a lakehouse, it's worth a read. Key points, regardless of what you run: 💸 The format itself costs $0. Iceberg, Delta, and Hudi are Apache-2.0. You pay for compute, storage, and the platform—so the real question is engine economics for your workload.

Article: tech-insider.org/apache-iceberg…

So the sane default isn't "pick the winner." It's: match the format to your dominant workload, translate metadata instead of rewriting data, and keep your options open — because these formats are interoperable in a way warehouses never were or even are. (Disclosure: I started Hudi.)

⚡ For heavy writes (upserts/CDC/high-frequency ingestion), Hudi remains the purpose-built option: indexing, merge-on-read, non-blocking concurrency. The calming part: this isn't the warehouse era where you need to make a hard choice between "open" and "closed". Tools like Apache XTable (incubating) and UniForm make "format" mostly a metadata choice, not a data migration. Write in what fits your write path, expose others for reads—no copy, no rewrite.

🧬 They came from different problems, and that still shows: Iceberg (Netflix append analytics), Delta (Spark/Databricks), Hudi (Uber streaming + frequent updates). 🔄 In Databricks and Fabric, Delta is effectively an optimized, Iceberg-compatible path. With UniForm, it can present an Iceberg interface, and is often cheaper/easier than "rolling your own Iceberg." "Move everything to Iceberg" is often a fake mandate.

JD.com built a 125+ petabyte retail data lake on Apache Hudi. Record-level mutations on commerce data at planetary scale ↓

Why inline pays off: • One read fetches metadata + bytes • 10K thumbnails = 10K rows, not 10K S3 objects • File sizing service consolidates into right-sized base files • read_blob(content) to materialize bytes

For 4 years, I left every data conference with the same thought: “We need to build that too.” A new engine. A new serving layer. A new database primitive. Another “must-have” platform capability. Last week at Data + AI Summit was different. For the first time, I didn’t come away with a missing category. When we started Onehouse in 2022, many justifiably said a startup couldn’t build a complete, open data platform. Today, it exists. Still a lot to do. But I’m incredibly proud of what this team has built. 🚀

GE Aviation: 30+ source systems in production, several hundred Apache Hudi tables, 14+ months in production, 10,000+ tables in the dev pipeline. Modernizing 150+ source systems across the aviation ecosystem ↓

If you read receipts, a new pricing page with a calculator to break this all down: quanton.dev/pricing/

My biggest learning yesterday at the Data + AI Summit: Most users said they "turn off Photon by default," or "run careful benchmarks" to decide if the ~2.9x Photon tax is worth it. Kind of wild that it's still the way things are. standard clusters (slow, tax-free) (OR) photon clusters (fast, taxed). This is no way to be running Apache Spark in 2026. We're still chatting at booth 403 — I'm here all day, entertaining questions and also healthy debates. Swing by! #ApacheSpark #Performance #DataAISummit #Databricks #Photon #Quanton

Our Data + AI Summit booth number is 403. Yes, the HTTP error code for "Forbidden". 🏴‍☠️🦜 But you've seen the movies. The treasure is always in the forbidden place. The sealed cave. The remote island. The door someone told you to stay away from. That's where they keep the good stuff. Booth 403 is that door. Behind it: Quanton, our new Spark engine — ~3x the Spark for the same spend. If you're paying a fortune and still watching progress bars crawl, this is the chest you've been digging for. Come plunder. See what else is out there. And throw your name in for some Nintendo Switch 2s while you're at it. 🎮 X marks the spot. Booth 403. #DataAISummit #apachespark #databricks

Blog : onehouse.ai/blog/databrick…

24 months in, one thing is obvious: this was never just a fight over open table formats. It’s about who controls the metadata, the catalog, and the center of gravity for where your data gets computed. By summer 2024, three communities had spent 6–8 years building the open lakehouse. We were finally on the doorstep of real interoperability. With Apache XTable (incubating), major vendors—Google, Microsoft, and others—lined up to build open bridges across formats. Maybe for the first time ever. Except one vendor didn’t. Databricks chose to buy and merge 2 of the 3 projects—framing it as “unification,” but effectively bending Iceberg toward Delta Lake to regain control. It’s disappointing how far we still are from the promised land, even after 24 months. Through it all, we’ve stayed open, collaborative, and patient. Onehouse supports all formats equally, as we set out to do from day one. Apache Hudi supports Iceberg as a pluggable table format. The future isn’t just open formats. It’s an open lakehouse stack. We’re building it in Apache Hudi—and doubling down on interoperability through the Apache XTable project.