Julien Le Dem

If you follow me on here, you should follow me on there: bsky.app/profile/julien…

Thank you for letting me keynote day 2 of the Iceberg summit. I hope you all enjoyed it! bsky.app/profile/julien…

Hey it’s @J_ @ Iceberg Summit. Great keynote!!

@J_ this morning at #icebergsummit.

The database landscape is going through its biggest shift in a decade. AI workloads are pushing OLTP and OLAP closer together, object storage is becoming the de facto foundation for AI search at hundred‑billion‑ to trillion‑document scale, and agents are spinning up databases and schemas programmatically in patterns classic systems were never optimized for. AI Council's Data Engineering & Databases track is where the builders re-architecting the stack come together. Curated by @saisrirampur, Director of Product at @ClickHouseDB, here's the lineup: → Hannes Mühleisen, Co-Founder & CEO at @duckdb: "Super-Secret Next Big Thing for DuckDB" → @nikhilbenesch, CTO at @turbopuffer: "Fast AI Search on Object Storage @ >1 Trillion Scale" → @J_ & Pierre Lacave, Principal Engineer & Staff Engineer at @datadoghq: "The Deconstructed Database at Datadog" → @iskakaushik, Engineering Lead at ClickHouseDB: "Building a Unified OLTP + OLAP Database for AI Workloads" → Bhargavi Reddy Dokuru, Staff Data Engineer at @netflix: "Democratizing Analytics via Self Service: Netflix Games Edition" → @kelvich, Principal Software Engineer at @databricks & Neon co-founder: "AI Needs a New Kind of OLTP: Lakebase & Serverless Postgres in the Agent Era" → Robin Tang, Co-founder & CTO at @artie_labs: “Scaling CDC to Trillions of Rows: What Broke, What We Rebuilt, and What AI Demands Next” A huge thank you to Sai for curating this awesome track! Join us SF, May 12-14! 🎟️ aicouncil.com

@J_ co-created Apache Parquet, Apache Arrow, and OpenLineage. Three projects. Three industry standards. Parquet at Twitter in 2013. Arrow at Dremio. OpenLineage at Datakin, acquired as part of Astronomer's $213M Series C. He is now Principal Engineer at Datadog and an officer of the Apache Software Foundation. That is an unusual track record of picking the right abstraction at the right time. His OpenXData talk argues that the current wave of challengers -- Lance, Vortex, Nimble, FastLanes, BtrBlocks, F3 -- are solving real problems but misreading what made Parquet succeed in the first place. The core contribution was not the encoding choices. It was the community consensus mechanism those choices were built inside. His case: use established open source communities to absorb these innovations rather than fragment the ecosystem across six competing formats. He published the written version of this argument at sympathetic.ink in December 2025. OpenXData is where you can push back live. 👉 Register here: openxdata.ai

If you follow me on here, you should follow me on there: bsky.app/profile/julien…

@changhiskhan My point is the upper layers can be built on top of the existing format. You don’t need to start a new stack from scratch. Not everything needs to be in parquet an you don’t need a new format to build them.

@J_ Great article and really thoughtful! Encodings is almost the least interesting part of Lance tho. Requirements for AI data and workloads spans multiple layers. Even IF parquet moves forward on encodings there’s still so much that’s missing for AI engineers and researchers *now*.

In the past few years, we’ve seen a cambrian explosion of new columnar formats, challenging the hegemony of Parquet. Presumably, the design of yore is not going to cut it moving forward. I spent some time to understand how things actually changed. sympathetic.ink/2025/12/11/Col…

I also gave a talk on this topic. Slides: docs.google.com/presentation/d… Recording here: youtube.com/watch?v=S_aoc9…

If you missed my talk "The advent of the open data lake" at AI By the Bay, the recording is now available. ai.bythebay.io/talks/the-adve… youtube.com/watch?v=xHGVCV…

There is some crazy (good) activity on the @ApacheParquet mailing list for new encodings. A sample: PFOR, FSST, ALP, Strings and Cascaded Encodings. 🤯 Huge kudos to Arnav Balyan, Prateek Gaur, and Micah Kornfield for driving this. @parquet.apache.org" target="_blank" rel="nofollow noopener">lists.apache.org/list.html?dev@…

If you missed me giving it live, the slides of my talk, "Column Storage for the AI era" are available here: docs.google.com/presentation/d…

I’ll be speaking in Mountain View on Thursday. Come say hi!

Higher latency but higher throughput on improving the overall data ecosystem.

Cool parquet metadata visualizer by @J_, powered by Hyparquet

If you've been wondering why we see a flurry of new columnar formats, come see me present "Column Storage for the AI Era". I'll talk about what has changed, new advances in data encoding and how that's pushing Parquet to evolve. Event tomorrow: luma.com/pxikwty3

@julianhyde Ironically it is simultaneously terrible and the thing everyone compares themselves against :)

@J_ I hope you get a dime every time someone says Parquet is terrible.

New file format just dropped !

I'm trying to understand a bit better real life deployments of open source Clickhouse. If you're using it, what does your deployment look like?

The FastLanes format paper from @afroozeh3 and @peterabcz contains interesting and practical ideas for representing SIMD friendly cascaded encodings. I think almost all of the ideas could be applied to extend @ApacheParquet as a new encoding scheme. github.com/cwida/FastLane…