Jack Vanlightly

@gunnarmorling So basically, it's like talking to a chat bot. Mix of decent, some incredible and some truly bizarre screw ups!

Working with Claude Code is such an up and down. One moment it identifies and fixes issues with unbounded state growth in a Flink job (noice!), the next minute it does really bizarre mistakes like inventing "as" imports for Java.

@zarinjo First I've heard of it! I'll check it out

@vanlightly Have you considered using Quint quint-lang.org?

It blows my mind sometimes the nuanced edges cases that TLA+ finds, edge cases that I had never considered. The foundations of what I thought I knew about the leaderless shared log recovery protocol I am designing have been so wrong!

@ConorMcCarter I wrote this jack-vanlightly.com/blog/2023/10/1… which also has a Further Reading section with more recommended reading.

@vanlightly i'd love to understand TLA+ and learn how to implement it. any recommendations or resources for ramping up?

I’ve just published the next installment in my "Theory of Durable Execution" series. Across frameworks we hear terms like workflows, activities, virtual objects, handlers, and functions. Underneath the terminology, they’re all describing a small number of execution patterns. In this new post, I break durable functions down into three forms: 🔹 Stateless functions for one-shot deterministic logic 🔹 Sessions for long-running, interactive orchestration 🔹 Actors for persistent, message-driven state To get there, I walk through: 🔹 The behavior–state continuum 🔹 How identity, lifetime, communication, and concurrency shape execution semantics 🔹 How frameworks like Temporal, Restate, DBOS, and Resonate map onto these forms This builds on the previous two posts in the series: 🔹 Demystifying Determinism in Durable Execution 🔹 The Durable Function Tree (part 1 and 2) This was the last post in this series, distilling the various frameworks down to a simpler logical model. Full post: The Three Durable Function Forms jack-vanlightly.com/blog/2025/12/1…

@flyingcroissant Thanks! That is really useful!

@vanlightly in the playlist menu, you can pick “Exclude from my taste profile” (one by one, for every playlist 🫠) which seemed to work for me

My Spotify Wrapped is ruined this year. It thought all the 40hz gamma binaural beats I listened to while working was my actual music taste! Virtually none of the music I love this year got in 🤦‍♂️

@pranaygp I've not looked at that yet, or the new AWS Lambda durable execution stuff. But it's on my radar.

@vanlightly Neat breakdown - curious if you've seen Vercel's useworkflow.dev and have thoughts?

New blog post: The Durable Function Tree (part 1 and 2) I’ve been looking at how durable execution engines actually organise work. Underneath all the abstractions, most of them end up building some form of durable function tree: functions calling functions, each returning a durable promise, with the engine suspending and resuming work through replay and memoization. There is also a key distinction between local-context side effects (synchronous, can’t suspend) and remote-context side effects (durable functions, timers, delegation), which define where suspension points exist. Different frameworks expose the tree in different ways, and also constrain the tree's structure: Temporal and DBOS constrains the tree to two levels (workflow → activities/step), whereas others such as Restate and Resonate allow arbitrary depth. In the end, we can obtain reliable multi-step workflows without durable execution. I tackle the question of what the true value-add is of durable execution, its limits and how it should be combined with other approaches like event-driven architecture. Full post: jack-vanlightly.com/blog/2025/12/4…

Functions belonging to different services is implied in the function tree concept, functions calling other functions as remote side effects. Each remote function could literally be in any service. But the more fragmented it gets, the further away from "a workflow as one simple function" you get. I'm not sure we can class message passing as not an API either. Regarding coupling, loosely coupled doesn't mean no coupling. Schema evolution manages that coupling on the event.

When comparing durable execution trees to queues, it is important to realize that the functions can belong to different services. To me, it sounds like you imply that having services without any APIs (which is EDA) is better than well-defined API boundaries. Another misconception is that EDA systems are loosely coupled. They are not, as any event format change can break an unknown number of consumers. Events are global variables of distributed systems.

@dat_nguyen94 You found a bug in my pseudocode! A pseudobug? Yeah, I should fix that thanks 😁

@vanlightly The example code increments 'it' but doesn't use it—should that be user.bad_logins += 1 instead?

New blog post: Demystifying Determinism in Durable Execution A topic that confuses many when they start using durable execution frameworks like Temporal, Restate, DBOS, or Resonate is determinism. Why do some parts of a durable function have to be deterministic while others idempotent or tolerant of duplication? In this post, I explain a simple mental model that generalizes the framework-specific terminology (“workflows”, “activities”, “handlers”, “durable steps”) into a set of underlying mechanics relevant to determinism: 🔹 Why durable execution relies on re-executing functions 🔹 How control flow and side effects play different roles 🔹 Where and why determinism is needed vs idempotency Post: jack-vanlightly.com/blog/2025/11/2…

But I didn't answer your question. Idempotency is a tough one. Idempotency is easier when it only has to cover a single action. It's also good if a repeat of the work is done by a key passed to the work itself (like you defining a primary key and not using an auto-inc PK). Some frameworks have idempotency keys which can be useful. If you have a multi-step database transaction then it should be atomic, so acts like one step. If you do run each command in a separate txn, then wrap each command in a durable step. The nice thing about durable ex is that for a behavior that you want to be idempotent, but which consists of multiple actions, then turn each sub-action into its own durable step. Idempotency is easier then, and the results of prior sub-actions can be reused.

@vanlightly This is a fantastic explanation. I'm curious how you think about the trade-offs for side effects that are notoriously hard to make idempotent, like multi-step database transactions that can partially fail. Is that where you see frameworks like this providing the most value?

Multi-step processes where you really want to avoid partial execution is a good use case. This thing will just keep going until completion (success, rollback, compensations executed etc). You can also achieve similar results via queues or event-driven architecture, but the nice thing about durable execution is that the logic and the observability are centralized in a function. I wrote about this topic if you are interested: jack-vanlightly.com/blog/2025/6/11…

I thing that differentiates the OTree from say a KD or quadtree is that the latter trees are balanced based on the median which is constantly changing, requiring redistribution (costly for large datasets). Firstly the OTree stores data in all nodes, not only the leaves and the normalization step means that the median-based rebalancing is also not needed. So the OTree also supports large datasets.

@andrew_rogers @vanlightly When you say large data here, what approximate scale are you talking about?

New blog post: Have Your Iceberg Cubed, Not Sorted: Meet Qbeast, the OTree Spatial Index In my last post on lakehouses I compared classic B-tree indexes of the RBDMS to the pruning-based approach in modern open table formats. This time, I introduce Qbeast’s OTree: a multidimensional, adaptive spatial index for Iceberg and Delta. In this model: 🔹 Static partitioning schemes and sort orders are replaced by a multidimensional space divided into subdividable hypercubes. 🔹 The OTree shapes the table’s layout without requiring query engines to use the index directly (making it compatible with any existing engine such as Spark, Flink, Trino etc). 🔹 The result: more effective data locality, better pruning, and layouts that adapt as the data evolves. Proof that there’s plenty of room for new ideas in the open table formats and the lakehouse. jack-vanlightly.com/blog/2025/11/1…

The multidimensional spatial index of course is not new, however, this is the first case (I have seen) of it being applied to table format layout optimization. The nice thing about this approach is that it optimizes layout to make the existing search structures of Iceberg more efficient at read time. In that sense, it isn't a heavy lift at all as it is invisible to the query engine and works with the existing disaggregated analytics stacks.

Sophisticated versions of this idea have been in production since at least 2010 but literature is scant. There are caveats: Write performance is poor unless the storage engine supports fast, granular resharding. It breaks cache replacement algorithms, as expected from theory, so custom I/O and execution schedulers that do deep latency hiding for locality are required. This is a heavy lift because nothing in open source provides anything like these capabilities. Polygons and constraints can be adaptively indexed this way too, not just point-like tuples, but the theory is much less obvious. It can definitely work, we were prototyping this on supercomputers in the 2000s. However, a production quality analytical database using these techniques requires an almost completely novel stack from top to bottom. Absolutely worth doing in my opinion if you can pull off the execution, some of the gains are qualitative, particularly at scale.

New blog post! "How Would You Like Your Iceberg Sir? Stream or Batch Ordered?" Many teams today are trying to unify their streaming and batch pipelines over Apache Iceberg. But physical unification comes with trade-offs: 🔹 Flink streaming mode and other stream processors want stream-order (temporal order). 🔹 Spark and analytical engines want batch-order (value clustering for pruning). In my latest post, I explore why these layouts are fundamentally at odds and how Apache Fluss and Confluent Tableflow each navigate this tension in very different ways. jack-vanlightly.com/blog/2025/11/5…

It's that time of year again, Christmas is around the corner and my wife has knitted many more shawls! Check them out, and use the code JACKV for a 10% discount. Extra points to guess who is modeling them in the photos (spoiler, it's me 😄). etsy.com/shop/SquiggleK…

@coltmcnealy You got an API for that? 😂

Use Real Intelligence not Artificial Intelligence. It works better.