Abdel

I had a strange concurrency bug at work last week, one that made me question my whole understanding of distributed locking. a data race was happening that looked like the standard concurrency issues i'd seen before, two requests were modifying a group of rows at the same time. the standard solution is to find the rows they're both modifying, put a lock on them, and the issue is resolved. this covers a huge class of races, lost updates, dirty writes and other nasty stuff. my bug wasn't like that, the transactions were touching different rows that has share the same "group_id", and i could have tried to lock every row that is being transformed but... that's its own kind of problem, groups in this case can have hundreds or thousands of rows, holding locks on all of them creates contention everywhere and you end up with lock queues that grind throughput to nothing, postgres has a whole class of issues that come from exactly this pattern. what i actually had was something subtler, the rows were connected by a rule i didn't quite see. the book DDIA (chaper 8, v2) used an example that made the problem click for me: We have two doctors on call for the same shift at a hospital, both feel sick at the same time and both open the app to take themselves off call, the app does the responsible thing in each transaction and checks whether at least one other doctor will still be on call before letting them go. both transactions run that check at the same moment, both see that yes the other doctor is still on call, both decide it's safe to proceed, and both commit. the hospital now has zero doctors on call and not a single database rule was broken, no lock would have caught this because each transaction was modifying a different doctor's row. the race wasn't on either doctor's row, it was on this abstract rule that connects them (the invariant) that at least one doctor must remain on call at all times. This is a bit weird to reason about because the rule isn't a row in the database, it's this abstract relationship between rows. this was the gap in my mental model of locking that took me a week to figure out, we get taught to lock rows but sometimes races that exists aren't on rows. the technique described for solving this problem is called materialising conflicts and it sounds obvious once you've seen it. and you would have probably guessed it from reading the above. if the thing you need to lock doesn't exist as a row you create one, a row whose entire job is to be the lock target for that invariant. The hard part is reasoning about the problem and trying to find that invariant. continuing with the example for the doctors you'd create a row representing the on-call shift itself and lock the shift when anyone tries to go off call. the actual on-call data doesn't change but suddenly the race has a concrete row to fail safely on. the pattern to extract is: anywhere two entities share an invariant the race lives in the invariant not in either entity, and if the invariant isn't represented as something you can lock then your locks are pointing at the wrong thing entirely.

Happy Friday from Notion HQ 🖤

Today we reduced headcount by 22%. The business is the strongest it's ever been. So I think it's important to be direct about what I'm seeing and why. First, I made this decision and I own it. I did it because the way to operate at the highest level of productivity is changing, and to win the future, ClickUp needs to change with it. Second, this wasn't about cutting costs. Most savings from this change will flow directly back into the people who stay. We'll be introducing million-dollar salary bands. If you create outsized impact using AI, you'll be paid outside of traditional bands. Most importantly, I have the deepest gratitude for those affected. We're doing this from a position of strength specifically so we can take care of people properly. Everyone affected receives a package aimed at honoring their contributions and easing the transition. I only see two options: wait for this to play out gradually in the market or be honest about what I'm seeing and act proactively. THE 100X ORGANIZATION The primary change is that we're restructuring around what I call 100x org. The goal is 100x output. The roles required to build at the highest level are fundamentally different than they were a year ago. Incremental improvements to existing systems won't get us there. We need new ones. That means creating enough disruption to rebuild rather than iterate on what's already broken. The common narrative is that AI makes everyone more productive. It doesn't. Many of the workflows of today, if left unchanged, create bottlenecks in AI systems. These roles will evolve. But waiting for that to happen naturally means falling behind now. The 100x org is actually heavily dependent on people - infinitely more than today. This is only possible with 10x people that have embraced and adopted new ways of working. THE BUILDERS, AGENT MANAGERS, AND FRONT-LINERS — THE BUILDERS: 10X ENGINEERS I don't think most companies have internalized what's actually happening with AI in engineering. The common narrative is that AI makes all engineers more productive. That may be true in isolation, but at an organization level - that is the farthest thing from reality. Here's what we've validated recently at ClickUp: the great engineers, the ones who can orchestrate, architect, and review, are becoming 100x engineers. They're not writing code. They're directing agents that write code. The skill is judgment. AI makes the best engineers wildly more productive, and everyone else using AI slows these engineers down. Think about it - the bottlenecks are (1) orchestration - telling AI what to do, and (2) reviewing - what AI did. Everything is leapfrogged and no longer needed. So who do you want orchestrating and reviewing code? And how do you want your best engineers to spend their time? If your best engineers are spending time reviewing other people's code, then this is inherently an inefficient bottleneck. These engineers can review their agent's code much faster than reviewing human code. The new world is about enabling your 10x engineers to become 100x. The wrong strategy is to push every engineer to use infinite tokens. Companies doing this are celebrating 500% more pull requests. But customer outcomes don't match the volume of code being generated. I call this the great reckoning of AI coding, and every company will face this soon if not already. More code is just another bottleneck to the best engineers, and ultimately to your company's impact as well. — THE BUILDERS: 10X PRODUCT MANAGERS Product management and design roles are merging. Designers that have customer focus, become more like product managers. And product managers that have intuition for UX become more like designers. The bottleneck of user research is gone. It takes us just one mention of an agent to kickoff research and analyze results. The bottleneck of product <> design iteration is also gone. The product builder iterates on their own, along with agents and skills that ensure alignment with quality and strategy. Also controversial today - I believe that the wrong strategy is to have your PMs shipping code - that just introduces another bottleneck that the best engineers will waste their time on. To be clear, PMs should be coding but they should do this in a playground to iterate, validate, and scope. That code should not go to production. Everything outside of managing systems, orchestrating AI, and reviewing output becomes a bottleneck. That's why the other roles that are critical along with these are the systems managers (to reduce bottlenecks) along with a bottleneck you can't replace - customer meeting time. — THE SYSTEM MANAGERS Ironically, the people that automate their jobs with AI will always have a job. They become owners of the AI systems - agent managers. We have many examples of these people at ClickUp. The underlying systems in which we operate are absolutely critical to get right. I think most companies are delusional to think they can iterate on existing systems and compete in this new world. You must create enough disruption so that old systems are deprecated entirely. If there's any definition for 'AI native' that's what it is. — THE FRONT-LINERS In a world that will become saturated with AI communication, the human touch will matter more than anything to customers. This is a bottleneck that you shouldn't replace - even when agents are high enough quality to do video meetings. One-on-one meeting time with customers is something that shouldn't be automated. The systems around the meetings should be - so that front-liners spend nearly 100% of their time with customers. REWARDING 100X IMPACT In a world where companies are able to do so much more with less, where does that excess money go? In our case, much of the savings in this new operating model will flow directly back to those that enabled it. We must reward people that create productivity accordingly. This aligns incentives on both sides. Plus, in a world where your best people create 100x impact, you can't afford to lose them. You should aim to retain these employees for decades. The context they have and their ability to efficiently orchestrate and review will be nearly impossible to replace. Compensation bands of today should be thrown out the door. We're introducing $1 million cash/year salary bands with a path available to nearly everyone in the company if they produce 100x impact by creating or managing AI systems. THE FUTURE Nearly every company will make changes like these. The ones that do it proactively will define what comes next. The future is not fewer people. It's different work, new roles, and better rewards for those who embrace it. We're already seeing entirely new roles emerge, like Agent Managers, that didn't exist a year ago. ClickUp is positioning to lead this shift, not just internally, but for our customers too. I've never been more certain about where we're headed.