kwindla

I’ve been building with Claude Cowork lately, and I figured I’d share my thoughts so far. I chose Cowork over Claude Code to see what it’s like to write code but never read it. Some early thoughts: 1. I’m using natural language, instead of code, but it’s still software development. I don’t need to worry about syntax, but the concepts underlying “traditional” software still matter: observability, client-server splits, pipelines, databases, logging, etc. Non-developers still need to think like engineers to be successful vibe coders – the syntax is gone, but the judgment isn’t (yet?) 2. The primary difference between vibe coding and “traditional” software is non-determinism. Traditional software teaches deterministic thinking – you can run the same function with the same inputs as many times as you want, and the output will always be the same. (When I was learning to code, I found this one of the stranger things; most other experiences in the world aren’t so deterministic!) Agent-driven software is not deterministic – ask an LLM the same question several times, and you will get several answers. The models’ laziness and tendency to cut corners – especially when inference is scarce – adds another twist. Understanding where to demand determinism from your software agents requires judgment. 3. I borrowed the concept of zero trust from security – the principle that an actor or system is only trusted after verification and never by default – to figure out when to demand determinism. I presumed model outputs unreliable until verified with human-in-the-loop checkpoints, retry logic with comparison (LLM as judge), or (deterministic) verification scripts. I realized observability, logging, and verification have to be first-class features of a vibe-coded system if it is to be reliable. Memory doesn’t cut it. 4. I did a lot of debugging through the models’ reasoning traces, which wasn't ideal. I imagine these interactions will improve a lot. If I were building an IDE for agentic coding, I’d start here. 5. What counts as a database? Cowork uses the local file system as the database, which is lazy and convenient and makes for fast prototyping, and comes with tradeoffs: the file system doesn’t enforce data schemas, and laziness means corners get cut. The input and output checks I built at each pipeline stage were repetitive but functional. If I kept pushing, I’d end up rebuilding versions of data integrity, normalization, indexing, query optimization, etc. over the local file system. 6. The tight coupling of the local machine as the client, server, database, etc. works when developing for yourself. To share work with others, you need a better client <> server split. When I asked Cowork to port my system to the cloud, it immediately suggested moving everything to Google Drive – swapping my local file system for a cloud file system. I had to coax it toward the architecture of a simple web app. 7. Cowork doesn’t use or expect version control, which makes tracking changes and multi-agent work near impossible. Developing without version control makes clear why we invented it. Version control is alive and well in the age of software agents. It feels like there’s a new iteration of systems design to be uncovered when building with agents. It’s not wholly new, and most of the primitives and principles of the past are still useful, but they need to be reassembled when some steps are deterministic and others aren’t. I’m excited to see what we uncover.

Gemini 3.5 Flash is out today. Here are numbers from my main voice and task agent benchmarks. Some notes: All the Gemini 3 models so far are too slow to work well for voice agents. Gemini 2.5 Flash was a *great* model for voice agents, when it was SOTA. It was fast and good at instruction following. Its big weakness was tool calling. It was quite difficult to prompt Gemini 2.5 Flash to perform tool calling reliably in long context, multi-turn use cases. With Gemini 3, Google improved the tool calling issues a lot. But time to first token is ~1s. We really need TTFT down below 700ms. Google isn't alone in this. All the SOTA models released this year have been reasoning models that aren't optimized for low latency. Claude Haiku 4.5 (released last October) remains the best-performing model with a TTFT under 700ms. Gemini 3.5 Flash is the first Flash model in the 3 family to be released as "generally available." It's quite different from gemini-3-flash-preview, which was released last December. That model actually scored a bit better on my voice agent benchmark. This new model is the new overall top scorer on my task agent benchmark. This benchmark tests a multi-turn task, requiring that models achieve a P50 turn execution time faster than four seconds. Gemini 3.5 Flash with a "high" thinking budget scores significantly better than any other model I've tested. So even though the TTFT isn't what we'd like to see from this model, the overall generation speed makes up for it, and allows us to use the "high" thinking budget and still achieve a per-turn P50 under two seconds. Very impressive. This performance costs money, though. I had become accustomed to thinking of Gemini models as aggressively priced. But Gemini 3.5 Flash is actually more expensive than GPT-5.4 and Claude Sonnet 4.6 on this benchmark. Also note that lower reasoning settings don't always save money. Gemini 3.5 Flash "minimal" costs more, on this benchmark, than "high," because it makes more mistakes, so it uses more tokens to complete the task. Please note that performance of this model on your benchmarks might be very different. My voice agent and task agent results are often wildly out of line with the reported results on standard benchmarks in the model cards and release notes. The voice agent benchmark is 30 turns, and heavily tests tool calling in a long-context scenario. The task agent benchmark injects large streams of structured data events into the context, all tool calls are asynchronous, and the test task takes at least 32 turns to complete. (My motto for evals is "30 turns or it didn't happen.") Make your own benchmarks! (And post the source code and the results for different models, if you can.)

Apologies for the accuracy on this one