Veeral Patel
4.5K posts
Which companies are building in the open about their benchmarks? - Ramp - Databricks - DoorDash
At 11k employees, our AI costs are going up. Which model & harness should we use to lower cost but also retain great quality? We didn't want to blindly trust public benchmarks. So we ran a comprehensive evaluation on our tasks, code base, infra. It's been produced by more than 3,000 software engineers, spans 3 hyperscalar clouds and many languages and tasks. The results are surprising. We find that for the SAME mdoel, the choice of harness can significantly save costs (~2x). We also find that GLM 5.2 performs extremely well. We run Omnigent in front of these and can easily multiplex different harnesses and models for different tasks. Check it out: databricks.com/blog/benchmark…
Zuck: “The pricing from some of the other labs is very extreme and has very high margins. We think that there’s a real ability to be able to offer frontier or very high-level intelligence at a much more affordable cost.” Epic pricing war breaking out among agentic models.
Every company used to start with paperwork. The next generation will start with a prompt. Ramp for Agents lets AI agents incorporate your company, apply for Ramp, and get your business ready to spend, pay bills, and manage money.
Excited to announce our partnership with @tryramp to offer day-one incorporation for startups directly in Cofounder. You can now apply for incorporation, an EIN, and bank account without leaving the roadmap!
Because of DashBench, we’re able to quickly discern which model combinations yield the best results and at the best cost. For example, with DashBench we’ve seen Kimi K2.6 + Fable 5 vastly outperform our current Sonnet 4.6 + Opus 4.8 harness at a cheaper cost. Having our own benchmark allows us to build confidence in leveraging the frontier intelligence and open-weight models without compromising on enterprise outcomes.
We can finally say AI isn't killing jobs. A new paper from me, @tryramp, and @RevelioLabs uses firm-level spend and workforce data across 21K U.S. businesses to measure AI's impact on jobs. Firms that adopt AI heavily grow headcount 10% over two years following adoption. Low adopters see no statistically significant change.
How to keep AI spend flat while token usage grows exponentially: Not with friction and spend alerts. With better defaults, routing, and caching. Better Defaults (not Usage Caps) – Engineers can choose any model they want, but defaults matter. We’re experimenting with defaulting to open weight models like GLM 5.2 and Kimi 2.7 through our LLM gateway, while still encouraging engineers to choose the right model for the task. 91% of our employees were never hitting their usage caps, so instead of lowering caps and driving up alerts, we're moving to cheaper defaults. Note that code reviews use a diversity of models, so they can check each other's work. Better Routing – In our custom harnesses, we preprocess prompts and route to the best model for the job, considering cache hits and model pricing. For instance, you may want a frontier model for planning, but not for execution where they can be overkill. Ultimately, humans shouldn't be choosing models - AI can automate this task. Better Caching – Cache misses are the easiest way to drive your cost up. All of our requests are cache aware, so we’re reusing a warm cache wherever possible. For example, our cache hit rate went from 5% → 60% in LibreChat once properly implemented. Keep Context Lean – Start fresh sessions when switching tasks. Scope file context narrowly. Disconnect unused tools. Don't just compact. The goal isn't fewer tokens used, it's fewer tokens wasted. Better Visibility – Our engineers can use as many tokens as they want, from whatever model they want, but we’ve made usage visible – and the more you spend on AI, the more impact we expect. The goal isn't to suppress usage. It's to build the infrastructure that makes exponential growth sustainable. Putting this into practice has cut our AI spend nearly in half, while our token usage continues to grow.
Today @karimatiyeh and I are both taking new titles as Co-CEOs of @tryramp. If you know us, this won't feel like a change. From when we first started building together twelve years ago, our partnership has run on a couple of motivating principles. On decision-making, we trust each other completely to make critical calls for the company across every function. And on organization design, technology is not a distinct part of the company - it is the entirety of it. That is why Karim has for years directly managed risk, operations, and marketing. Most importantly, at Ramp there is no line between the people who build and the people who do everything else. Everyone is a builder. For the last 2,656 days, we have run the company this way. This only makes it formal. We thought it was important to do it now because of how we see the AI exponential reshaping what Ramp can be. Decisions of company strategy are increasingly decisions of technology and systems design. We have always believed every function should be approached as a systems-engineering problem (even when the system was primarily human) but the rise of machine intelligence makes this existential. Every part of the company must be positioned to leverage the continued explosion in model intelligence and capabilities. If we do this well, each step-change in what models can do compounds automatically into better products and faster execution without anyone having to rebuild the company to capture it. If we fail to operate this way we will ultimately be outcompeted by a new company that does. We are also making Rahul Sengottuvelu our CTO. @rahulgs has led Applied AI at Ramp since joining us three years ago through the acquisition of his prior company, Cohere. Before that, his first company was building customer-service agents on GPT-3 at a time when almost no one knew what a large language model was, and he has spent every year since pushing the frontier of what existing models can do. He has also been right on nearly every major technical direction in AI well before it was obvious. Building Ramp now means applying AI to every part of it, and Rahul is the person stepping up to lead that work. We are still very early in the history of Ramp. Our current chapter is perhaps the most dynamic, but we have never been more optimistic on where it is going and the mission has never been more important. The businesses that trust us are navigating the same shift we are, and we intend to be there for all of it: managing their token spend, supercharging their finance teams, and helping them get more out of every dollar and hour. - Eric & Karim
