Sinan Ozdemir

Almost every agent product has an intent-routing layer: an LLM that looks at a task and decides what should handle it. For a proof of concept, a prompt and an OpenAI API key may be enough. But once you are routing millions of production requests, cost and latency start to matter. You could send every decision to the best frontier model and expect it to work. In our test, GPT-5.5 reached 91.7% routing accuracy with 1.19-second P50 latency, costing $2.40 per 1K requests. To see how open source performs, I asked my coding agent to “vibe tune” a router using Fireworks’ Training Agent skill. It selected managed LoRA SFT and trained Qwen3-8B on 600 labeled examples. Fine-tuning improved Qwen’s routing accuracy from 80.0% to 93.3%, with 0.38-second P50 latency. On this small holdout, its quality was effectively comparable to GPT-5.5, while latency was roughly three times lower. The economics depend on scale. One dedicated H200 costs $5,040 per month, breaking even with GPT around 2.1M monthly routing requests. At 10M requests, tuned Qwen is about 4.8 times cheaper. Below break-even, GPT remains cheaper. Closed APIs are often the fastest way to get started. But when production volume makes cost, latency, and control important, custom-trained open-weight models become compelling.

You can now try GLM 5.2 in Cursor! Excited to see more useful open models, thank you to Fireworks for partnering here. Results from our evals ↓