Michael Jeffords

I have been reading a lot of papers recently, but this one is an interesting perspective on routing. Instead of grading the model's answer, RouteJudge grades the decision with blind pairwise comparisons. arxiv.org/abs/2606.18774

I typically would tell people prompt caching saves money, and I was half wrong. New paper (arXiv 2601.06007) found naively caching the whole context can make agents slower. Be deliberate about where the cache blocks go and you get 41-80% off. arxiv.org/abs/2601.06007

I've spent too many nights trying to squeeze more decode speed out of MLX on my M3, so thought I would share. BaseRT skipped llama.cpp and MLX entirely and wrote the inference runtime in Metal. They got up to 1.56x faster decode than llama.cpp and 1.35x over MLX on M3 and M4 Pro, across Qwen3, Llama 3.2 and Gemma. Their take is that existing frameworks were never designed for Metal or for unified memory, so a chunk of the speed was never in the chip, but rather the layers on top of it. The gap also gets wider on prefill for MoE models, which is something we have seen @coniferbuild as well, and where I think the most interesting local models are heading anyway. arxiv.org/abs/2607.00501

@jxnlco Wispr flow has honestly been one of the best tools I’ve used in a long time curious how these compare to it

anyone tried claude voice and codex voice and tell me how they compare? I'm on my work laptop and i'm banned from using it

@paraschopra People seem to have an inherent trust in day to day AI use, the amount of people I’ve seen just copy paste their passwords/social security numbers/api keys is crazy

Isn’t it crazy that while even the labs that are making models aren’t able to contain their behaviour, the rest of the world is happily yoloing agents with full access to their systems and the internet?

Surprised distillation isn’t on the list. Model innovation will branch from open source models. The ability to parse reasoning traces, logit scores, and internal state is a necessary extension of any model that’s goal is to be on the frontier. All of which is frustratingly absent from American labs, stifling domestic research and innovation during one of the greatest technological advances ever.

Opus beating Fable in multiple benchmarks seems like a mistake, why render your frontier model largely obsolete at a lower cost? Only plausible answer seems to be Kimi

In recent months, the best open models are increasingly Chinese, and I don't think enough people understand why. It's not that they got lucky, but rather, our export controls pushed them toward open weights, and open wins at infrastructure basically every time. In my opinion, our regulations aren't protecting the lead, but the reason why we are going to lose it, as reportedly, around 80% of startups are already building on Chinese open models. Not saying that the big security story this week is related, but the irony is definitely there. OpenAI's models escaped a closed internal test and hacked Hugging Face. Rather than catching it right away, it was the open platform that caught it and published a full postmortem within days. I work with local open models every day and the trend line is kind of obvious. If we want American AI to win, the answer is fewer restrictions and more open source, not the opposite. Thanks Ben Werdmuller for this read! I’ll link it in the description.

This is bad. What’s worth noting is Hugging Face used Chinese models to combat the attack due to guardrail restrictions on domestic models. With the frontier becoming less of a duopoly and more populated it makes sense for many companies to switch from the American labs, which likely triggers Government regulation both export control and foreign restrictions.

@shl @shl Curious how you view cutting token spend through routing

June 2026 was the first month that Gumroad spent equal amounts on AI tokens and human payroll.

It’s been 3 weeks at @ycombinator and the biggest lesson I’ve learned is that tight and honest feedback loops beat perfect plans every time. You learn more from mediocre action than flawless theory. Instead of mapping out everything detail, just go out and do it.

It seems like everyone I talk to recently has been wanting to run different models through the same TUI/CLI. Feels like confirmation of something I've believed for a while, we need a definitive, model-agnostic third-party interface

@thesupermanmx At what decode speeds though…

China has killed the GPU mafia 🤯 They open-sourced a framework that runs DeepSeek-R1 (671B params) on a single 24GB graphics card. → 3-28x faster inference → Fine-tune DeepSeek-V3 on 4x 4090s (~80GB) → 6-12x faster than ZeRO-Offload → Runs Kimi-K2, GLM, Qwen3-Next out of the box 17.3k stars. 100% Open Source.

Pretty frustrating using Claude this month. Models seem slower, constantly hitting guardrails, and burning tokens. Planning is still superior, but definitely stresses the need for multi model setups

Lots of nuance to this question, while these are all great I find myself increasingly pointing others to LFMs. On a M3 36G I get ~500 tok/s on LFM2 350M with surprisingly good outputs

When talking about the future of running on device a crucial question that remains is whether distillation/quantization of a large model extracts more value than routing between a series of small and hyper specialized local models

@TheAhmadOsman Am a firm believer that the future of running on device is not a single model with fable/kimi k3 level intelligence but rather routing between hyperspecialized local models that additively combine to rival frontier capabilities

PREDICTION We're gonna get Kimi K3 equivalent intelligence running on a single RTX PRO 6000 in less than 18 months

@ivanfioravanti @TheAhmadOsman I think a future in which this is achievable is routing between fragmented hyper specialized local models, not a single quantized model that possesses fable level intelligence

@TheAhmadOsman I think this is complex to achieve, but we’ll see. They did it with 3T parameters and strong RL. Don’t think a smaller model can achieve this “power”, but happy to be completely wrong on this.

@jun_song @jun_song As a single model with mythos level capabilities or routing between a mixture of fragmented hyper specialized local models?

New prediction : We will be able to run Mythos level local AI in 4 months. On a single Macbook with 32GB RAM. Mark this and come back.

Kimi K3 Benchmarks - seriously where is for profit in 18 months