Ryan Smith retuiteado
Open Source Models you have been sleeping on! x.com/i/broadcasts/1…
English
Ryan Smith
46 posts
Ryan Smith
@rnsmith49
Mechanistic Interpretability research, LLM Routing, and general GenAI optimization at https://t.co/eKknXT901I
Se unió Temmuz 2025
40 Siguiendo23 Seguidores
@withmartian Humans in this benchmark are built better than me, because I am definitely losing vs a crocodile
English
I unironically love the Lotka-Volterra equations, so yes I may spend my Wednesday morning diving into the code of an April Fools’ post
Martian@withmartian
Traditional Precision-Recall curves tell you how your code review tool performs on static benchmarks. They don't tell you how it performs against a Hawk. Introducing Fight Index (FI).
English
Slow Deep Review vs. quicker reviews. Is slower actually better? Which helps improve the Software Factory velocity more? Which helps improve maintainability and reduce bugs over the lifespan of the codebase more?
Martian@withmartian
We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇
English
Personally, I love the shift in the industry of optimizing for two separate workflows - fast and iterative, vs slow and offline.
And I'm always a sucker for seeing data back up intuition 😅
Martian@withmartian
We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇
English
Ryan Smith retuiteado
If this policy is not revoked, I won’t be reviewing/ACing for #NeurIPS
Science requires open exchange of ideas!
When participation gets shaped by geopolitics, it ends up reflecting power structures, not merit--narrows what science can be and powerful nations get full control!
Mathieu@miniapeur
NeurIPS 😬
English
Ryan Smith retuiteado
I've been playing around with eval-ing AI code review tools at work. We track 22 different ones. @greptile V4 had the single biggest improvement I've ever measured.
Recall increased 47% from 38.7 → 56.9%
Martian@withmartian
How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI
English
Ryan Smith retuiteado
Our code review tracker caught the release of Claude Code Review before @AnthropicAI announced it.
Greptile v4 hit #1 on CRB. The tracker caught it before their announcement.
The data is predicting something new from @Devin Review in the near future.
Here's how. 🧵
Martian@withmartian
How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI
English
Coding tools are getting way better, but also way more convincing to humans (or me at least). We definitely need to continue building robust evals for code review so we know we are keeping the "LGTM effect" in check
Martian@withmartian
How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI
English
Just released frfr—lightweight runtime type validation for Python.
One function. Zero dependencies. 57KB installed. Validates your dataclasses, TypedDicts, and NamedTuples directly.
That's the whole API. Here's why this exists:
English
Verification is easier than Generation - the same is true for Code, and I am really excited to see the pipeline of: Code Review getting more robust evals → better code review tools → better coding tools
Martian@withmartian
Introducing Code Review Bench v0: codereview.withmartian.com The first independent code review benchmark. 200,000+ PRs. Unbiased. Fully OSS. Updated daily. Tool performance highlights 🧵👇 Featuring: @augmentcode @baz_scm @claudeai @coderabbitai @cursor @GeminiApp @github @graphite @greptile @kilocode @OpenAIDevs @propelcode @QodoAI
English
Ryan Smith retuiteado
A new ARES tutorial from @Narmeen29013644:
Getting started in long-horizon interp.
When do agents fail to accurately model their environment? How de we fix them? And how can you run these experiments on your own machine?
Narmeen Oozeer@Narmeen29013644
LLM agents ignore their own environment — skipping outputs, misreading tools, repeating failures. Turns out the model knows when it’s wrong. 87% probe accuracy from activations. How we found it, fixed it, and how to try it with ARES + TransformerLens 🧵
English
@alexML @withmartian My favorite meetings are the ones where by step 2 you can tell you're allowed to tune out though
English
Your LLM agent knows it's about to fail and does it anyway. 87% probe accuracy in the residual stream *before it acts* — the model literally encodes "this is a bad idea" and then proceeds. @withmartian 's ARES + TransformerLens lets you intercept that moment and steer it mid-generation. Works best early though — by step 15 the model is too far gone. We've all been in meetings like that.
Narmeen Oozeer@Narmeen29013644
LLM agents ignore their own environment — skipping outputs, misreading tools, repeating failures. Turns out the model knows when it’s wrong. 87% probe accuracy from activations. How we found it, fixed it, and how to try it with ARES + TransformerLens 🧵
English
This is one of the biggest takeaways for me - model internals change over steps after interacting with the environment! I love this other figure @Narmeen29013644 made that shows this too - a matrix of cosine similarity between optimal steering vectors at each step:
Narmeen Oozeer@Narmeen29013644
But you can't just compute one steering vector and reuse it for the whole episode. The representation of "valid vs. invalid" drifts as the conversation goes on;per-step vectors outperform a single static one. PCA on the vectors at different time steps shows they point in genuinely different directions.
English
ARES 🤝 Mech Interp
We’re excited that you can now easily dig into model internals across long-horizon tasks with ARES! We’ve built some nice integrations with TransformerLens, with even better support for hooks coming soon!
Narmeen Oozeer@Narmeen29013644
LLM agents ignore their own environment — skipping outputs, misreading tools, repeating failures. Turns out the model knows when it’s wrong. 87% probe accuracy from activations. How we found it, fixed it, and how to try it with ARES + TransformerLens 🧵
English
Really great to see the tooling here converging on an interface!
Tyler Griggs@tyler_griggs_
SkyRL now implements the Tinker API. Now, training scripts written for Tinker can run on your own GPUs with zero code changes using SkyRL's FSDP2, Megatron, and vLLM backends. Blog: novasky-ai.notion.site/skyrl-tinker 🧵
English
Ryan Smith retuiteado
Thinking of training an RL model specialized for @openclaw using ARES. So it gets better performance and much lower token costs.
Is this something folks would be interested in using? 100 likes and I put up an endpoint
English
Ryan Smith retuiteado
We'll be presenting the ARES roadmap at office hours tomorrow at 2pm PT.
If you're interested in agents | RL | interp and want to contribute to open-source send me a DM for more info.
github.com/withmartian/ar…
English
Ryan Smith retuiteado
Paper alert 🚨: LLMs build a shared multilingual latent space for meaning, decoding into languages only later. 🌍 Performance gaps come from tokenizer bias & weaker late-layer circuits, not missing concepts. We show this mechanistically with Cross-Layer Transcoders. 🧵👇
English
Ryan Smith retuiteado
