Alex Zverianskii

3

18

J@thejacobclyne·1 Nis

this is the first time i’ve had to think about horizontal scaling for ants lol

Traditional Precision-Recall curves tell you how your code review tool performs on static benchmarks. They don't tell you how it performs against a Hawk. Introducing Fight Index (FI).

English

0

9

57

Alex Zverianskii@alexML·1 Nis

@shriyashku Out Of Evolution

English

12

Shriyash Upadhyay@shriyashku·1 Nis

OOD Evals

Traditional Precision-Recall curves tell you how your code review tool performs on static benchmarks. They don't tell you how it performs against a Hawk. Introducing Fight Index (FI).

English

0

9

132

Alex Zverianskii@alexML·1 Nis

Now I see why no code review company is called "CodeMonkey”

Traditional Precision-Recall curves tell you how your code review tool performs on static benchmarks. They don't tell you how it performs against a Hawk. Introducing Fight Index (FI).

English

13

57

Alex Zverianskii@alexML·25 Mar

@shriyashku @withmartian

QME

2

19

Shriyash Upadhyay@shriyashku·25 Mar

@alexML @withmartian I like to think of myself as a *very deep* review agent

English

0

3

33

Martian@withmartian·25 Mar

We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇

English

8

6

54

39K

Alex Zverianskii@alexML·25 Mar

@withmartian This turnaround is definitely faster than what @shriyashku usually takes for a review, so I’m all for AI.

English

0

2

25

Martian@withmartian·25 Mar

The speed data makes the split obvious. Some tools review in ~1–2 minutes. Others take 20+ minutes. This isn't a performance gap — they're optimizing for completely different things.

English

2

0

12

726

Alex Zverianskii@alexML·25 Mar

@joshgreaves_ml Well, now you can rest assured that every typing error will be caught!

English

2

34

Josh Greaves@joshgreaves_ml·25 Mar

CRB now tracks deep review agents - agents that take the time they need to perform a thorough review. Does more compute actually mean better reviews? Now we can measure it.

We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇

English

0

10

185

Alex Zverianskii@alexML·25 Mar

@rnsmith49 Most importantly, we don’t want to end up in a reality where humans are stuck in a slow loop. "Strange loop" is fine btw

English

3

18

Ryan Smith@rnsmith49·25 Mar

Personally, I love the shift in the industry of optimizing for two separate workflows - fast and iterative, vs slow and offline. And I'm always a sucker for seeing data back up intuition 😅

We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇

English

0

9

100

Alex Zverianskii@alexML·25 Mar

"Fast, cheap, and high-quality—you can only pick two". Now the Code Review Benchmark lets you compare AI code-review tools by speed. Submit your pull request in the evening and get a Deep Review by morning—finally!

We've been tracking AI code review tools across OSS, and a new category is emerging. We're calling it "Deep Review": → Standard AI review: PR-level, fast, human in the loop → Deep Review: repo-wide context, runs autonomously in the background 🧵👇

English

8

89

Shriyash Upadhyay@shriyashku·13 Mar

Our code review tracker caught the release of Claude Code Review before @AnthropicAI announced it. Greptile v4 hit #1 on CRB. The tracker caught it before their announcement. The data is predicting something new from @Devin Review in the near future. Here's how. 🧵

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

3

5

21

1.4K

Martian@withmartian·13 Mar

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

24

26

110

47.5K

Alex Zverianskii@alexML·13 Mar

@AshleyZhang110 @greptile Nice work! I love the price tracking feature—it's awesome.

English

2

75

Alex Zverianskii retweetet

Ashley Zhang@AshleyZhang110·13 Mar

I've been playing around with eval-ing AI code review tools at work. We track 22 different ones. @greptile V4 had the single biggest improvement I've ever measured. Recall increased 47% from 38.7 → 56.9%

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

5

9

60

6.4K

Alex Zverianskii@alexML·13 Mar

@shriyashku @AnthropicAI @devin Next time, let's go to Polymarket instead of X

English

2

0

4

53

Alex Zverianskii@alexML·13 Mar

@jhoysbou Looks good to me. Where’s my $25?

English

2

33

Artem@jhoysbou·13 Mar

$25 per review is wild until you remember how much human reviews cost. still wild though

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

0

8

113

Alex Zverianskii@alexML·13 Mar

@antiagc Wait, there’s a v2? How did I miss v1?

English

0

2

34

Antia García@antiagc·13 Mar

"which AI is actually good at reviewing code" is a harder question than it looks! We've been working on v2 of Code Review Bench: It has more models, more details, and sure it looks the part too.

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

0

9

98

Alex Zverianskii@alexML·13 Mar

@mwilliammyers Why limit yourself to a sonnet? Don’t be stingy.

English

0

2

34

Will Myers@mwilliammyers·13 Mar

v0.2.0 out already 🚀 I use Opus and Sonnet every day and love them but how does Claude stack up in PR reviews? Check out the data. Know your tools.

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

0

13

344

Alex Zverianskii@alexML·13 Mar

@rnsmith49 LGTM

Dansk

3

22

Ryan Smith@rnsmith49·13 Mar

Coding tools are getting way better, but also way more convincing to humans (or me at least). We definitely need to continue building robust evals for code review so we know we are keeping the "LGTM effect" in check

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

0

10

114

Alex Zverianskii@alexML·13 Mar

@withmartian @augmentcode @baz_scm At $25 or more per review, I finally understand what people mean when they say AI will replace developers!

English

6

121

Alex Zverianskii@alexML·13 Mar

22 tools. Open methodology. A lot of vendors came back with genuinely great ideas on how to measure this better. v1.0 is cooking and it's going to be a banger.

How good is Claude Code Review really, and is it worth $25+ per review? We scraped every OSS repo on GitHub that's using it to figure out how devs actually use it. Here's how it stacks up against 22 other tools: codereview.withmartian.com Featuring: @augmentcode @baz_scm @CodeAntAI @coderabbitai @cognition @cubic_dev_ @cursor_ai @GeminiApp @greptile @kilocode @kodustech @mesa_dot_dev @QodoAI

English

Josh Greaves@joshgreaves_ml

12

101

Alex Zverianskii@alexML·5 Mar

For Runtime For Runtime

Just released frfr—lightweight runtime type validation for Python. One function. Zero dependencies. 57KB installed. Validates your dataclasses, TypedDicts, and NamedTuples directly. That's the whole API. Here's why this exists:

English