Sabitlenmiş Tweet
ack3
316 posts
ack3
@ack3ai
AI-native cybersecurity audits and tools | Creators of @WakeFramework | Ethereum | Solana | Infra ▮
EVM Katılım Ocak 2022
1.5K Takip Edilen10.1K Takipçiler
ack3 retweetledi
1/
LLM performance is not uniform across programming languages: it tracks training-data representation, and Solidity sits at the low-resource end.
In 2026, the strongest agent evaluated detects 45.9% of real Solidity vulnerabilities; LLM-written Solidity tests are 2–4.6% correct.
The evidence:
English
@WakeFramework — an open-source security framework for human auditors and LLMs.
We built it to vibe-code tests, so auditors can focus on the big picture.
Deedy@deedydas
When you extend the analysis to 100 programs, C / Java are nearly 2.5-3x less information dense than Python. In other words, Python is the most token efficient way for LLMs to program in
English
@raopreetam_ @pashov FP rate is hard to evaluate at this scale + we focused only on highs and criticals. Would be great if you share your benchmarks as well (this measurement is also a side product of our internal AI benchmark)
English
@0xTomass @pashov We benchmark AI on our own audit pipeline too. Findings-per-dollar is the wrong axis on its own;
what are the FP rate and the critical recall for each? On real client scopes, we'll take fewer findings with near-zero false positives over a longer list we have to manually triage
English
We just spent $644 to benchmark @pashov with Opus 4.7 MAX:
Opus 4.7 — 48/94 found · $2.18/finding
Pashov skill — 49/94 found · $11.01/finding
Same model underneath, 5× the spend for one extra finding.
Anyone having similar results? Dataset in comment.
pashov@pashov
🚨Ethereum Developers: you can now install your first AI Auditor in 1 minute - fully autonomous, available 24/7, with multiple sub-agent helpers. Open Source. FREE to use (with your AI model) and already finding vulnerabilities in smart contracts. Link below🫡
English
@balakhonoff @pashov Both Claude Code here. Do you plan to release benchmarks?
English
Which harness do you use guys? It impacts the results a lot! We’ve just tested our Nethermind AuditAgent vs Claude Code with Opus 4.8 - the result is
- Pretty much the same Precision ~56%
- Two times lower recall ~28%, which means our specialized agent finds two times more correct security issues
English
@akashjana__ put plz the comment here so we don't forget to evaluate
ack3@ack3ai
Who's waiting on the next School of Solana cohort? If 100 real builders comment on this, we start it. SCHOOL OF SOLANA / NEXT · AGENTIC ▮
English
@lonelysloth_sec @pashov open to suggestions for a better dataset that a) has verified all findings (recall), b) is public (reproducibility)
English
ack3 retweetledi
Who's waiting on the next School of Solana cohort?
If 100 real builders comment on this, we start it.
SCHOOL OF SOLANA / NEXT · AGENTIC ▮
Solana Developers@solana_devs
gm developoooors, ready to take your Rust and Solana skills to the next level? - 100% free course for Solana developers - 9 weeks - Fully online Apply for @AckeeBlockchain's School of Solana by October 13th 👇
English
Q4 2025 was our wake-up call.
We benchmarked current models on unpublished audits.
Our auditors: 31 findings.
Our AI system: 35 findings.
We won't ship a manual review without AI again. And we won't ship AI without a human. The gap to the attackers is the AI system we develop.
HUMAN + AI: ZERO HACKS · '26 HACK SEASON ▮
English
