Tweet Disematkan
Sentient has introduced a new framework called CryptoAnalystBench to evaluate Crypto AI Agents.
We know AI models are improving rapidly, but grading them in a fast-moving field like crypto is difficult. New information appears constantly, and standard tests often miss the mark.
Old benchmarks usually focus on simple retrieval tasks, like checking a token's price. But today, users are asking complex questions that require reasoning, such as why the market is moving a certain way. We need answers that explain the context, not just the raw data.
This benchmark was built to fix that evaluation gap. It is a test designed to see how well an AI can handle deep, time-sensitive questions. The goal is to help developers understand what real users actually need.
Sentient built this by looking at real data. They started with millions of queries and filtered them down to 198 high-quality questions across 11 categories, including Project Fundamentals and On-Chain Analytics.
To grade the answers, they look at four specific factors:
● Relevance (Did it answer the specific prompt?)
● Temporal Relevance (Is the data fresh?)
● Depth (How detailed is the analysis?)
● Consistency (Is it factually correct without contradictions?)
The judging is handled by DeepSeek-v3.1 in a blind test to ensure fairness.
The results are worth looking at. @SentientAGI open-source model, SERA Kimi K2.5, achieved the highest score, beating out Gemini 3 Pro and GPT-5.2. It also ranked first more often than any other model in head-to-head comparisons.
This proves that open-source models are becoming highly competitive with top-tier systems. However, the test also highlighted that factual accuracy is still a challenge for everyone in the space. Hopefully, we see more progress there soon.
You can read the full breakdown here:
blog.sentient.xyz/posts/cryptoan……
GitHub: github.com/sentient-agi/C…
English
