Mitch Lewis

376 posts

Mitch Lewis banner
Mitch Lewis

Mitch Lewis

@Mitch_Lewis21

Performance Analyst @Signal_65.

Denver, CO Katılım Haziran 2023
275 Takip Edilen182 Takipçiler
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
When a small team shares one AI appliance, the number that matters is not just single-user speed but how the system holds up when several people query it at once. We ran the @nvidia DGX Spark as a RAG chatbot on a 30B model across 1, 2, 4, and 8 concurrent users to find out. Per-user throughput dips under load, but total system throughput climbs about 4x as the vLLM continuous batching interleaves the request streams rather than serializing them. Full case study: signal65.com/research/ai/ra… What we found in our testing: ➡️ Single-user throughput ran about 56 tokens per second, tapering to about 28.5 tps at 8 users, still above the comfortable reading rate ➡️ Aggregate throughput rose from about 56 tokens per second at 1 user to about 228 tps at 8 users, roughly 4x more total work under load ➡️ 4 concurrent users saw a 7.4s time to first token at about 39 tokens per second each, a responsive experience for interactive chat ➡️ Latency and throughput taper gradually rather than collapsing, so a team can grow into the appliance and sense the limits before they turn disruptive
Signal65 tweet media
English
0
5
11
96.1K
Mitch Lewis retweetledi
Ryan Shrout
Ryan Shrout@ryanshrout·
Every vendor is racing to advertise a bigger context window. We ran 91 models through a new, custom built contamination-resistant retrieval benchmark (in partnership with @KamiwazaAI) and only 3 held above 95% accuracy at 200K. The token-count race is a marketing number, not a capability. If you are buying AI on context length alone, you are buying a spec, not a result.
Signal65@Signal_65

The AI industry has a benchmark problem. Training contamination, LLM-as-judge guesswork, and gameable test sets have produced leaderboards that tell enterprises almost nothing about real-world retrieval. Full report: signal65.com/research/ai/be… So we partnered with @KamiwazaAI and built one that fights back. RIKER generates a fresh, contamination-resistant document corpus with deterministic answer keys and zero human annotation, then grades retrieval and hallucination against ground truth it fully controls. We ran 91 models through it. The results should make every vendor selling on context-window size nervous. ➡️ 27 models cleared 95% accuracy at 32K context. At 200K, only 3 survived. ➡️ Multi-document aggregation accuracy collapsed more than 2x faster than single-document retrieval. ➡️ Bigger context windows did not mean better retrieval. The token-count arms race is marketing, not capability. ➡️ Thinking models boosted retrieval by up to 64% and swept every top spot in our testing. ➡️ Hallucination held steadier than retrieval as context grew, a failure mode most evals never even isolate. Advertised context length is not a proxy for usable retrieval. If your AI strategy assumes otherwise, read this first.

English
1
3
4
677
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
The AI industry has a benchmark problem. Training contamination, LLM-as-judge guesswork, and gameable test sets have produced leaderboards that tell enterprises almost nothing about real-world retrieval. Full report: signal65.com/research/ai/be… So we partnered with @KamiwazaAI and built one that fights back. RIKER generates a fresh, contamination-resistant document corpus with deterministic answer keys and zero human annotation, then grades retrieval and hallucination against ground truth it fully controls. We ran 91 models through it. The results should make every vendor selling on context-window size nervous. ➡️ 27 models cleared 95% accuracy at 32K context. At 200K, only 3 survived. ➡️ Multi-document aggregation accuracy collapsed more than 2x faster than single-document retrieval. ➡️ Bigger context windows did not mean better retrieval. The token-count arms race is marketing, not capability. ➡️ Thinking models boosted retrieval by up to 64% and swept every top spot in our testing. ➡️ Hallucination held steadier than retrieval as context grew, a failure mode most evals never even isolate. Advertised context length is not a proxy for usable retrieval. If your AI strategy assumes otherwise, read this first.
Signal65 tweet media
English
1
3
12
263.2K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
The economics of persistent AI agents look different from chatbot workloads. Always-on agents consume orders of magnitude more tokens, and per-token cloud pricing isn't built for that pattern. Our report: signal65.com/research/ai/th… The most striking case in our analysis came from software development workloads. One @Dell Pro Max with @nvidia GB300 workstation delivered 87% lower cost than the equivalent cloud API spend and saved $926K over two years. That is one workstation, under a desk, doing the work of a $1.06M cloud bill. The full report covers two more workload profiles and scales up through the Dell workstation portfolio to PowerEdge.
Signal65 tweet media
English
3
2
25
471K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
Persistent AI agents don't consume tokens like chatbots do. They run continuously, generate at least 4x to 15x more tokens than single-turn interactions we've gotten used to, and autonomous agents may push that closer to 1000x. It fundamentally changes the math on token production, and on-prem compute vs cloud APIs. We modeled three persistent agent workloads on @Dell AI Factory with @nvidia infrastructure vs. cloud over a two year period. What we found: ➡️ On-prem reduced costs by 28% to 90%+ across all workloads ➡️ Dell Pro Max with NVIDIA GB300 Ultra delivered 87% lower cost for software development, $926K in two-year savings from a single workstation ➡️ Most platforms broke even in under a year, some in as few as 2 months ➡️ The portfolio scales from desktops handling 8 agents to PowerEdge systems supporting 18,000+ Full report: signal65.com/research/ai/th…
English
1
2
6
5K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
AI cloud costs are one of the largest line items for enterprises running training and inference at scale. Our new 3-year TCO analysis, now featured on the CoreWeave site, evaluates AI-native infrastructure against general-purpose hyperscalers across compute, storage, networking, orchestration, and support. Key findings from our analysis: ➡️ Up to 47% lower 3-year total cost on CoreWeave versus leading hyperscalers ➡️ Up to 96% more TFLOPs per dollar ➡️ Up to 54% lower TCO when normalized for GPU efficiency ➡️ Eliminating data egress and API fees removes what can be millions in hidden cost at petabyte scale ➡️ Higher model FLOPs utilization (50%+) on AI-native infrastructure versus 35 to 45% on general-purpose clouds Read the full report: coreweave.com/reports/ai-clo… @CoreWeave
English
0
2
2
4.1K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
Cloud cost structure is one of the biggest variables in AI infrastructure strategy. We ran a 3-year TCO analysis comparing @CoreWeave to average hyperscaler pricing across GPUs, storage, networking, observability, and support. Key findings from our analysis: ➡️ Up to 47% lower TCO over 3 years vs. hyperscaler competitors ➡️ Up to 96% more TFLOPs per dollar ➡️ Up to 54% lower TCO when normalized for GPU efficiency ➡️ At the Small configuration (72 GPUs, 1.785 PB), CoreWeave came in roughly $14M below the hyperscaler stack, with GPU and storage costs driving the majority of the delta Full report: signal65.com/research/ai/co…
Signal65 tweet media
English
1
3
9
170.3K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
Download our infographic (and full report) looking at the advantages of @AMD EPYC CPUs for AI host node work: signal65.com/research/ai/im… We tested AMD EPYC vs. competitive options as host CPUs across 7 AI models to measure the real-world impact. Key findings from our testing: ➡️ Up to 14% higher throughput for request processing and output tokens ➡️ Up to 46.5% faster time to first token ➡️ Up to 11.4% lower inter-token latency The takeaway: choosing the right host CPU is a practical lever for improving AI inference performance and cost efficiency at scale.
Signal65 tweet media
English
1
1
2
119
Mitch Lewis retweetledi
Ryan Shrout
Ryan Shrout@ryanshrout·
We previously published an analysis at @Signal65 looking at how the shift from dense to MoE models was rewriting AI inference economics. The core argument: cost per token, rather than FLOPS per dollar, is what determines whether AI services can scale profitably. Today @nvidia published a new article around the same thesis, with "cost per token" as the defining TCO metric for AI infrastructure and introducing the concept of the "inference iceberg," where the real value sits below the surface in platform-level codesign. It tracks closely with what we found in our testing. Using available benchmark data, we showed that GB200 NVL72 delivered as low as 1/15th the cost per token of competing platforms on DeepSeek-R1, despite a ~2x per-GPU price premium. The "more expensive" platform was actually the cheaper one to operate. NVIDIA takes it further with generational Hopper-to-Blackwell comparisons showing 35x lower cost per million tokens. The convergence here is great to see. Analysis arriving at the same conclusion through different paths gives the argument more weight for enterprise buyers evaluating infrastructure decisions. Our analysis: signal65.com/research/ai/fr… The NVIDIA blog: blogs.nvidia.com/blog/lowest-to…
English
0
3
18
7.4K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
AI-generated code is accelerating development velocity, but it is also creating a quality assurance gap. To assess how well today's AI code review tools fill that gap, Signal65 tested five tools against 60 real historical bugs across six open source repositories spanning Python, Java, JavaScript, TypeScript, Go, and Ruby. Key findings from our testing: signal65.com/research/ai/ev… ➡️ CodeRabbit identified the most critical bugs (25) of any tool evaluated ➡️ CodeRabbit achieved 95.88% precision, with only 4 false positives across all repositories ➡️ Qodo Merge found the most raw bugs (129) but generated 7.5x more false positives than CodeRabbit ➡️ GitHub Copilot produced the most noise, with 41 false positives and just 64.35% precision ➡️ CodeRabbit led in critical bug detection in 5 of 6 repositories tested The chart tells the story clearly. The ideal position is upper right: high true bug detection, minimal incorrect findings. @coderabbitai sits there. Tools that chased volume paid for it with noise.
Signal65 tweet media
English
0
2
3
1.1K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
As we get into @NVIDIAGTC week, one topic I expect to get some attention is on the storage front. GPU utilization in AI inference is a storage problem as much as a compute problem. @Signal_65 worked with @HPE and @KamiwazaAI to test KV-Cache offloading to the HPE Alletra Storage MP X10000, and the results were significant. Full report: signal65.com/research/maxim… Key findings from our testing: ➡️ Output token generation rates increased up to 19.4x compared to systems with no KV-Cache ➡️ Time to first token improved up to 21.5x vs. no KV-Cache ➡️ Even vs. memory-only offload, adding the X10000 delivered a 5.9x token rate increase and 5.6x TTFT reduction ➡️ RDMA for S3 storage delivered up to 2x the throughput of traditional S3 over HTTP with 80% lower latency and dramatically reduced CPU overhead That last point matters. The table below shows why GPU Direct Storage via RDMA changes the equation: ~20 GB/s+ throughput, 5.1x latency reduction, and consistent P99 performance where traditional S3 showed high jitter.
Signal65 tweet media
English
5
3
26
331.7K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
Efficiency is the new benchmark. 🔋 Signal65 lab testing shows the @MediaTek Kompanio Ultra 910 delivers nearly 5x the efficiency of the latest x86 Chromebooks and uses up to 85% less power under load. All-day battery + flagship performance = no more compromise. Full report: signal65.com/research/the-k…
Signal65 tweet media
English
1
3
14
210.2K
Mitch Lewis retweetledi
Ryan Shrout
Ryan Shrout@ryanshrout·
I'm really excited to finally get this second round of our research out into the wild! 🚀 At @Signal_65, we’ve been heads-down working with the @KamiwazaAI KAMI v0.1 benchmark to solve one of the biggest problems in AI: benchmark "gaming." Seeing GPT-5 hit a 95.7% accuracy score in our testing is impressive, but the real story is how we got there. By using multi-layer randomization, the benchmark moved past static tests that models can just memorize. This framework is a huge milestone, it gives us a rigorous, uncheatable foundation to test the next generation of agentic AI. This is just the beginning of how we’re going to be measuring model reasoning and enterprise readiness moving forward.
Signal65@Signal_65

The Agentic AI Era Needs Different Benchmarks. As enterprise AI moves from simple chatbots to complex "agentic" workflows, traditional benchmarks are hitting a wall. They often suffer from data contamination (memorization) and fail to reflect real-world business tasks. The latest report from @Signal_65 that uses the @KamiwazaAI KAMI v0.1, a benchmark that uses dynamic randomization to ensure models are actually reasoning, not just repeating training data. After testing across 170,000 items and 5.5 billion tokens, the results are incredibly interesting. Key Highlights from the Report: ✅ GPT-5 Leadership: GPT-5 (Medium Reasoning) takes the #1 spot with a staggering 95.7% mean accuracy score, showcasing leading agentic capability. ✅ The Rise of Competitors: GLM-4.6 (#2) and DeepSeek-v3.1 (#3) are showing incredible strength, outperforming many established proprietary models. ✅ Preventing "Gaming": Unlike static benchmarks, KAMI’s randomization makes it nearly impossible for models to "memorize" the test, providing a true measure of enterprise readiness. ✅ Reasoning Matters: The data shows that "Medium Reasoning" modes significantly boost task success rates across the board. For anyone evaluating model deployment for agentic tasks, this report is a must-read for understanding the trade-offs between open and proprietary leadership. Read the full analysis here: signal65.com/research/ai/be…

English
0
1
3
953
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
The Agentic AI Era Needs Different Benchmarks. As enterprise AI moves from simple chatbots to complex "agentic" workflows, traditional benchmarks are hitting a wall. They often suffer from data contamination (memorization) and fail to reflect real-world business tasks. The latest report from @Signal_65 that uses the @KamiwazaAI KAMI v0.1, a benchmark that uses dynamic randomization to ensure models are actually reasoning, not just repeating training data. After testing across 170,000 items and 5.5 billion tokens, the results are incredibly interesting. Key Highlights from the Report: ✅ GPT-5 Leadership: GPT-5 (Medium Reasoning) takes the #1 spot with a staggering 95.7% mean accuracy score, showcasing leading agentic capability. ✅ The Rise of Competitors: GLM-4.6 (#2) and DeepSeek-v3.1 (#3) are showing incredible strength, outperforming many established proprietary models. ✅ Preventing "Gaming": Unlike static benchmarks, KAMI’s randomization makes it nearly impossible for models to "memorize" the test, providing a true measure of enterprise readiness. ✅ Reasoning Matters: The data shows that "Medium Reasoning" modes significantly boost task success rates across the board. For anyone evaluating model deployment for agentic tasks, this report is a must-read for understanding the trade-offs between open and proprietary leadership. Read the full analysis here: signal65.com/research/ai/be…
Signal65 tweet media
English
1
1
3
1.1K
Mitch Lewis retweetledi
Six Five Media
Six Five Media@TheSixFiveMedia·
Agentic AI is exposing the limits of traditional LLM benchmarks—and forcing a rethink of how enterprise AI value is actually measured. Analysts @ryanshrout & @Mitch_Lewis21 join Kamiwaza AI Platform Engineer @RoigJV to unpack new research from @Signal_65 and @KamiwazaAI that demonstrates why legacy benchmarks miss real-world agentic work, and how the Kamiwaza Agentic Merit Index (KAMI) delivers a more meaningful way to evaluate enterprise-ready agentic AI.
English
1
6
24
166.4K
Mitch Lewis retweetledi
Signal65
Signal65@Signal_65·
Our latest research shows how Amazon Bedrock AgentCore from @awscloud cuts through the complexity that usually slows teams down. We ran hands-on evaluations, and the results were hard to ignore. 📈 Teams saw 2.1x faster end-to-end agent development. 📈 They spent 75 percent less time wrestling with infrastructure and integrations. 📈 And deployment moved 5.2x faster. Why does this matter? Because the next wave of AI adoption hinges on whether enterprises can move from prototypes to production without the usual friction. If you are building agents or modernizing your AI stack, this is worth a closer look. Dive into the full report: signal65.com/research/ai/ac…
Signal65 tweet media
English
0
2
6
10.3K
Mitch Lewis retweetledi
Ryan Shrout
Ryan Shrout@ryanshrout·
Watching the @Signal_65 engineers go through this testing was super interesting. The ease of access to these tools and the kind of capabilities you can drive as a small team are really impressive.
Signal65@Signal_65

Our latest research shows how Amazon Bedrock AgentCore from @awscloud cuts through the complexity that usually slows teams down. We ran hands-on evaluations, and the results were hard to ignore. 📈 Teams saw 2.1x faster end-to-end agent development. 📈 They spent 75 percent less time wrestling with infrastructure and integrations. 📈 And deployment moved 5.2x faster. Why does this matter? Because the next wave of AI adoption hinges on whether enterprises can move from prototypes to production without the usual friction. If you are building agents or modernizing your AI stack, this is worth a closer look. Dive into the full report: signal65.com/research/ai/ac…

English
0
1
1
748

Keşfet

@nvidia @KamiwazaAI @Dell @GeminiApp @OpenAI @grok @SpaceX @MichaelDell