Abheer Singh

211 posts

Abheer Singh banner
Abheer Singh

Abheer Singh

@abheer

Accelerating AI-Native Startups @NVIDIA // Prev (GTM @tenstorrent @oracle) // Views are my own.

San Jose, CA Katılım Şubat 2020
351 Takip Edilen196 Takipçiler
Sabitlenmiş Tweet
Abheer Singh
Abheer Singh@abheer·
NVIDIA Dynamo comes up in almost every large scale inference conversation. Ik a lot of people in the local AI community and beyond have seen the name floating around and are curious. Even if it never applies to your setup, the engineering is very cool. Worth breaking down.
English
6
4
48
17.6K
Abheer Singh retweetledi
Eagle
Eagle@EagleCorp·
EAGLE 3.1 has been merged into @nvidia TensorRT-LLM. The integration brings EAGLE's latest speculative decoding architecture to one of the most widely used inference frameworks for NVIDIA GPUs. EAGLE 3.1 introduces architectural improvements that increase drafter robustness, improve acceptance length across multiple model families, and deliver more consistent performance under real-world deployment conditions. Making these improvements available in TensorRT-LLM simplifies adoption for developers building high-performance inference pipelines on NVIDIA infrastructure.
Eagle tweet media
English
2
1
103
16.1K
Wendy
Wendy@wendylabsinc·
The lineup for the WendyOS release is going be sick!
Wendy tweet media
English
3
1
20
1.4K
Abheer Singh
Abheer Singh@abheer·
Important clarification; Dynamo can be used with or without disaggregated inference.
Vikram@msharmavikram

@abheer Thanks for the great write up. Nit: Dynamo can be used even with or without disagg inference!

English
0
0
11
2.4K
Vikram
Vikram@msharmavikram·
@abheer Thanks for the great write up. Nit: Dynamo can be used even with or without disagg inference!
English
1
0
5
2.7K
Abheer Singh
Abheer Singh@abheer·
NVIDIA Dynamo comes up in almost every large scale inference conversation. Ik a lot of people in the local AI community and beyond have seen the name floating around and are curious. Even if it never applies to your setup, the engineering is very cool. Worth breaking down.
English
6
4
48
17.6K
Abheer Singh retweetledi
Ben Pouladian
Ben Pouladian@benitoz·
Dynamo may be one of the most slept-on pieces of NVIDIA’s stack The next leg of AI economics won’t come only from faster chips it will come from orchestrating inference better: separating prefill and decode, dynamically allocating GPUs, and driving down cost per token at scale🧵
Abheer Singh@abheer

NVIDIA Dynamo comes up in almost every large scale inference conversation. Ik a lot of people in the local AI community and beyond have seen the name floating around and are curious. Even if it never applies to your setup, the engineering is very cool. Worth breaking down.

English
1
5
39
7.3K
Ben Pouladian
Ben Pouladian@benitoz·
@abheer Nicely done. I’ve been screaming from the rooftops about how important and slept on Dynamo is.
English
1
0
1
79
Kyle Kranen
Kyle Kranen@KranenKyle·
@abheer Dynamo can be helpful at the scale of 2 GPUs!
English
1
1
7
263
Abheer Singh
Abheer Singh@abheer·
Dynamo runs the inference engines and routes requests to them. If one engine crashes mid-run, the work in progress isn’t lost. Dynamo sends it to another engine and picks up where it left off. The engines hold nothing but the current model weights, so a crash only costs whatever was in flight at that moment. cognition.com/blog/swe-1-7
English
0
0
6
115
Abheer Singh
Abheer Singh@abheer·
@NaderLikeLadder Some of the most important work in the ecosystem rn. We must protect local AI. Thank you for amplifying 🫡
English
1
0
6
97
Abheer Singh
Abheer Singh@abheer·
It serves two workloads for them. Large-scale LoRA in Lab, and large-scale MoE inference. LoRA adapter support was co-designed with NVIDIA to place and route adapters across engine instances. Dynamo is being integrated into their prime-rl library as the backbone for agentic RL rollouts. primeintellect.ai/blog/nvidia-co…
English
0
0
9
208
Abheer Singh
Abheer Singh@abheer·
Everyone’s talking about @PrimeIntellect right now, for good reason. Their open superintelligence stack is built off excellent first principles. Under the hood, that stack is powered by Dynamo.
Abheer Singh@abheer

NVIDIA Dynamo comes up in almost every large scale inference conversation. Ik a lot of people in the local AI community and beyond have seen the name floating around and are curious. Even if it never applies to your setup, the engineering is very cool. Worth breaking down.

English
1
2
40
4.1K
Abheer Singh
Abheer Singh@abheer·
@jtlin Agreed. Luckily there are some excellent developers/startups who are making this a reality and we will do our best to enable them as much as possible.
English
0
0
4
169
Justin Lin
Justin Lin@jtlin·
@abheer Thanks for this great overview. Honestly, we need some of this in local AI too - esp. disaggregated prefill / decode and tiered KV cache. eg. pair an eGPU with an RTX Spark. 8+ concurrency for long-context agents will soon be a common need for local AI setups.
English
2
0
5
241
Dee
Dee@dee_hw·
It's Saturday. A good day to build your own Personal AI Computer. Adding another 4x RTX PRO 6000 rig for the team.
Dee tweet media
English
20
9
199
11.2K
Abheer Singh
Abheer Singh@abheer·
Dynamo is built for scale. It serves trillion-parameter models like Kimi to large concurrent user counts, and boosts requests served by up to 7x on Blackwell in the SemiAnalysis InferenceX benchmark. Dynamo makes sense at high volume, where disaggregation and KV-aware routing pay off. Read more: developer.nvidia.com/blog/nvidia-dy… baseten.co/inference-engi…
English
1
1
18
455
Abheer Singh
Abheer Singh@abheer·
The planner is an autoscaler. It watches live traffic and adds or removes prefill and decode workers to hit your latency targets, TTFT and inter-token latency. If long inputs surge, it spins up more prefill workers. KVBM handles where cache lives. Hot KV blocks stay in GPU memory, colder ones drop to host memory, SSD, then object storage. This frees GPU memory without recomputing. NIXL is the transfer layer. It moves data between GPUs over whatever link exists, NVLink, InfiniBand, RoCE, or Ethernet, behind one API, so the rest of the system doesn’t care about the hardware. A single model can also span two or more GPU nodes as one replica, usually with expert parallelism for large MoE models.
English
1
0
14
608

Keşfet

@nvidia @wendylabsinc @msharmavikram @benitoz @KranenKyle @cognition @NaderLikeLadder @PrimeIntellect