Hayate Iso

Announcing NVIDIA Nemotron 3 Super! 💚120B-12A Hybrid SSM Latent MoE, designed for Blackwell 💚36 on AAIndex v4 💚up to 2.2X faster than GPT-OSS-120B in FP4 💚Open data, open recipe, open weights Models, Tech report, etc. here: research.nvidia.com/labs/nemotron/… And yes, Ultra is coming!

What if you could ask a chatbot a question the size of an entire encyclopedia—and get an answer in real time? Multi-million token queries with 32x more users are now possible with Helix Parallelism, an innovation by #NVIDIAResearch that drives inference at huge scale. 🔗 nvda.ws/4eCXxqh

😤

Inference at scale is pushing the boundaries of what today’s systems can handle. One promising direction? #DisaggregatedInference — splitting the serving pipeline into distinct stages like prefill and decode — to unlock better performance across the throughput-interactivity spectrum. While the concept has gained traction and sparked a wave of open source experimentation, real-world deployments remain rare. Why? Because the optimization space is vast, and orchestrating system-level coordination is incredibly complex. 📄 In our latest #NVIDIAResearch, we present a comprehensive systematic study of disaggregated inference at scale, evaluating hundreds of thousands of design points across varied model sizes, traffic patterns, and hardware configurations. Key findings: 1️⃣ Disaggregation shines in prefill-heavy traffic patterns and with larger models. 2️⃣ Dynamic rate matching and elastic scaling are essential to achieving Pareto-optimal performance. 3️⃣ There's no one-size-fits-all architecture — workload-aware tuning is critical. Whether you're building production inference infrastructure or exploring #LLM optimization techniques, this work offers actionable insights to balance system throughput with low-latency responsiveness. 🔬 Read the paper and dive into the data — let’s shape the future of scalable #AI serving, together. 📗research.nvidia.com/publication/20…

👀New #NVIDIAResearch on boosting MoE model performance with disaggregated serving. Learn how our NVIDIA Dynamo and GB200 NVL72 work together to boost the performance of #AI data centers running MOE models like DeepSeek R1 and the new Llama 4. ⚡ Technical deep dive➡️ nvda.ws/45eElfW

🚀 Want to improve your LLM responses? Read our tutorial for implementing AmbigNLG! Addressing task ambiguity in Natural Language Generation to drive more accurate, context-aligned outputs. @iso_map megagon.ai/ambignlg-a-tut… #AmbigNLG #NLP #tutorial #LLMs #EMNLP2024 #MLEngineering

🌴Heading to #EMNLP2024! Presenting AmbigNLG with @ayaniwa1213 on Tuesday at 4pm (Riverfront Hall). Paper: aclanthology.org/2024.emnlp-mai… Data: github.com/megagonlabs/am… You can also stop by our @MegagonLabs sponsor booth or DM me to chat about full-time and internship opportunities :)

The CFP of DAIS@ICDE2025 has been released 👇

📢Excited to bring the DAIS workshop to ICDE'25 (w/ @SainyamGalhotra @FarihaAnna @MikeCafarella @sairamgv) The focus is on the emerging idea of compound AI systems with a specific emphasis on data discovery, interactions w/ data, architectures for #AgenticAI+#LLM, and evaluation.

Also paper link 🫠 🗒️ arxiv.org/abs/2410.11996 7/

We hope our work sets a new standard for evaluating holistic reasoning in LLMs! Proudly co-authored with @SAYg_7 (equal first author!) and Nikita Bhutani at @MegagonLabs. 💻 Code: github.com/megagonlabs/ho… 🗂 Dataset: hf.co/datasets/megag… 6/

Introducing 🫧 HoloBench—a new benchmark for measuring LLMs' reasoning capabilities across massive document collections! - RAG models fetch info well but struggle with multi-doc reasoning. - HoloBench evaluates how LLMs synthesize and aggregate info to solve complex tasks. 🧵 1/

⚡️Are you working with #NLP or #AI-driven products for Natural Language Generation (#NLG)? Task ambiguity is a common pain point and AmbigNLG changes that! AmbigNLG is designed to solve task ambiguity in instructions for NLG. What it is... 👇🧵

🎉 Our long paper has been accepted for the #EMNLP2024 main conference! In "AmbigNLG," co-authored with @iso_map, we tackle task ambiguity in NLG instructions to better align LLM outputs with your expectations. 📃 Read more: arxiv.org/abs/2402.17717

Let’s push the boundaries of #LLMs in text editing tasks! XATU is the first-of-its-kind benchmark that addresses the nuances of text editing, revolutionizing how we edit text with large language models. Don't miss the presentation at #Coling2024! #AI #NLP megagon.ai/xatu-fine-grai…

📢New Preprint📢 LLMs excel at ranking items for retrieval/recommender systems. But, what if we reduce the number of items and instead enforce multiple conditions as ranking instructions? It turns out, LLMs have a long way to go. See our new paper: arxiv.org/abs/2404.00211. 1/n