Pipecat AI

GTC! Head on over to AWS Booth 921, Kiosk 3. Check out building realtime AI with @NVIDIAAI Nemotron, @awscloud and @pipecat_ai

@marcuswquinn @ShaneMac 🔥

@ShaneMac amazing — could totally make a personal and private Grok with that do calls as well, and we can use @pipecat_ai

What if there wasn’t even an API? What if Language was the API? x.com/shanemac/statu…

Today's @NVIDIA Nemotron 3 Super launch is an exciting development for voice AI developers. We’re proud to be a launch partner, with day-0 @pipecat_ai support. Developers now have a meaningful open stack for realtime voice, with @NVIDIAAI — Nemotron 3 Nano, Nemotron Speech ASR, Nemotron 3 Super. Open models, open training data. Review how Nemotron 3 Super matches proprietary models in our long-conversation voice agent benchmarks. Happy building, with open source!!

NVIDIA Nemotron 3 Super launches today! We've been building voice agents with Super's pre-release checkpoints and running all our various tests and benchmarks. Nemotron 3 Super matches both GPT-5.4 and GPT-4.1 in tool calling and instruction following performance on our realtime conversation, long context, real-world benchmarks. GPT-4.1 is the most widely used LLM today for production voice agents. So an open model that performs as well as GPT-4.1 on hard, voice-specific benchmarks is a big deal. (Side note: we don't think a benchmark "tells the story" about a model's voice agent performance unless it tests model correctness across at least 20 human/agent conversation turns.) The Nemotron models are *fully* open: weights, data sets, training code, inference code. Nemotron 3 Super is 120B params, with a hybrid Mamba-Transformer MoE architecture for efficient inference. You can run it on NVIDIA data center hardware or on a DGX Spark mini-desktop machine. 1M token context. Blog post with full benchmarks, thinking budget notes, inference setup on @Modal, and where we think this goes next. 👇

🔥 Streaming now, @NVIDIAAI Nemotron Labs session. How to build voice agents on @nvidia open modals, using @modal @trydaily @pipecat_ai cc @kwindla

New STT realtime release: @AssemblyAI Universal-3-Pro, supported in Pipecat

Join @NVIDIAAI for a Nemotron Labs livestream on building voice agents with open models. Tuesday, 3/3, link in thread, @modal @pipecat_ai @nvidia @kwindla Ben Shababo

Register, luma.com/8jv5zur5

Bo Xie of @OpenAI joins our Thursday voice AI meetup. He'll talk with us about training the new gpt-realtime-1.5 speech-to-speech model, @OpenAIDevs. SF + livestream link in thread.

One of my 2026 predictions is that we're going to see a lot of interesting new experiments with LLM-powered games. There are just so, so many possibilities. The main barrier is inference cost. But that's dropping fast. My friends Vanessa and Sunah have been tinkering with a voice game called Crush Quest. Crush Quest has multiple characters, a bunch of really good prompting, and you can play on the web or (clone the repo and) wire up a telephone number. It's, you know, totally open source and that's radical. As you can maybe tell from my hip use of slang, Crush Quest is set in the early 1990s. It's an homage to a classic electronic board game called Dream Phone. Check out the thread below for a link to the most perfectly 1991 TV commercial for Dream Phone. I can taste the Lucky Charms when I watch this commercial. h/t to @chelcietay who I had a great conversation with recently about our 2026 predictions and where social and gaming is going.

Brand new speech-to-speech model from @OpenAIDevs today! GPT Realtime 1.5 achieves a very nice jump in tool calling and instruction following performance on our voice agent benchmarks. @charlierguo's demo video shows a great example of perfect performance on a hard end-to-end audio understanding and speech production task: the model captures a seven-character order number (mixed digits and numbers), and repeats it back. The demo video made me hungry. I definitely need some Inference Chips with my OpenAI Neural Net Burger.

🔥 Try gpt-realtime-1.5 for free, at pipecat.ai

Benchmarking Claude Sonnet 4.6 for Voice AI. @kwindla mentions we'll be talking benchmarks and voice AI at our Thursday meetup - link in thread.

RSVP, luma.com/8jv5zur5

🔥 Voice AI Meetup, next Thursday, 2/26! @Speechmatics @tavus @trydaily

Voice AI turn taking is a solved problem. The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.) @mark_backman made a @pipecat_ai PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it. The approach combines three layers of processing: 1. Voice activity detection, with a short (200ms) trigger. 2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed. 3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context. None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year. Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection. Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response. - ✓ means the agent should respond normally (immediately) - ○ is a "short incomplete" - the agent should wait 5 seconds - ◐ is a "long incomplete" - the agent should wait 10 seconds The wait times, and the details of the prompt, are configurable, of course. Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency. Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful. The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

Wake up, babe. New Pareto frontier chart just dropped. Benchmarking STT for voice agents: we just published one of the internal benchmarks we use to measure latency and real-world performance of transcription models. - Median, P95, and P99 "time to final transcript" numbers for hosted STT APIs. - A standardized "Semantic Word Error Rate" metric that measures transcription accuracy in the context of a voice agent pipeline. - We worked with all the model providers to optimize the configurations and @pipecat_ai implementations so that the benchmark is as fair and representative as we can possibly make it. Entirely open source. You can run the benchmark yourself and reproduce the results.

🙌 New @SarvamAI TTS model, fully integrated with @pipecat_ai s/o @SarvamForDevs

Today we're thrilled to announce our newest flagship TTS model, Arcana v3! - 120ms latency - Multilingual in 10+ languages - Word-level timestamps - 100+ concurrency - Cloud + on-prem Start building with it today!