Cheng-I Jeff Lai

@rdesh26 great post!

Voice AI has grown a lot recently, and definitions of models/systems have become somewhat vague. Let's put down some basics. 1. AI "models" are not AI "systems". Models are the core units that build up a system. For text-only systems, the two are trivially equivalent (discounting the BPE tokenizer/detokenizer), but not for voice. For voice AI systems, examples of model may be ASR, TTS, LLM, SpeechLLM, OmniLLM, etc. 2. A model is the smallest replaceable unit within a system. For example, an STT model (user speech in / agent text out) often contains a speech encoder + an LLM, but neither of these components can be replaced without having to train the model again. 3. A speech-to-speech "system" (often called a voice agent) may take many forms and comprise many components, but it is always based on two requirements: (A) response generation --> what/how to respond (B) duplex control --> when to talk. Traditionally, (A) has been handled through an ASR/LLM/TTS cascade. Most of the current S2S modeling research aims to replace this pipeline with fewer models (either STT+TTS or S2S). Most systems still rely on external VADs and WebRTC for (B), with the famous exception of "full-duplex" models like Moshi. 4a. A SpeechLLM is a model that takes text+speech input, but only generates text output. It is also called a "speech understanding" model. 4b. An OmniLLM is a SpeechLLM that also generates speech (either codecs or continuous latents). It is also called a "speech generation" model (not to be confused with a TTS). 5. A speech-to-speech system is considered "realtime" if it satisfies 3 conditions: low latency (< 1s), streaming audio in/out, and barge-in/interruption handling. It can also be called a full-duplex system (not to be confused with a full-duplex "model").

@amytam01 love the post, thanks for sharing

@jiatongshi @WavLab @LTIatCMU @CarnegieMellon @AnuttaconGames @shinjiw_at_cmu Congratulations! 🎈🍾

🎓 Officially graduated from @WavLab @LTIatCMU @CarnegieMellon Excited to start a new chapter as an Audio Researcher @AnuttaconGames 🚀 Grateful for amazing mentors @shinjiw_at_cmu , collaborators, and the speech/audio community along the way.

@rdesh26 too risky

For fellow GC holders, do you always carry your original card on you?

@rdesh26 @nvidia Congrats Desh!

I’m happy to share that I’m starting a new position as Senior Research Scientist at @nvidia! Looking forward to open science for speech full-duplex models :)

@unilightwf This is for listening tests?

Ponodos makes some really interesting stuff... podonos.com/resembleai/cha…

@rdesh26 The same website template?

Overheard at the startup website builder's meeting. Reflection: That's bone. And the lettering is something called Silian Rail. Sesame: Eggshell with Romalian type. Inworld: Raised lettering, pale nimbus white. Impressive. Very nice. Let's see Paul Allen's website.

@rdesh26 Dual channel data is hard to scale

🔥 Generative Spoken Dialogue Language Modeling Learning the joint distribution P(User, System) isn’t new. In fact, Nguyen et al. (dGSLM, 2021) did this using a dual-tower transformer (figure below + link: speechbot.github.io/dgslm/index.ht…) Their recipe 👇 • HuBERT tokens • 6-layer Transformer backbone • Trained only on ~2k hrs of Fisher Even with that modest setup, the generated dialogs were shockingly fluent. The model could continue conversations with turn-taking and backchannels. Pretty cool for its time. 🔥 Now imagine scaling this up with: ✨ modern codecs (Mimi, nano-codec) ✨ a bigger backbone (8B+ MoE) ✨ orders of magnitude more podcast data …we likely get end-to-end, natural dialog generation that feels like two humans riffing on a topic. 🤯 So why didn’t we go all-in on this? I think there are two bottlenecks: 1. Two-channel dialog data is scarce + expensive. 2. Dual-tower limits utility: it only models 2-channel speech to 2-channel speech. No grounding. No knowledge infusion. You can’t feed it text, images, a PDF, or a blog and expect a meaningful podcast out the other end. The result? A brilliant proof-of-concept that generates free-flowing chatter, but doesn’t carry the world knowledge or controllability required for real applications. Still, dGSLM was quite ahead of its time and laid the foundations for modern FDX systems!

@rdesh26 I’ve wanted to write one myself

It's another grad school application season. Some of you will get into your dream programs, while others will have to make compromises. Yet more will face disappointment all round. But this won't be the end of the road. Here's something I wrote more than 8 years ago after getting 12/12 rejections in my first grad applications: @rdesh26/what-i-learned-from-my-12-grad-school-rejections-ec7c6dc0f8b9" target="_blank" rel="nofollow noopener">medium.com/@rdesh26/what-…

@pratyusha_PS Congratulations 🎉

📢 Some big (& slightly belated) life updates! 1. I defended my PhD at MIT this summer! 🎓 2. I'm joining NYU as an Assistant Professor starting Fall 2026, with a joint appointment in Courant CS and the Center for Data Science. 🎉 🔬 My lab will focus on empirically studying the science of deep learning and applying deep learning to accelerate the natural sciences. Very broadly interested in questions at the intersection of language, reasoning and sequential decision making. (Plus any other fun problems that catch our eye along the way!) 🚀 I am recruiting 2 PhD students for this cycle! If you're interested in joining, please apply here: cs.nyu.edu/dynamic/phd/ad… cds.nyu.edu/phd-admissions…

Even in chaotic moments, people can be kind

@jiatongshi Careful ablation studies are a thing of the past

This is exactly the reason we worked for ESPnet-Codec, but being really hard to keep tracking as people are fast nowadays. The similar issue happens at most speech tasks from ASR, TTS, to general speech LLM. It's a bit sad time for driving scientific findings 🥲

@hikushalhere I exactly get what you mean. 🥲

A company/org/institution that strips the people of their dignity can win but not earn respect. What's the point of anything if the people are marginalized?

@rdesh26 @JonathanLeRoux wow!

I am attending the SANE 2025 workshop today at Google NYC!

Data is more important than ever

@chenwanch1 exactly where they are now 😉

@jefflai108 I don’t know where those people are now, but it feels like the field has instead moved towards scaling supervised/pseudo-labeled data and focusing on the multi-task problem via LLMs. Obviously this works, but it has anti-synergy with the low-resource direction

a while back, almost all of the most well-known speech researchers were pushing for low-resource or even zero-resource speech. The premise of building speech technology for all spoken languages with min supervision was very alluring. I wonder what they’re up to now?

Happy Halloween!

@drlucylai that's hot

boo 👻 @ChatGPTapp @OpenAI #Halloween2025

@qqyuzu @tydsh Congrats! Catch up soon too

Hard to see the layoffs at Meta — so many brilliant people and mentors I learned from. I went through those same struggles @tydsh mentioned: the uncertainty, the long nights, the hope things would turn around — and the disappointment before finally deciding to leave. After two months of rest and reflection, I’m grateful to be joining @_inception_ai to work on diffusion LLMs — continuing my research on discrete diffusion from FAIR and exploring its potential for ultrafast, scalable reasoning in language models. 🚀 Wishing my former teammates all the best as they carry the work forward.