Xiaofei Wang

🚨 New paper! SHANKS lets spoken language models (SLMs) think while listening💭👂 This enables the SLM to interrupt the user in a timely manner and make early tool calls when the speaker is still speaking. Paper: arxiv.org/abs/2510.06917 Project page: d223302.github.io/SHANKS/

🎉 Excited to share that our paper on audio-LLM-as-a-judge has been accepted to EMNLP 2025 Findings! 🔗 arxiv.org/abs/2506.05984… 🗝️ Highlights: 🧑‍⚖️ Agreement between human and audio-LLM-judge can be as high as human-human agreements 👑 Gemini-2.5-pro outperforms GPT-4o-audio as a speaking-style judge 🗣️ There's still room for improvement in style following & natural dialogue generation for SLMs

Please take a look at our latest work and the insights we've uncovered. Thanks to all co-authors for their contributions.

Best Paper Awards of SLT 2024: Contextualized Automatic Speech Recognition with Dynamic Vocabulary. E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTS. Congratulations!

Microsoft presents ELaTE Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like paper page: huggingface.co/papers/2402.07… ELaTE is a zero-shot text-to-speech (TTS) system that can generate natural laughing speech from any speaker based on a speaker prompt to mimic the voice characteristic, a text prompt to indicate the contents of the generated speech, and an input to control the laughter expression.

SpeechX: Neural Codec Language Model as a Versatile Speech Transformer paper page: huggingface.co/papers/2308.06… Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks.

SpeechX from our new paper is a single generative model that edits, enhances & creates speech, enabling zero-shot TTS, spoken content editing (while preserving ambience), speaker extraction & speech/noise removal. Demo: aka.ms/speechx Paper: arxiv.org/abs/2308.06873