Cheng Han Chiang (姜成翰)

🚨 New paper! SHANKS lets spoken language models (SLMs) think while listening💭👂 This enables the SLM to interrupt the user in a timely manner and make early tool calls when the speaker is still speaking. Paper: arxiv.org/abs/2510.06917 Project page: d223302.github.io/SHANKS/

Glad to see that all four reviewers I spent a lot of time writing for are classified as full human-written. Kind of

LLM reasoning/tool use improves results but adds latency, hindering real-time dialogue. SHANKS is a new method that enables simultaneous "hearing" and "reasoning/tool use" to reduce this lag. Work from Cheng-Han Chiang & Microsoft researchers. arxiv.org/abs/2510.06917

If you are interested in thinking and reasoning for SLM, our previous work, STITCH, is a must-see. STITCH enables SLMs to think while speaking. Paper: arxiv.org/abs/2507.15375 Demo: d223302.github.io/STITCH/

This work was done during my internship at Microsoft with @Orpheus_wang, @LINJIEFUN, Chung-Ching Lin, @linkeyun2, Shujie Liu, @ZhendongWang6, @zhengyuan_yang, @HungyiLee2, Lijuan Wang

🚨 New paper! SHANKS lets spoken language models (SLMs) think while listening💭👂 This enables the SLM to interrupt the user in a timely manner and make early tool calls when the speaker is still speaking. Paper: arxiv.org/abs/2510.06917 Project page: d223302.github.io/SHANKS/

SPS SLTC/AASP Webinar: Foundational Speech Models and their Efficient Training with NVIDIA NeMo Date: 27-August-2025 Time: 9:00 AM ET (New York Time) Presenter: Dr. Piotr Żelasko Registration link: landing.signalprocessingsociety.org/ieee-sps-webin…

This work was done during my internship at Microsoft GenAI, mentored by @Orpheus_wang, collaborating with Chung-Ching Lin, @linkeyun2, @LINJIEFUN, Radu Kopetz, Yao Qian, @ZhendongWang6 , @zhengyuan_yang , @HungyiLee2 , Lijuan Wang Thanks for the great mentorship and collaboration.

Our work shows strong potential for using audio LLMs to evaluate speech generated by spoken language models (SLMs) 🎙️✨ Excited to see future work explore fine-tuning SLMs with rewards or feedback from these audio-LLM judges 🧑‍⚖️🔁

🎉 Excited to share that our paper on audio-LLM-as-a-judge has been accepted to EMNLP 2025 Findings! 🔗 arxiv.org/abs/2506.05984… 🗝️ Highlights: 🧑‍⚖️ Agreement between human and audio-LLM-judge can be as high as human-human agreements 👑 Gemini-2.5-pro outperforms GPT-4o-audio as a speaking-style judge 🗣️ There's still room for improvement in style following & natural dialogue generation for SLMs

This work began as my summer intern project last year, and has since grown into a long-term effort in large-scale pretraining. Huge thanks to project leaders @Hu_Hsu @ShangwenLi1, and @yangli625 @dongwang218 for all the dedication! @metaai @MIT_CSAIL 🧑‍💻 github.com/facebookresear…

Excited to be at #ACL2025! 🎉 Looking forward to sharing our research and learning from the community. Open to discussions on: - LLM-as-a-judge: our paper, TRACT, was presented this morning - Audio-LLM-as-a-judge for speaking style - STITCH: Our newest SLM that can think and talk simultaneously Let’s connect! 🤝 #ACL2025NLP

@JulianSlzr Thank you for sharing! I really agree that real-time thoughts can make the SLM generate better quality and make SLMs more performant. Looking forward to seeing more works in this direction.

Good to see research tackling real-time thoughts for streaming speech! Will be essential to closing the LM-SLM gap.

7/7 📈 On math reasoning datasets, STITCH improves over baselines that don’t think before responding and matches full CoT methods. 🚫 Non‑reasoning SLMs: Low latency, low accuracy 🐢 Full CoT before responding: High latency, high accuracy ⚡ STITCH‑S: Low latency, high accuracy

6/7 🆕 We explore another variant called STITCH‑S, where the model generates a spoken response chunk before the first reasoning chunk. 🏃 STITCH‑S has the same latency as a non‑reasoning model by design but delivers much better performance.

1/7 🔗 Introducing STITCH: our new method to make Spoken Language Models (SLMs) think and talk at the same time. Paper link 👉 arxiv.org/abs/2507.15375