Xu Tan

Microsoft presents Chain-of-Model Learning for Language Model

Todays, I want to introduce my latest research - "Chain-of-Model Learning for Language Model (arxiv.org/abs/2505.11820)", which presents a learning paradigm named Chain-of-Model (CoM) and devise Chain-of-Language-Model (CoLM), to unlock new functionalities of foundation models.

Congrats to the leading authors @SongKaitao @WangXiaohua and all contributors! Thanks @_akhaliq for posting this work! Hope CoLM can become a new paradigm for efficient model scaling and elastic inference in LLMs, Diffusion Models, CNNs, and any other neural networks! 10/10

4) Elastic inference: use varying chains for varying capacity and latency in inference. Due to resource limit, authors cannot scale up model any further. Call for interests to try CoLM in larger scale to unleash its potential in efficient scaling and elastic inference. 9/n

🔥 Chain-of-Model, a novel learning paradigm using the concept of chain in hidden dimension level (differ from Chain-of-Thought): 1) progressively add new chain for efficient model scaling; 2) offer varying model sizes (chains) for elastic inference. arxiv.org/pdf/2505.11820 1/n

🔥Kimi-Audio, a universal audio foundation model pre-trained on 13+ million hours of audio data and achieving SOTA performance on 10+ audio benchmarks. Tech Report: arxiv.org/abs/2504.18425 Model & Code & Evalkit: github.com/MoonshotAI/Kim… Congrats to the excellent team!

1/n: 🚀 Announcing YuE (乐) – the most powerful open-source full-song music generation model! 🎵 Tackle the lyrics-to-song task (like Suno.ai) with support for diverse genres, stunning vocals, & multiple languages. Bonus? It’s Hugging Face & LLAMA-compatible for easy fine-tuning. 🛠️ Code: github.com/multimodal-art… Demo: map-yue.github.io

@heiga_zen @KeiichiTokuda Congrats, Heiga!

Honored to be elevated to #IEEE Fellow! Huge thanks to my colleagues & collaborators throughout my career, especially Prof. @KeiichiTokuda. Grateful to #Google & Google #DeepMind for providing the opportunity to work on impactful real-world problems. ieee.org/content/dam/ie…

Scaling law for domain-specific continual pre-training scenarios.

🚀 Excited to announce that the tech report of MAP-Neo (map-neo.github.io): a fully open-source and transparent bilingual LLM suite with superior performance to bridge the gap with closed-source models, is now available: arxiv.org/pdf/2405.19327 🔧MAP-Neo's workflow encapsulates the entire process of building an LLM from data curation, model training, to evaluation and we provide the *FULL details* for every single stage, including access to its data, training code, models, and evaluation protocol 👇 [Tweet 1]

@Nor072635987 If someone has interests in studying mel-spectrogram or other representations and architectures, we can have discussions.

1/n Scaling law is the key to LLMs. How about scaling law for multimodality (e.g., audio, visual)? We plot some speech synthesis/recognition models and speech scaling law. Seems most synthesis models are OVER-parameterized compared to the compute-optimal model/data allocation.

@ch3njus Currently, there is no paper available, maybe paper in the future or just slides.

@xutan_tx Can you share a link to the paper?

Thanks Zhi Tian, @nicolaus625, @abc43992899, @GeZhang86038849, Zili Wang for the discussions.

7/n We also collect scaling law coefficients for multimodality (text/speech/image/molecules/code) and get the optimal model/data allocation. Although most modalities show similar scaling strength on model and data (a~=b~=0.5), there are clear differences... TO BE CONTINUED...