🔊 New Paper Alert — Training-Free Inference-Time Timbre Transfer
Check out our latest #ICASSP paper on timbre transfer in music audio! 🎶
➡️ Diffusion Timbre Transfer Via Mutual Information Guided Inpainting; Ching Ho Lee, Javier Nistal, Stefan Lattner, Marco Pasini & George Fazekas
📜 Paper: arxiv.org/pdf/2601.01294
🥁 Audio Demos: anon-audio-demo-25.github.io/audio_demo/
In this work, we rethink timbre transfer as an inference-time editing problem — and show that you don’t need to retrain or fine-tune heavy models to change the instrumental color of a piece while preserving its musical structure.
🎯 What’s New?
Instead of training separate models or adding control modules for each instrument:
✅ We start from a pre-trained latent diffusion model and steer it on the fly using two simple controls:
• Mutual-Information guided noise injection: add noise only in latent channels most informative of timbre.
• Early-step clamping: “lock in” melody and rhythm by restoring structure-dominant channels during denoising.
This lightweight, training-free procedure lets you control timbre without sacrificing the original melody, harmony or rhythm — and works with text or audio conditioning (e.g., CLAP).
✨ Why It Matters
🎵 Practical music production tools for re-orchestration and sound design
🛠️ Efficient editing with no added model training
🔍 A framework that could extend beyond timbre to other label-driven audio edits
📌 Compatible with strong diffusion backbones and generative audio models
@SonyCSLMusic @SonyCSLParis
English
