Benjamin Elizalde

We have started hiring research interns for 2025. I am looking for a PhD student with background in multimodal generation/understanding. metacareers.com/jobs/374372872…. Flexible with the timing but summers are usually the best :) #internship2025 #multimodal #audio #speech @AIatMeta

100 fully-funded PhD positions in Finland! I am one of the potential supervisors for PhDs in this new doctoral programme, come do research with us! Details on the application process and selection procedure -> fcai.fi/doctoral-progr… Deadline 2nd of April.

10 PhD positions available in the Bioacoustic AI network, including one position in Audio Research Group of @TampereUni. Apply to the TAU position here: tuni.rekrytointi.com/paikat/?o=A_RJ…

``PAM: Prompting Audio-Language Models for Audio Quality Assessment,'' Soham Deshmukh Dareen Alharthi Benjamin Elizalde Hannes Gamper Mahmoud Al Ismail Rita Singh Bhiksha Raj Huaming Wang … ift.tt/6HIcVpn

Paper proposing to leverage audio-language models to measure audio quality in multiple tasks (TTA, TTM, speech denoising, etc.), in a reference-free way. One of the tricks is to compare against multiple *and opposite* quality-related text prompts. arxiv.org/abs/2402.00282

This is all thanks to the amazing collaborators at Microsoft and CMU! @benjaminelizal, @Ydhira, Dimitra Emmanouilidou, Bhiksha Raj, Rita Singh, Huaming Wang (5/5)

Prompting Audios using Acoustic Properties for Emotion Representation. Existing audio-text models are bad at speech tasks like Speech Emotion Recognition. We provide different ways to improve this performance and introduce the Emotion Audio Retrieval task (4/5)

Training Audio Captioning Models without Audio. We show how to train audio captioning models with only text and further improve performance with LLM-generated text. The paper also introduces stylized audio captioning. The data is released here: github.com/microsoft/NoAu… (3/5)

Natural Language Supervision for General-Purpose Audio Representations. We make improvements to the CLAP model and scale it to 4.6M audio-text pairs. The model is publicly available at: github.com/microsoft/CLAP (2/5)

Do you work with audio-text models? If so, we have three relevant papers appearing at ICASSP 2024 #ICASSP2024 (1/5)

🎉 Exciting News! 🎙️ Join us in exploring Explainable AI for speech and audio with our IEEE ICASSP 2024 workshop. 📝 Call for Papers: Two tracks with deadlines - January 20 & February 20. Don't miss out! 🚀 #ICASSP2024 #ExplainableAI #SpeechAndAudio

@_akhaliq The code for our new 2023 👏CLAP model is out! github.com/microsoft/CLAP

Natural Language Supervision for General-Purpose Audio Representations paper page: huggingface.co/papers/2309.05… Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.

The paper "Pengi: An Audio Language Model for Audio Tasks" is accepted at #NeurIPS2023! 🥳

The call for abstracts is open for the 22nd annual Auditory Perception, Cognition, and Action Meeting (APCAM) on Nov 16th in San Francisco, CA. Deadline 09/17/23 for abstract submission and travel award applications. Visit apcsociety.org . @acousticsorg @AuditoryAPCAM

Two good papers on LLM-based audio understanding: Listen, Think, and Understand: arxiv.org/abs/2305.10790 Pengi: An Audio Language Model for Audio Tasks: arxiv.org/pdf/2305.11834…

Pengi: An Audio Language Model for Audio Tasks introduce Pengi, a novel Audio Language Model that leverages Transfer Learning by framing all audio tasks as text-generation tasks. It takes as input, an audio recording, and text, and generates free-form text as output. The input audio is represented as a sequence of continuous embeddings by an audio encoder. A text encoder does the same for the corresponding text input. Both sequences are combined as a prefix to prompt a pre-trained frozen language model. The unified architecture of Pengi enables open-ended tasks and close-ended tasks without any additional fine-tuning or task-specific extensions. When evaluated on 22 downstream tasks, our approach yields state-of-the-art performance in several of them paper page: huggingface.co/papers/2305.11…

Checking ICLR final decisions, it made me wonder why audio research is treated so poorly in ML conferences? 🥲 I did a simple keyword search on OpenReivew and here is what I found! 👉 @iclr_conf #ICLR2023

Amazing progress in the text-audio generation space!!