Edouard Grave

Gemma 4 is here! 🧠 31B and 26B A4B for models with impressive intelligence per parameter 🤏E2B and E4B for mobile and IoT 🤗Apache 2.0 🤖Base and IT checkpoints available Available in AI Studio, Hugging Face, Ollama, Android, and your favorite OS tools 🚀Download it today!

@giffmana This reminds me of the XLNet paper: arxiv.org/abs/1906.08237

Amazing! Truly open review, through which we all gained more insights, i love it! Result: in multi epoch setting, making AR learn multiple orderings ~closes the gap to diffusion, explaining much of the difference. How the truly open review happened (from my vague memory): Mihir posted paper on diffusion being more sample efficient than AR and benefit from more epochs. Hypothesis in paper and from most commenters was due to augs, but no experiment to check this hypothesis in paper. After some discussion (on his thread, here on x) about this and what experiment might check this, including a first negative result Mihir mentioned and tried, we converged towards thinking it might be ordering instead. I suggested an experiment to check, but it had some issues. @YouJiacheng chimed in with connections to another paper, based on which we got the idea of how to make the experiment perfect. Then Mihir ran it, and it now looks like we have reasonably conclusive evidence PLUS potentially more confidence in a method to make AR better at multi-epoch. All in the open in realtime here on x dot com the everything app™

Unmute meets Moshi 🫂💖 Talk to unmute.sh!

Today, we release our 🇫🇷 to 🇬🇧 simultaneous speech-to-speech translation system, called Hibiki. It runs on-device & the model, inference code and tech report are available. This is built using the same audio LLM as Moshi, showing its versatility. 🟢

Excited to release a preview of Helium-1, our 2B LLM targeting edge and mobile devices. 🚀 More to come in the future: training code, support for more languages, data pipeline, tech report & more… 🟢

Meet Helium-1 preview, our 2B multi-lingual LLM, targeting edge and mobile devices, released under a CC-BY license. Start building with it today! huggingface.co/kyutai/helium-…

Local voice models FTW 🚀

Moshi goes to #ICML2024 in Vienna! Try the demo at moshi.chat

I am at ICML in Vienna! Let me know if you want to chat about (or to) Moshi, multimodal LLMs, Kyutai & more.

@soumithchintala @kyutai_labs Thank you Soumith!

this is amazing, but more importantly, liberating! @kyutai_labs leading the charge on real time voice assistants and as a true open-science non-profit, will release the code and details. congrats to a bunch of my former FAIR colleagues at Kyutai on the launch!

@Thom_Wolf @kyutai_labs Thanks Thomas!

The @kyutai_labs fully end-to-end audio model demo of today is a huge deal that many people missed in the room Mostly irrelevant are the facts that: - they come a few week after OpenAI ChatGPT-4o - the demo was less polished than the 4o one (in terms of voice quality, voice timing…) Relevant: - the model training pipeline and model archi are simple and hugely scalable, with a tiny 8+ people team like Kyutai building it in 4 months. Synthetic data is a huge enabler here - laser focus on local devices: Moshi will soon be everywhere. Frontier model builders have low incentive to let you run smaller models locally (price per token…) but non-profits like Kyutai have very different incentives. The Moshi demo is already online while the OpenAI 4o one is still in limbo. - going under 300 ms of latency while keeping Llama 8B or above quality of answers is a key enabler in terms of interactivity, it’s game changing, This feeling when the model answer your question before you even finished asking is quite crazy or when you interrupt the model while it’s talking and it react… Predictive coding in a model, instantly updated model of what you’re about to say... Basically they nailed the fundamentals. It’s here. This interactive voice tech will be everywhere. It will soon be an obvious commodity.

@giffmana Thanks Lucas!

Kyutai Moshi - first real-time Audio LLM. Basically no delay - the LLM even interrupted the speaker a few times. It was actually a bit eager to answer very quick. :) All to be open-sourced. Quality still a bit robotic though, but ok for v1. Pretty cool overall, congrats!

Looking forward to discuss open research at @kyutai_labs. If you want to work on large scale multimodal LLMs, come and talk to us, this is what we look like 👇☕️

✈️ I will be attending #NeurIPS2023: let me know if you want to chat about the future of LLMs, and how to democratize them. 🌐 We are also hiring members of technical staff and interns @kyutai_labs. Happy to talk about the lab and our mission.

@ylecun Merci Yann !

@EXGRV Félicitations !

/kyutai has landed! Super excited to build this new research lab. Pure focus on research. As open as it gets.

@soumithchintala Thanks for the kind words Soumith! Really excited by this new lab.

a great new open-science lab out of Paris, with a solid amount of initial funding! A very strong talent bench. i had the privilege of working with most of them before, and they're awesome researchers and great human beings!

@abacaj Yes, we have a couple of papers on that exact topic with @gizacard and @PSH_Lewis. Combining these advances lead to the Atlas language model (paper: arxiv.org/abs/2208.03299, code: github.com/facebookresear…).

Anyone have success fine tuning models with retrieval? Fine tuning the model to answer the question based on the context, trying this for code will see how it goes

@francoisfleuret @elonmusk The posters definitely deserve more than 6 likes (or 28 now)!

That I get 500 likes for criticizing @elonmusk and 6 for this masterpiece is quite sad.

@yoavgo The idea is cute, but I would not take the experimental results too seriously as the baseline numbers seem to be off.

so, two cents on the gzip classification thing: apparently the idea has been around in some form of another for a while. but was treated as a curiosity because how inefficient it was compared to all other classification methods. enter BERT.