Luc Georges

we've been pushing commits to transformers discretely, time to talk about we've been cooking the last few months: ⚡️ Continuous Batching is in transformers ⚡️ this will simplify, most notably, evaluation and your training loop: no need for extra dependencies or infra to get fast inference, and no need for convoluted code to update your weights note that speed is currently not on par with the best inference frameworks and servers out there and probably never will be the goal is *not* to become as fast: we want to complement the existing landscape with features like these, aiming for transformers to be the toolbox for tinkering with and building models

@eliebakouch 🥺🫶🔊🔊🐐

today is my last day at hugging face feeling really grateful to have worked with such an amazing team and learned so much along the way. i’m proud of what we accomplished together, especially the smollm series. building that project from scratch, putting so much into it, and getting to iterate on a model and training recipe that pushed the frontier for its size was really rewarding i hope i was able to play a part in making model training more accessible and in pushing the open model ecosystem forward. i’m also very thankful to hf for giving me the chance to share my passion for llm research, especially here, and to connect with so many awesome people things can get quite intense in this field, but i’m still very excited about the next challenges and about the good this technology can do but first, taking a few weeks break :)

I seem to have found somewhat of a sweet spot. Talk into Claude for the ideation phase, write down the plan, and do everything by hand myself, apart from tests maybe, who likes writing test amiright I question / rework / ignore everything written in plan as it often misses the target, but it does help me think through the problem in great detail. I go from one big plan to smaller in depth plans for each substep which works quite nicely. Co-ideating with Claude keeps the fun alive imo, so long you ask it to tweak / give feedback on your original ideas and have vision for what you want to do. It kind of feels like pair programming!

@ggerganov Incredible news, let's go 🔥🔥

Today ggml.ai joins Hugging Face Together we will continue to build ggml, make llama.cpp more accessible and empower the open-source community. Our joint mission is to make local AI easy and efficient to use by everyone on their own hardware.

@XciD_ @huggingface Claude climbing the echelons of HF leadership was not on my bingo card

Finally updated the org chart. Yes, Claude gets a @huggingface.co email. No, we're not discussing their compensation.

Transformers v5's FINAL, stable release is out 🔥 Transformers' biggest release. The big Ws of this release: - Performance, especially for MoE (6x-11x speedups) - No more slow/fast tokenizers -> way simpler API, explicit backends, better performance - dynamic weight loading: way faster, and enabling: MoE now working w/ {quants, tp, peft, ...} We have a migration guide on the main branch; please take a look at it in case you run into issues. Come in our GH issues if you still do after reading it 😀

@steeve you clearly recognise the dance moves lol

Okay that one was worth it

safetensors save_file on mac go brrrr ⚡️ been working hard these last few weeks on trying to make safetensors loading & writing faster found that skipping the os page cache with `F_NOCACHE` for write operations yields about 30% speed improvement more coming up, stay tuned

@art_zucker @jaredpalmer @github or in any huggingface repo 🥹

@jaredpalmer @github Would be happy to try it with Hugging Face on github.com/huggingface/tr…!

We are sending out a proposal for Stacked Diffs on @GitHub to trusted design partners to gather initial feedback over the next few days. From there we’ll iterate and share the gameplan

@silasmarvin2 Well I think there are multiple things happening at once. I wouldn’t say the loop is “hot” per se, maybe ~160 calls I think the issue is that the ok_or call was chained to a `&PyBound<PyDict>::get_item` call, in a context where the GIL wasn’t released (no allow_threads)

@LucSGeorges Oh wild! Did you figure out why it added such a large overhead? Was it just in a tiny hot loop?

fun 🦀 fact: you can nuke performance with a misplaced `ok_or`

@nicolas_clark2 Yes sir

🥺🥺

@remi_or_ 👁️👄👁️

thanks @LucSGeorges for the idea of this graph

this is what it looks like when you query an llm api with 500 requests each white pixel is an actual token, each black pixel is padding the issue is not that you send too many requests. it's that they are decoding for too long

@FerdinandMom 🥔

i love potatoes

did I tell you v5 was out?

@xenovacom 😂

that's a crap-ton of contributors

@remi_or_ ✨✨

So humbled to have had the chance to contribute to this amazing release 🚀 V5 of transformers is out, check it out NOW 🤗

LESSSGOOOO kudos team, incredible work everyone 🔥🔥🔥

@LucSGeorges Letting you know I am stealing this

when people ask me how HF makes money