Kilian Haefeli retweetledi
How to scale the batch size in LLM pretraining?
New paper on a scaling law that splits token budget D into training steps and batch size.
💺 arxiv.org/abs/2607.01487
English
Kilian Haefeli
304 posts
Kilian Haefeli
@khshind
pretraining @cohere & master @ETH
Zurich, Switzerland Katılım Ağustos 2012
888 Takip Edilen675 Takipçiler
Kilian Haefeli retweetledi
You know what model won’t reject your requests?
Cohere North Mini Code
English
Kilian Haefeli retweetledi
Kilian Haefeli retweetledi
@kimbochen yeah imo this is why and have a great latency right? previous cohere model had this as well btw iirc
English
interesting open model by cohere with lots of unusual architecture choices, here is a recap:
> parallel transformer, so MoE and attention are computed in parallel. likely doing some kind of MLP/attention disaggregation here?
> lots of query heads, query total dim is 4x hidden size
> big shared expert, 4x router size
> no scaling after normalization of the top k
> LayerNorm instead of RMS norm
> 32 layer only, no dense layer at the start
Cohere@cohere
Introducing: Cohere Command A+ We’ve created our most powerful LLM yet, optimized it to run on as little hardware as possible, and released it open-source for all.
English
Kilian Haefeli retweetledi
Kilian Haefeli retweetledi
just your friendly reminder to throw away any RL paper that only tests their method on Qwen models :)
English
@SemiAnalysis_ @dylan522p your social media intern is catfishing again
English
Hi i'm dwarkesh! Grew up all over the US, now sf-based and always down to nerd out about AI, science & history :) a lil about me:
🟠 Host of the dwarkesh podcast
🟠 Studied at UT Austin
🟠 Just published a book on the history of AI scaling
Lets grab coffee or do a fun activity this summer
English
Kilian Haefeli retweetledi
Ever wondered how Speculative Decoding interacts with production MoE models?
Conventional wisdom: MoE + speculative decoding = too many experts to load, gains disappear.
Reality: MoE amplifies speculative decoding.
Checkout Cohere Blogpost: cohere.com/blog/mixture-o…
English
Kilian Haefeli retweetledi
I work on the JAX team. If you're new to the field ignore this bait. The things you should focus on are understanding the math and how to program accelerators - really master your hardware and your methods. We try to make a great tool but don't obsess over tools early.
François Chollet@fchollet
When looking at deep learning profiles, one of the most obvious tells between a mediocre and great candidate is whether they list PyTorch or JAX.
English
Kilian Haefeli retweetledi
🤯 big update to our flow map language models paper!
we believe this is the future of non-autoregressive text generation.
read about it in the blog: one-step-lm.github.io/blog/
full details in the paper: arxiv.org/abs/2602.16813
we introduce a new class of continuous flow-based language models and distill them into their corresponding flow map for one-step text generation.
we beat all discrete diffusion baselines at ~8x speed!
v2 gives a complete theory of the flow map over discrete data, with three equivalent ways to learn it (semigroup, lagrangian, eulerian). it turns out you can train these with cross-entropy objectives that look very similar to standard discrete diffusion — but without the factorization error that kills discrete methods at few steps.
beyond improving results across the board, we showcase properties that are unique to continuous flows. in particular, inference-time steering and guidance become straightforward. autoguidance brings generative perplexity down to 51.6 on LM1B, while discrete baselines completely collapse at the same guidance scale.
we also show reward-guided generation for steering topic, sentiment, grammaticality, and safety at inference time — and it works even at 1-2 steps with our flow map model. simple, well-understood techniques from continuous flows just work incredibly well in practice for language.
we’re extremely excited about the future of this class of models.
stay tuned for results on scaling, reasoning, and reinforcement learning-based fine-tuning. 🚀
English
Awesome deepdive by kimbo!!
Kimbo @ ICML26@kimbochen
Previously when writing the NVIDIA Tensor Core evolution article, I realized how scarce the info about Blackwell micro architecture is So I set off doing this to bridge the gap. Super thankful for my colleagues and collaborators Thank you @snowclipsed for reviewing too
English
@giffmana @LLMenjoyer I recently found out heidi was made by the ghibli founders in jp and never officially aired in swiss tv
English
@LLMenjoyer Isn't anime Japanese and "cartoon" the more general term?
English
but.. but… budt…. heidi is anime too..! wht are u saying..
Lucas Beyer (bl16)@giffmana
Hey anime profile pic cracked cudamode hackers... Time to swap your profile pic to Heidi!
English
CP + EP for the win.
Am I right to say the community is generally and gradually dropping TP?
Cursor@cursor_ai
We're releasing a technical report describing how Composer 2 was trained.
English
Kilian Haefeli retweetledi
Join us at @CVPR in Denver for a full-day tutorial about Analytic Understanding of Diffusion Models.
The training objective of diffusion models has a closed-form solution -- yet it only memorizes. How do real models generalize? We'll unpack this paradox and the emerging analytical theory behind it.
@yuancy @CScarvelis @MasonKamb @WangBinxu @vincesitzmann @JustinMSolomon @SuryaGanguli
English
Kilian Haefeli retweetledi
Optimization theory for adaptive methods actually predicts most of what we know about hyperparameter scaling in LLM pretraining, and suggests new strategies as well. We did a deep dive here.
English
Kilian Haefeli retweetledi
You may think discrete distillation is fundamentally flawed, you are (surprisingly) wrong. 🤯
Meet Discrete Moment Distillation (D-MMD). It is a new method that brings fast, few-step sampling to discrete diffusion models! 🧵👇
English
Kilian Haefeli retweetledi
ZurichNLP#20 is on April 1st at the @ETH_AI_Center!
Fabian Schaipp (Inria) on recent trends in training algorithms for ML and Valentina Pyatkin (Allen Institute, ETH) on lessons from training open-source LLMs.
RSVP below before spots run out!
English
banger of a course by one and only Onur Mutlu
I made the mistake of not taking comp arch in my undergraduate so I was grinding through these lectures for an entire summer before grad school
kapilansh@kapilansh_twt
Apple's M4 chip engineers earn $450k+ Intel's CPU architects earn $400k+ NVIDIA's hardware engineers power every AI model on the planet They all understand one thing almost no software dev ever studies: How a computer actually works at the hardware level "ETH Zurich – Digital Design & Computer Architecture" by Onur Mutlu Free on YouTube. 30+ full lectures. Spring 2023 By a professor who holds joint appointments at both ETH Zurich and Carnegie Mellon Starts from a single transistor. Ends with a complete CPU you understand entirely: • Logic gates – how electricity becomes computation at the most fundamental level • Instruction Set Architecture – the contract between software and hardware every dev ignores • Pipelining – how your CPU executes multiple instructions simultaneously without you knowing • Out-of-order execution – why your CPU secretly reorders your code to run faster • Memory hierarchy – the design decision that determines the speed of every program ever written Every line of code you've ever written ran on hardware you don't understand The engineers who built that hardware earn $450k Now you know where to start
English