Hen Davidov

Recently accepted to ICML 2026: Knowing when to quit. We refuse to complete failing LLM generations as they unfold, token by token, instead of judging the prompt up front or scanning the final output. The method utilizes a 2-layer probe to estimate expected correctness.

@JoshPurtell Wait but if they use the value function (critic) trained on each completed trajectory verified reward, what difference does it make the split into subtraces? Am I missing a prm here?

Valhalla! OSS is liberated from the GRPO grift at last No hate to DeepSeek for innovating on it as a solid baseline for math. But, good riddance

@saranormous See ya there!

@yoavgo קצת יותר נמוך מהסיכוי לחנינה

מה הסיכוי שהוא מתפטר?

@ReddyAarya36823 @noclearnogga Lol dope

Hen, first year PhD, pretraining an LLM with academia compute for the first time. Wish me luck.

@redtachyon The Fable nerf came right after the first tweet

@hendav136 What?

Someone at anthropic saw my tweet I guess? Can't explain the timing any other way

@ThakurAryancha4 Best of luck with that!

@hendav136 Good luck dude I am also currently doing patent research in Ai specially edge ai

@ThakurAryancha4 Starting with a tiny gpt3-esque 100m param, and building up on the positive signals

@hendav136 Which base model ?

@noclearnogga Let's start by changing your profile pic from a dude with a swastica on his forehead?

@hendav136 bro, im studying Physics but considering switching to CS, which i love, but I'm worried AI may automate much of what a CS degree teaches. My interests are AI theory, robotics and maybe quantum computing. If you were starting today, what would you do? I'd like working in deep tech

@mesken_stefan Thank you!!

@hendav136 You got this 💪

@ruth_rallph Thank you!

@hendav136 All the best Hen!

@JerrryKun I have a transformers arch bet in mind, hoping to make a dent in this possibly saturated line of work

@hendav136 🫡 (pretraining for what?

@gradascetic Less than you'd expect. Compute is kinda siloed off in different labs. I mainly use my lab's gpus

@hendav136 How much compute does ox stats have now?

@ZacharyBamberg1 Thank you! Right now I'm just working off of a llama style recipe, iterating on my arch bet

@hendav136 Happy to compare notes if you’re feeling confident nonetheless 😅 (It helps to pre-train decoder-only models with existing kernels — avoid pre-training encoder-decoders like the plague)

@yoav_gelberg I'm not in the kindergarten bro lets be real

@hendav136 Don’t forget to qk norm

@Besteuler Thank you!

@hendav136 You can check Appendix C in our paper. The actual cost depends on whether you use 2nd momentum in Pion. Without it, Pion will be more efficient than both Muon and AdamW.

🚀 We are very excited to introduce Pion — a spectrum-preserving optimizer for LLM training. Pion shows strong training stability in practice. This is the project that we have been working on since POET. The central idea is to turn POET/POET-X (spherelab.ai/poet; spherelab.ai/poetx) into an easy-to-use optimizer. Instead of additive updates like Adam & Muon, Pion takes a different route by updating each weight matrix via coupled left & right orthogonal transformations, keeping its weight spectrum stable throughout training. This different update mechanism is directly inspired by the empirical effectiveness of Orthogonal Finetuning (oft.wyliu.com; boft.wyliu.com) and POET/POET-X, with roots tracing back to minimum-energy training methods such as MHE (arxiv.org/abs/1805.09298) and OPT (opt-training.github.io). Pion's key features: ✨ Competitive on LLM pretraining, SFT & RLVR ✨ μP-compatible by construction ✨ Stabily trains ultra-deep LLMs and even normalization-free LLMs, where AdamW & Muon diverge 🌐 Project: spherelab.ai/pion 📜 Paper: arxiv.org/abs/2605.12492

God I love science sometimes