Assaf Ben Kish

Excited to share that Inspiration Seeds has received an Honorable Mention award this year at SIGGRAPH! 🎉 👉kfirgoldberg.github.io/InspirationSee… Huge thanks and congrats to the best team! @kfir99 @EladRichardson Looking forward to seeing you in LA in July!🌱

Your RL post-training may be sabotaging your LLM’s test-time scaling! Conventional RL pretends that you can collapse all reward signals *upfront* into a single *scalar reward*. We introduce Vector Policy Optimization (VPO), which natively maximizes *vector-valued* rewards, boosting test time search performance, even on the original scalar.

@MichaelHassid @HebrewU @royschwartzNLP @adiyossLC @AIatMeta Congrats Michael! 🎉

After a decade at HUJI (@HebrewU), I've submitted my PhD thesis! Huge thanks to my awsome advisors, @royschwartzNLP & @adiyossLC , my FAIR colleagues (@AIatMeta), lab mates, and of course - my wife (we met at HUJI!). Above all, thanks to my parents for teaching me how to learn.

Language is discrete. Language models don’t have to be. 🧚Introducing ELF🧚‍♀️: Embedded Language Flows—a class of diffusion models in continuous embedding space based on continuous-time Flow Matching 🧵

Most of the information in LLM weights is knowledge, not learning algorithms. The known learning algorithms that emerge (e.g. induction heads) trace back to just 2 attention heads (0.001% of GPT-3's parameters). It is plausible that the DNA encodes better learning algorithms.

@SunnySanyal9 @ShamKakade6 Can you explain how?

@ShamKakade6 I got the answer. It is a little different.

1/8 Introducing Recurrent Transformer (RT). At 300M params, RT improves validation CE over standard Transformers. The best RT model is only 6 layers, but wider at 2048 — beating deeper 12- and 24-layer Transformers by trading depth for width.

@a1zhang Longbench

does anyone know of any evals / benchmarks that are particularly sensitive to prompting? can be for any range of reasons, e.g. requires ICL, requires in context examples, is hard, has strict requirements, partial information, etc. ideally also not super long horizon or hard to set up

I am *very* excited to announce our SIGGRAPH 2026 workshop: Lines & Minds: Visual Abstraction in Art, Psychology, and Computer Graphics 🎨🧠🫖 🔗 lines-and-minds.github.io 📅 Sunday, July 19 Join us to explore how visual abstraction shapes how we think, create, and communicate.

Creative work often starts before we can describe what we're looking for. What role can generative models play at this stage? 🌱Our new work, Inspiration Seeds, reveals hidden visual connections between images, creating a purely visual exploration space. 🔗kfirgoldberg.github.io/InspirationSee…

good morning folks we're going live in about 2h to have a jolly discussion about making models self-teach themselves hard stuff like @justinskycak would

🚨new paper!🚨 RL makes LLMs smarter - but it also causes diversity collapse. Check out Multi-Answer RL - a method that trains LMs to capture and output a distribution of answers in a single generation 👀

Introducing Look Where It Matters — High-Resolution Crops Retrieval for Efficient VLMs. VLMs don't need to process full high-res images. AwaRes uses tool-calling to retrieve only the high-res regions needed to answer a given query🧵 arxiv.org/abs/2603.16932 nimrodshabtay.github.io/AwaRes/

@osieberling Very cool This learning profile correlates with how different layers communicate over the residual stream during the forward pass: transformer-circuits.pub/2021/framework…

A very interesting observation on backpropagation is that no matter how nonlinear the forward pass, once the forward pass is fixed, backpropagation itself is purely linear. This allows for all kinds of gradient analysis. For example, one can decompose the backward pass by the depth of the backpropagated signal: Each forward and backward pass can be viewed as involving 2^{2L} different paths, where L is the number of blocks (2L to account for Attn/MLP subblocks). Because the forward pass is nonlinear, we can’t just compute each of the paths separately to decompose the forward pass. However, for the backward pass we can. Of course, 2^{2L} is computationally intractable as we would need to do 2^{2L} separate backward passes (>1e19 for a 32-block transformer). But if we are only interested in the depth of gradients, we can use simple dynamic programming to decompose the backward signal by depth: Let x_l be the residual stream at depth l. Instead of just maintaining dL/dx_l as in normal backpropagation, we maintain (dL/dx_l)^k for each gradient depth k, i.e., a table consisting of 2L+1 gradients. Because backpropagation is linear, we have dL/dx_l = \sum_{k=0}^{2L} (dL/dx_l)^k. Then, after backpropagating each subblock, we can update our table of gradients with (dL/dx_l)^k = (dL/dx_{l+1})^k + (Jacobian_{subblock_l}(x_l))^T (dL/dx_{l+1})^{k-1}. This way, we can efficiently compute for each weight update how much comes from each gradient depth. Interestingly, because there are C(2L, k) (2L choose k) paths of depth k (left plot), by sheer number of paths we would expect the decomposition to concentrate around depth L (ignoring cancellations/correlations), but this is not what we observe in practice: The actual decomposition of gradients by depth is much more shifted towards shallower depths (right plot), which suggests that after normalizing by the number of paths, shallower paths contribute gradients of much larger magnitude than deeper paths do.

@vivek_2332 @karpathy Curious - After all the hyperparameter tuning on the eval set - Was the final model evaluated on a test set?

introducing autoresearch-rl, autonomous research for rl post-training. inspired by @karpathy autoresearch, and i think rl post-training is honestly one of the places where this idea fits perfectly. there are at least 50+ hyper parameters to tweak, learning rate, batch size, rollouts, clipping ratios, kl penalties, schedulers, the list goes on. instead of sitting there for hours turning knobs one at a time, just let the model figure out the right starting config on its own. some things worth mentioning: -> built on @PrimeIntellect prime-rl (my favourite rl post-training framework) and @willccbb verifiers for reward verification. -> ran qwen2.5-0.5b-instruct on gsm8k across 60+ autonomous experiments. eval score went from 0.475 to 0.550 and the agent actually found a way to do it in fewer steps (20 instead of 30). less compute, better results -> the whole thing was surprisingly smooth to set up and run. point the agent at the config, go to sleep, wake up to a full experiment log. i really wish i could try this on a bigger model but gpu poor for now lol -> the agent discovers things you wouldn't think to try. like how rollouts = 4 beats rollouts = 8, or how a constant lr schedule outperforms cosine. it just methodically tests everything i think the real value here is that rl training is so fragile and noisy that having an agent patiently run experiment after experiment is genuinely more effective than a human doing it manually. check it out: github.com/vivekvkashyap/…

🎙️ Introducing ID-LoRA: the first open-source model to jointly generate a video with a person's appearance and voice in a single pass from just a reference image + short audio clip. No more cascaded pipelines where the audio can't follow your prompt. youtu.be/6bWcMh18K6g?si…

Pre-pre-training LLMs on synthetic NCA trajectories is better than pre-pre-training on real language! Check out why below. Great job @Danicmhlee and @seungwookh!

Introducing Self-Distillation Fine-Tuning, a new approach to continual learning. SDFT distills a model’s own outputs when given expert demonstration in-context. The result is faster learning than off-policy distillation, combined with reduced forgetting. x.com/IdanShenfeld/s…

@N8Programs @lateinteraction Also, it is much more efficient to re-use neural circuits in later steps rather than "spreading copies" of them throughout depth

@N8Programs @lateinteraction Another issue with feed-forward models: for many p-hop reasoning task, the order of information encoding (in depth) matters. There will always be "adversarial" p-hop trajectories that aren't feasible in a single forward pass. aclanthology.org/2024.emnlp-mai…

Buried in the massive progress in LLMs over the past few years is how all your favorite Transformers/DNNs still can't solve even just grade school math problems above a "B" grade through a forward pass. Unless they're in a scaffold like CoT, ReAct, RLM, etc. And this is true even at trillions of params and bajillions of FLOPs. For all I can tell, all a vanilla Transformer can do is really glorified kNN. Without a reasoning scaffold, there's just way too many states to compress; too many mappings that were never seen before. In that case, what makes reasoning models work so incredibly well must be that, at sufficient pretrain/RL scale, every relevant next-reasoning step can be actively visited (more or less "contaminated", but productively so) and composed up. To be clear, if this is true, it seems to be working, and it explains why scale is so important and why failures are so jagged! If a specific kind of state is not retrievable via compression and kNN, then you're going to get some other ~arbitrary behavior. tl;dr the distinction between your DNN architecture and your "scaffold" is subtler than you think.

1/3 🎉 Our paper is accepted at #ICASSP2026! TL;DR: Speculative decoding struggles for speech because exact token matching is too strict: acoustically similar tokens get rejected. We propose a fix. machinelearning.apple.com/research/coars…