rohan anil

It turns out multi step backpropaganda is better. paper has a beautiful way of improving backpropagation. One iteration cleanly gets us backprop, multiple iterations get us a preconditioned update.

@BruXaPe @invincible4587 I still remember the goal from Roberto Carlos from the halfway line.

@invincible4587 @_arohan_ Isso te diz alguma coisa? 🤷🏻‍♀️

Today I learnt that Erling Haaland can only score if someone passes him the ball.

@0xaltyni Yes, it’s a question of where to put computation. Scaling residual nets which are ensemble of shallow representations gets us really amazing results, but we don’t yet the counterfactual of having models with deeper representations yet.

@_arohan_ You still believe, optimization > scaling?

A fun story: This was my first ever research paper that I lead the work. Idea was to accelerate distributed training at its limits via communicating information between models. You could accelerate close to 2x by adding 2 models trained on different subset of data. The work itself had been very successful for Search and Ads pipelines at google before it. I wanted to goto ICLR to present in Vancouver and my visa was rejected. BUT, the rejection notice had someone elses name on it. And the system was not designed to handle this. Geoff Hinton tried to help but was adviced to stay out, and after 1 month of trying, finally they had figured out what was wrong and told me that my original application was approved but had 30 day expiry on stamping. I had one day left, and original process involved sending the passport to one location and the application fee cheque to another location in another state. So I found the cheapest flight to LA at 5am and got a day hotel, and submitted my passport directly at the embassy on the last day of expiry when it opened and finally got the visa. However, I largely stopped going to conferences relatively afterwards given the amount of energy it takes to get a visa.

@Tex__El @zoidz00 🤣

@zoidz00 @_arohan_ didn’t realize a joke needed a disclaimer lol

@SaraCsit Plausible.

@_arohan_ All of the above - extra weight to middle one 🙄

Other fun story is that this was the first time common crawl landed in Brain and was like 1T tokens — I remember Noam cornering me in MK and saying we should train a 1T model on it. this is back in 2018 - I was very production pilled and thought Noam is a bit ood, and we should instead work on optimization methods.

One of my research goal since then has been trying to add N models and get close to Nx speedup in training. There are few fundamental questions to ask: 1. Why does distillation loses help? 2. How do we communicate more information that is identifiable between the networks? 3. How can each network implement a way to match the output of another network, while having its own beliefs on each prediction?

@gaunernst @marksaroufim @drisspg A Jacobi impl would be a banger. Eternalize it for ever

@_arohan_ @marksaroufim @drisspg If only i have time to do the eigen and the next problem 😭

I am one of the shortest person at core auto and I am 5’ 8.

@gaunernst @marksaroufim @drisspg However though you tower over them with most beautiful implementation of QR kernel.

@_arohan_ I thought I'm tall until I met @marksaroufim and @drisspg

@marksaroufim Ah thats because I have deceptively long neck

@_arohan_ Bro you’re at best 5’7’’

AlphaEvolve now available on GoogleCloud, and as all the customer use-cases below suggest, it is pretty transformative.

There is always cheaper memory.

Core kebab