rohan anil

I think Google is very poised to do well on datacenter and energy management due to the fundamental advances they continue to make on core algorithms for planning and particularly decision making systems under uncertainty.

Google is now the first cloud provider to integrate 1 GW of flexible demand into long-term utility contracts. Our ability to shift or reduce our energy demand when it’s needed can help utility companies balance supply/ demand and plan for future capacity needs. This is a big milestone for responsible data center growth and helps keep costs lower for local communities. blog.google/innovation-and…

@Miles_Brundage Is the problem you are seeing model overthinks — Have you tried asking it to give you intermediate steps?

Did the heat wave do this to u, Claudey

Pls get Claude Code working reliably before shipping new features, pls pls pls

Soon self driving car everywhere, I know.

Writing code character by character has become equivalent to riding horses. You can instead learn to drive a car to get places. And occasionally do what people in middle ages did when claude is down.

@einsums @MillionInt I actually cannot disagree, sometimes do feel that way.

@MillionInt basically @_arohan_

AI labs need a wallfacer project. AI researcher not having to explain themselves to anyone. performing seemingly random actions with hidden inscrutable agenda to create a SOTA model in a way no one would deem possible

JUST IN: Nvidia CEO Jensen Huang calls on tech leaders to "be careful not to scare people" regarding AI.

New episode of the Information Bottleneck! We talked with @StefanoErmon about why he thinks diffusion LLMs will replace autoregressive ones. Stefano co-invented DDIM, FlashAttention, DPO, and score-based diffusion models. He's a Stanford professor and now runs @Inception_AI, where they built Mercury II. We go deep but also cover the bigger picture - the startup journey, PhD vs industry, and where AI is heading. A few things that stuck with me: - He thinks of autoregressive models as typewriters and diffusion models as editors. One goes left to right. The other starts messy and refines. - Mercury II (their text difussion model) already beats the fastest autoregressive models on latency-critical stuff as voice agents, code suggestions, anything where you have a tight time budget. And it does it because diffusion generates tokens in parallel instead of one at a time. - We also got into whether AI will actually replace software engineers (his answer: no), PhD vs industry advice, and what it was like going from an ICML best paper to raising money.

Perhaps I should get married again so that the media has a more recent man they can reference any time they mention me or my work.

At this nerdiest of all nerdy sessions 💞, Jeff Dean said he doesn’t think we’re running out of data. “I think there’s still an enormous amount of data in the world that we haven’t really used yet for training these models. We train on some video data, for example, but there’s a lot more video out there, along with associated audio, that we’re not necessarily making full use of yet. I also think real-world robotics data, and autonomous vehicle data, is going to be fairly plentiful. And then synthetic data is another resource. If you can generate really interesting, high-quality data, then you can effectively inject more compute and get more training data that way. Now, of course, there’s a reasonable question here: aren’t you eventually just regurgitating the same stuff? If you train on data, then use that model to generate synthetic data, are you just making another version of what you already had? Maybe to some extent. But I still think it can help, especially if the model generating the synthetic data is itself very powerful. At least so far, that does seem to be useful. And beyond that, there are also a lot of techniques we’re not really using much right now that used to be very common in other domains, like convolutional image models years ago. Things like data augmentation are interesting. That’s one way to think about synthetic data. Techniques to prevent overfitting are also interesting. You can use dropout, distillation, and other forms of regularization. So I think there’s still a lot of opportunity to make models better with more compute and more passes over the data, without necessarily running into overfitting.“ A fascinating conversation between @JeffDean and @BillDally @NVIDIAGTC

@yifan_zhang_ I liked it a lot - a cool title for this gem would be The Deep Remembers!

Oh, Rohan the Legend, liked it! Definitely worth reading!

Time is life’s most valuable resource.

This is funny! Although in hindsight I think we should give due credit to all the new works that improve on it and scale, think of about deploying in a real training run (solving for memory growth) An advantage Google had was that there was extremely strong folks left alone to think for a longer time thus able to ascend in creative directions like this.

lmao > google cooks paper, "meh its probably not gonna work, pass" > chinese lab cooks exact same thing one year later, everyone gets super hyped EVERY SINGLE TIME

Actually these ideas predate a lot earlier as well. But let’s celebrate all of the papers and the current results too 🤝 Memory 📈

“Timing is very important. You need to pick hard problems to solve and be ambitious with them. But you've also got to pick the right time when the world and the context that you're in is the right kind of environment for those ideas to flourish.” In his official Nobel Prize interview, Demis Hassabis discussed how his aspirations as a young gaming programmer were ahead of their time. Watch our official interview: bit.ly/41DGkXr

• According to the story, the dog's cancer has not been cured. • Absent all regulatory and manufacturing constraints, we could not just synthesize magic mRNA cancer cures. The technology is very promising, but it's not yet any kind of panacea. • The emergent system of regulators and manufacturers is indeed far too conservative, and small-scale experimentation is much harder than it should be. More people should read the first part of The Rise and Fall of Modern Medicine. Recommend @RuxandraTeslo, @PatrickHeizer for more.

Our co-founder Terence Tao is announcing SAIR Foundation's inaugural competition: the Mathematics Distillation Challenge. Co-organized by @damekdavis, Terence Tao, and SAIR Foundation. competition.sair.foundation/competitions/m…

@DarwinianVyas Memory prices go 📈

@_arohan_ If trends hold, Sonnet 4.5 is estimated to have near 100% retrieval at 1B context

Hold up, nvm. Continue generating