Rajat V D

New blog post on Understanding the Neural Tangent Kernel: rajatvd.github.io/NTK Find out how this visualization depicts the training dynamics of neural networks by reading the post!

@BaigYasa That's crazy high praise Yasa, super appreciate it :)

This was the single best math talk I’ve EVER attended. The animations are 3B1B levels (highest praise I can give) and the actual work is also really exciting. Never more mogged in my life by our fresh Doctor @rajat_vd

I defended my PhD last week! The main focus of the talk was my recent work on fast sketching on GPUs via co-design, FlashSketch (arxiv.org/abs/2602.06071, ICML 2026 Spotlight). Recording: youtube.com/watch?v=9voMsL… Step through the animated slides: eigentales.com/talks/flashske…

@thsottiaux Let codex models access memory and personalization from my GPT account. Also allow it to interact with an existing conversation/context in chatgpt. I find myself asking 5.1 pro/5.1 thinking to generate instructions to feed to codex after discussing ideas in the chatgpt interface.

Codex team is working on a few experimental projects that are starting to shape up and I’m excited to share more about soon. But I’m curious, what would you like to see ship or improved by the end of the year other than better models?

1/10 ML can solve PDEs – but precision🔬is still a challenge. Towards high-precision methods for scientific problems, we introduce BWLer 🎳, a new architecture for physics-informed learning achieving (near-)machine-precision (up to 10⁻¹² RMSE) on benchmark PDEs. 🧵How it works:

five years ago, when I was struggling to get a grasp of NTK, I remember coming across @rajat_vd 's blog which is when it all properly clicked in my head.

most foundational concept in deep learning that no one understands is probably the Neural Tangent Kernel (NTK) this line of work studies neural networks of *infinite width*, which explain a lot about normal finite-width NNs and there is exactly one Very Good blog post on them:

New blog post on Floating Point numbers: eigentales.com/Floating-Point/ I highlight a perspective and intuition about floating point that I haven't seen emphasized before. TL;DR: Floating point is fixed point in log space!

just finally grokked how NNs are basically gaussian processes

@clcrozier @jeremyphoward For animations that aren't too long, this package might be helpful in the future: github.com/rajatvd/gifify

I spent a humiliating amount of time learning how to make animated graphs, just to illustrate a fairly obvious point. “Forecasting s-curves is hard” My views on why carefully following daily figures is unlikely to provide insight. constancecrozier.com/2020/04/16/for…

@deliprao Reveal.js?

I want to look beyond Google slides for talks. What are the cool kids using these days? Beamer people will be reported.

In 2017, I did something dumb. There was a total solar eclipse passing through the US, but I was in NY, far from the path. Was I really gonna get on a plane to see a cool thing for two minutes? Nah. I had shit to do. The day came. I put my stupid glasses on and saw the sun become a little less big and then back to being full size again. It was mildly interesting. Then the reports started coming in from the people who had seen it from the totality zone. People were like "it was an indescribably profound, perspective-shifting, life-altering experience." They were like "I'm a different person now, someone who can only be understood by other people who saw the total solar eclipse." It was massively fomo-y and upsetting. Solar eclipses happen when the moon passes directly through the invisible line connecting the sun and Earth, something that doesn't happen very often because space is big and the invisible line is skinny. Earthlings are very lucky, eclipse-wise. Most planets don't have a big enough moon to create a total solar eclipse. Not only is our moon big enough, it's about exactly the size of the sun in our night sky because, by sheer coincidence, the sun is about 400 times farther from us than the moon and also about 400 times bigger than the moon in diameter—making our eclipses especially breathtaking. There are about 70 total solar eclipses every century, each resulting in a thin path of total sun blockage. For most of history, there was no way to know when or where they would happen. Only the very lucky few who happened to be in the right place, at the right time, with the right weather, got to experience a total solar eclipse. Today you can ensure that you see one—and I had passed up the opportunity. I would not make the mistake again. I went online and learned that there would be another total solar eclipse passing through the US in 2024, and then that would be it until 2045. It went on my calendar that day. As fortune would have it, the 2024 eclipse's path of magical totality would be passing right over my new hometown of Austin, Texas. It was perfect. Then came the weather reports. Austin was going to be cloudy on eclipse day. Nope. Not okay. It wasn't an option to not see this eclipse. My friend @Liv_Boeree was equally psychotic about this, so we decided on Sunday night that Monday morning we'd get on a flight to somewhere in the eclipse's path that was forecast to have clear skies. We settled on Arkansas. Early the next morning we flew to Little Rock, got in a car, and drove northwest to get to the dead center of the totality path, where the total eclipse would last for more than three minutes. We ended up in a big open rural field that may or may not have been part of someone's farm. It was us and some cows. The sky was perfectly clear. 30 minutes until totality. I looked through my glasses at a crescent sun. It seemed a little dimmer out than usual, but only a little. 20 minutes. Thinner crescent, a tad dim, maybe a tad cooler than it was before? 10 minutes. Razor thin crescent now, definitely weird lighting. Because all of the light is coming from one small area, shadows are very sharp. You can see the shadow of individual hairs on your head. 1 minute. It's very dim, like early evening, but still feels like daytime generally. Waves of light and dark ripple across the ground, like the way light moves at the bottom of a swimming pool. 5 seconds. Diamond ring! I take off my glasses and the diamond ring looks strikingly beautiful and strange (google "eclipse diamond ring" to see what I'm talking about). 4 seconds to 1 second. The Earth's dimmer switch suddenly goes downnnnn as dim daylight drops into night. Totality. Imagine a world where there was always cloud cover, and one day every few years, in certain places, the sky cleared at night, and you could see stars for the first time in your life. It would be a totally surreal experience, something that reminded you that you don't live in a big world but on the edge of a tiny rock in vast outer space. It would show you the truth about reality. We see stars all the time, so we're well-acquainted with our reality living in outer space (even if it's easy to forget during the day). But when I looked up at the sky during the total eclipse, it was the first time I had experienced another, totally different way to see with my eyes that I lived in outer space. I saw one sphere positioned in front of another sphere, with two other spheres—Venus and Jupiter—floating nearby. More than ever before, it felt obvious that I was standing on the edge of a fifth sphere. For the first time in my life, I was looking at the Solar System. I looked around. There was a dim 360° sunset along the entire horizon—another first. It was dark. At 2pm. By a minute in, there was a chorus of chirping crickets that hadn't been there before. Birds were flying around overhead that hadn't been there before. The cows continued being cows but I assume they were super confused. I looked back up at the Solar System and noticed a little imperfection on the edge of the black moon circle, which I realized must be a solar flare. A solar flare I could see with my naked eye. Only half of my brain was focused on the eclipse because the other half was frantically trying to figure out how to best use the precious three minutes. I told myself I wouldn't spend more than 30 seconds of the three minutes doing stuff with my camera, but I was not gonna not take pics. I got things all focused and snapped this gem: Just kidding that's @AJamesMcCarthy's photo. But mine was pretty good too. Okay fine that one was taken by @NASA. Here's mine. Whatever. Anyway, eventually it ended. The glorious diamond ring reappeared, followed by me being blinded before remembering to not look at the sun anymore. Earth's dimmer switch swooped back up, as night turned to day in a few surreal seconds. It was over. Thoughts were processed. Emotions were felt. It was very very VERY worth the last second trip. For any of you who pulled a 2017 Tim and decided not to see this one, I hope I've sufficiently fomo'd you into making sure you see this for yourself, sometime soon, in some part of the world. 🌻

@sp_monte_carlo That's helpful, thanks!

@sp_monte_carlo Can you elaborate a bit on what these practical conditions look like? And how the analogous practical optimization conditions might look like?

@gabrielpeyre Can you elaborate a bit on what you mean by random linear section? Are the contour plots || x A + y B || for x,y >= 0 and A and B psd?

The cone of positive semi-definite matrices is a fundamental object of convex analysis and optimization. One can encode or approximate convex constraints as linear sections of this cone. en.wikipedia.org/wiki/Definiten…

@amanrsanger This looks awesome. What languages does it support, specifically does it work with latex? Also any chance we see a vim/neovim extension for this sometime in the near future?

Introducing Copilot++: The first and only copilot that suggests edits to your code:

# on technical accessibility One interesting observation I think back to often: - when I first published the micrograd repo, it got some traction on GitHub but then somewhat stagnated and it didn't seem that people cared much. - then I made the video building it from scratch, and the repo immediately went through hockey stick growth and became a verty often cited reference for people learning backpropagation. This was interesting because the micrograd code itself didn't change at all and it was up on GitHub for many months before, stagnating. The code made sense to me (because I wrote it), it was only ~200 lines of code, it was extensively commented in the .py files and in the Readme, so I thought surely it was clear and/or self-explanatory. I was very happy with myself about how minimal the code was for explaining backprop - it strips away a ton of complexity and just gets to the very heart of an autograd engine on one page of code. But others didn't seem to think so, so I just kind of brushed it off and moved on. Except it turned out that what stood in its way was "just" a matter of accessibility. When I made the video that built it and walked through it, it suddenly almost 100X'd the overall interest and engagement with that exact same piece of code. Not only from beginners in the field who needed the full intro and explanation, but even from more technical/expert friends, who I think could have understood it if they looked at it long enough, but were deterred by a barrier to entry. I think as technical people we have a strong bias to put up code or papers or the final thing and feel like things are mostly self-explanatory. It's there, and also it's commented, there is a Readme, so all is well, and if people don't engage then it's just because the thing is not good enough. But the reality is that there is still a large barrier to engage with your thing (even for other experts who might not feel like spending time/effort!), and you might be leaving somewhere 10-100X of the potential of that exact same piece of work on the table just because you haven't made it sufficiently accessible. TLDR: Step 1 build the thing. Step 2 build the ramp. 📈 Some voice in your head will tell you that this is not necessary, but it is wrong.

@solidangles Another cool perspective is that exponentials are eigenfunctions of the derivative operator. So, calculating the dot products is performing a change of basis that "diagonalizes" the derivative operator, which turns differential equations to algebraic equations.

I think it makes sense we'd test "alignment" against e^-st when doing differential equations because [1] they behave so well under differentiation and [2] they show up in all sorts of real-world applications. (6)

I THINK I GOT IT! I've been trying to understand this for years, but the missing magic words that made it finally click were "inner product." Thank you @math_vet! I'll try to explain my understanding in a short thread in case it helps anyone. 🧵 (1/8)

@TivadarDanka Really cool. This is very similar to lattice multiplication (en.m.wikipedia.org/wiki/Lattice_m…) which imo is the best method to multiply large numbers by hand.

The Japanese multiplication method makes everybody feel "I wish they taught math like this in school." It's not just a cute visual tool: it illuminates how and why long multiplication works. Here is the full story.