Pavan Jayasinha

I implemented an LLM end-to-end in hardware, and ran it on an FPGA. Zero Python. Zero CUDA. Just pure SysVerilog. All my progress + everything I learned from 200h of LLM chip design (demo at the end)👇

- 230 training runs - 1,623 GPU hours (67 B200 days) - 76 TB of training data - a 2x faster model Every paper said it can't be done. Quantization Aware Distillation made it possible.

@MajmudarAdam Soldidy supreme

"intention density" is behind the visceral difference between AI outputs that feel beautiful, human, designed vs. uninspired/slop it points at something much more specific than taste: how many distinct, willful decisions went into an output? how much of its structure can be attributed to intentionality vs. inevitability? when I watch a Ghibli film, I know that every detail and expression in every frame has been crafted with specific intent (Miyazaki personally drew/edited 80,000 of 144,000 frames in Princess Mononoke). I can feel the creator through the creation. in contrast, AI tools encourage work with far lower intention density by default. starting from a blank canvas, you're forced to confront thousands of micro decisions to create a final output. but now that you can write a one-sentence prompt and get a full app or video one-shot, all of these decisions get outsourced, often without you noticing they exist. there can still be high intention in the final work (ex: codex generated apps still feeling pretty good), but the source of this intention is "the way things are usually done" (coming from the model) rather than a particular vision or design. there's no reason AI output has to be like this though we can think of the creative process in 2 parts: 1. intention - what do you want to create? why? 2. execution - how do you create it? AI agents will clearly replace ~100% of the execution part of the creative process. they already have in software and will soon be in film/animation. as they shift up the chain and replace intention as well, creative output starts to feel more trite and indistinguishable. but for those who are careful to preserve and expand rather than offload their intentionality, they have more time than ever to focus on the details and create far more/better software, art, etc.

@MajmudarAdam Code and writing

@pavanjayasinha thanks dogz, in what context?

@MajmudarAdam Those two words themselves unlock much for me since ive been dancing around that concept without a good compressed ptr word for it.

@AliesTaha 🔥⚡️ a soldid bog piece of work

we quantized the best open-source diffusion model on the market 4 bits huge speedup (almost) no quality loss this is a full explanation of the trillion dollar industry's oldest trick

@hayou_soufiane @saagnikkk @lifan__yuan @dilekhakkanitur @haopeng_uiuc I see, weights are in BF16, gradients/optimizer states in fp32 is default in VERL. Relevant work you might be interested in: arxiv.org/pdf/2510.26788

@pavanjayasinha @saagnikkk @lifan__yuan @dilekhakkanitur @haopeng_uiuc yeah my original question is about precision, which in that figure I can only see 1e-5 tolerance. Is your bitwise comparison in FP16?

🚨New Blog Alert: Is AdamW an overkill for RLVR? We found that vanilla SGD is 1. As performant as AdamW, 2. 36x more parameter efficient naturally. (much more than a rank 1 lora) 🤯 Looks like a "free lunch". Maybe It’s time to rethink the optimizers for RLVR 🧵

@hayou_soufiane @saagnikkk @lifan__yuan @dilekhakkanitur @haopeng_uiuc exactly-zero means weight_after_update is bitwise identical to weight_before_update. Does not necessitate that the gradient is 0, nor even the update is bitwise 0.

@saagnikkk @pavanjayasinha @lifan__yuan @dilekhakkanitur @haopeng_uiuc So "Exactly-Zero Updates" means the gradient is within 1e-5 distance from 0?

Long live SGD! Surprising to me that LR=0.1 (!!) with zero momentum worked so well. We invite you to ponder: what would an RL-native PEFT algorithm that turns RLVR’s parameter efficiency into compute efficiency look like? 🤔

Literally who’s Adam

@josancamon19 @saagnikkk @lifan__yuan @dilekhakkanitur @haopeng_uiuc It is 36x more param efficient in the sense that it achieves same validation performance while modifying 36x less parameters. Re compute efficiency - we observed SGD GRPO to be faster than Adam GRPO but not by a lot (expected). The highlight here is instead memory efficiency.

@saagnikkk @pavanjayasinha @lifan__yuan @dilekhakkanitur @haopeng_uiuc 2nd point is bait? Had to think for a second if it was more computer efficient, no, right?

@sanepplichar @hyhieu226 I see 😂

@pavanjayasinha @hyhieu226 Thats the joke sir…

Yeah. The nuance of that story is even more impressive. Not only did Google not use any NVIDIA GPUs, they also completely avoided CUDA. I bet your favorite tech bros didn't tell you that.

@Almondgodd Super Soldidy

I spent the past year building AI for robots at Tesla Optimus and Dyna Now I'm introducing ANDROID DREAMS: an essay of my predictions for the next 20 years of robotics, inspired by Situational Awareness and AI2027. I predict EGI by 2031 and more robots than humans by 2045👇🏼

@edenchan @omkizzy @omkizzy 🐐

@pavanjayasinha management engineering is tuff look at @omkizzy

Ngl it should be rated lower

@itsclivetime ❤️

Really happy to be announcing the chips we’ve been cooking the past 18 months! OpenAI kicked off the reasoning wave with o1, but months before that we’d already started designing a chip tuned precisely for reasoning inference of OpenAI models. In January 2024, I joined OpenAI as a hybrid gpu programmer & custom chip designer, the first IC on an oddly positioned hardware team that hadn’t yet committed to the idea of custom chips. These past 21 months I’m so lucky to have gotten the chance to learn from this incredibly talented and tiny team, accelerated by tight codesign with our ML team, Broadcom, and a few really cool new AI tools ;) Now we’re 9 months away from what is I think the fastest and largest volume ramp of any first time chip. Looking forward to pushing the cost and latency of intelligence to zero.