ayush

I added KV caching and INT8 KV quantization to our transformer inference, improving throughput by 35x. All of this was done from scratch in Rust + CUDA, on top of a homemade ML framework. On a 4-token prompt with 252 generated tokens: - Original: 0.76 tok/s - KV cache fp32: 27.21 tok/s - KV cache int8 (quantized): 27.29 tok/s Try it out yourself here: mni-ml.github.io/demos/kv-cache/ In practice: - KV caching gave us about a 35x end-to-end speedup - INT8 KV cache kept roughly the same speed as fp32 but cut KV cache memory by 3.78x FP32 cache used 4.5 MB in this run while the INT8 cache used only 1.19 MB This simple change to inference created a huge impact on performance. To learn more about the KV cache and other optimizations like this, check out the blog at mni.ml!

@_reesechong @MankyDankyBanky @srush_nlp my roommate 🤞

Behind the scenes of mni-ml: January 4th 2026 - my roommate @MankyDankyBanky and I wanted to do a big project together. ”maybe we should try to build pytorch from scratch” We found @srush_nlp's minitorch curriculum and committed to grinding through it Jan to April. February - autodiff and tensor internals done. lots of late night PR reviews, stacked diffs, Kinton ramen runs to Toronto when I'd visit Aadi at Shopify. We started posting on X to keep ourselves accountable. March - the month of parallelization: Aadi shipped tiled matmul using the same algo @nvidia teaches in their CUDA guide, wrapped by end of month - pooling, conv1d/2d forward+backward, softmax, dropout. March 22-23 — @socraticainfo symposium & we see the tinytpu team on the stage which filled us with determination 🫡 cc: @evanliin @XanderChin @suryasure05 @kennykgguo March 24 - chose the mni-ml brand and started the educational blog March 30 - minitorch is DONE ahead of schedule. now we build on top of the framework. April 5-6 - cuBLAS matmul via koffi FFI. buffer pooling, strided batched GEMM, kernel optimizations. CUDA backend takes shape. April 7 - huge day. cross-platform CI pipeline, prebuilt npm binaries, v0.3.0 — CUDA live on @npmjs. flatten the monorepo, add @WebGPU + Windows CUDA build targets by eod. April 12 - flash attention CUDA kernel ships. we caught a bug where head dim > 32 was truncating. April 14 (during exam season), we recorded the demo in @Shopify recording studio during Aadi’s lunch break. Everything over the last 4mo finally came together. Cc: @fnthawar @tobi @alspee April 17: launch post and bought the domain mni.ml and we’re just getting started. We have so much in store for this summer, stay tuned 🫡 cc: @sundeep @GavinSherry

@VishnuSatish_ my goat

I built and trained a ~6M parameter GPT-2 entirely from scratch in C++, and it actually generates English text with mostly correct grammar! No PyTorch and no external dependencies. Just pure C++ 20. More info, GitHub link, and screenshots below!

@adiprasadd unironically pu 9 pm tn lets run it 😭

@ayushrgarg next step is quit poker

99% of gamblers quit before they win I won

@MankyDankyBanky my goats @_reesechong and @MankyDankyBanky this is insane

I trained a 12M parameter LLM on my own ML framework using a Rust backend and CUDA kernels for flash attention, AdamW, and more. Wrote the full transformer architecture, and BPE tokenizer from scratch. The framework features: - Custom CUDA kernels (Flash Attention, fused LayerNorm, fused GELU) for 3x increased throughput - Automatic WebGPU fallback for non-NVIDIA devices - TypeScript API with Rust compute backend - One npm install to get started, prebuilt binaries for every platform Try out the model for yourself: mni-ml.github.io/demos/transfor… Built with @_reesechong. Check out the repos and blog if you want to learn more. Shoutout to @modal for the compute credits allowing me to train on 2 A100 GPUs without going broke cc @sundeep @GavinSherry

@liao_lucas check your DMs!

@ayushrgarg @ayushrgarg slide me a referral tho

is ts tuff

@KlausCodes u coming?

@ayushrgarg TUFF

we do things a lil diff around here

@forwarddeploy ill bring the parle-g

Thinking of hosting more chai and samosas in SF at ours ☕️ Want to come hang out with good people and have fun snacks, lmk below! 🙌

The future of artificial intelligence is physical. @physical_int runs robotic control inference on Modal with >2x lower latency than the lag between your brain and your finger.

@ksgat_ yep

@ayushrgarg quads?

anybody really good @ flying drones located in the bay? will pay you $$$ to fly drones all day free lunch + unlimited snacks & drinks

@PoG_Shmerb DM me proof of your drone skills; as for the job itself you'll fly a drone in an environment we choose (we'll provide drones + transmitters and anything else you may need) & we'll collect that data

@ayushrgarg What kind of footage do you want to capture?

@krupaad lets chat, we're building autonomous drones

bit late to the recruiting cycle, but looking for a summer internship in ML/hardware/inference!! i've been working on CUDA kernel writing, FPGA acceleration and RTL. would love to find a team doing similar work this summer dual US/Canada citizen, can relocate anywhere DMs open :)

@NehaKasoju ts not tuff

relentlessly build the world you want to live in

recently rediscovered an impulse purchase from 2 years ago