Junu Park

I want to feel excited about the future, not afraid of it. I want to thrive with what I’m given, not fear AI. I want to do things for the love of the game, not because of fomo :)

@jino_rohit congratulations jino!

@v0xium nice work!

Implemented the Attention Mechanism from scratch in C. At its core, it is just: • Matrix Multiplication • Transpose • Scaling • Softmax If you are learning how LLMs work, spend 30 minutes implementing it yourself. You will understand more than hours of reading articles.

been on x for a while, but never properly introduced myself, so here goes :) i'm pradheep. mostly just a curious person who loves math, c++ and reinforcement learning. alphago is what pulled me into rl in the first place, mainly because of how learning, search and mathematical ideas all came together so beautifully. i usually understand things properly only after trying to build them myself, so most of what i share here ends up being projects, experiments, paper notes, and whatever i'm currently trying to figure out. this is where i log most of what i build and write: pradheep.dev i've posted enough stuff now that most of it gets buried after a while, so i'm keeping this thread for the things i've built and written. i'll keep adding more whenever something feels worth keeping here 🧵

@sodakeyEatsMush rl seems to be so much fun!

[update] - Currently reading about curriculum learning in RL. Basically the main idea is that in case of humans, you don't expect a child to directly be able to understand something complex like integrals. There is a proper education system which has been designed where they are taught things of increasing order of complexity instead of directly giving them difficult problems to solve. Similarly curriculum learning explores this idea but for teaching something to a model(Atleast this is what I've understood after reading the intro,I'm yet to actually read the rest of the stuff). - Also I've been doing the cs336 lectures in my free time and after completing the first few lectures I'll do the assignment. Currently after each lecture I'm trying to internalize each idea properly instead of just speedrunning it coz imo the main reason for me to actually do this course is to be able to properly internalize a lot of ideas which I've seen before but haven't properly tried to understand them.

@mohitwt_ Nice!

going deeper with FlashInfer and reading multiple things related to serving kernels in LLM inference I want to experiment with these kernels and build a few systems ideas I have in mind that could complement them well, gonna read more about this and start with something cool

@jino_rohit congratulations!!!

i had an amazing interview experience with nvidia, ill try and compile my thoughts when i get some time

@sodakeyEatsMush let's go!

[update] - So, I had generated some synthetic data using deepseek where I first pregenerated optimal solutions for some puzzles and then for each puzzle I kept on sending it moves one by one and also the state of the board and asked it to create a rationale behind the move. But then I noticed that because of the way that the prompt was structured it sometimes strting arguing about the solution even tho it was correct. So I've updated the prompt, now synthetic data is generating again and then we do SFT before moving to RL. - Will be doing the CS336 course also in the side. Will also read additional stuff along with the lectures if I feel like going more in depth

@jino_rohit doing all this with full time job is crazy. really inspiring bro

over the last 6-8 months, ive been trying to move towards the ml systems and ai infra space. these are some of my favorite work ive done - 1. a small python inference engine which is mainly a testbed for me to implement the major inference techniques, its about 600 toks/s on an rtx 4060 ti with prefix caching - github.com/JINO-ROHIT/tac… 2. wrote a blog post that outperforms cublas on ada using cute dsl - jino-rohit.github.io/blogs/08_cute_… 3. a detailed post on ncccl collective communication - jino-rohit.github.io/blogs/11_colle… 4. a blog series on how torch compile works and internal mechanics - jino-rohit.github.io/blogs/ 5. i maintain my notes and experiments of most of the work around ml systems here - github.com/JINO-ROHIT/ml-… 5. open source work in sglang and llm-compressor(vllm). im trying to become a stronger ml systems and inference engineer. what should i spend my next months getting better on?

@jino_rohit yup

@junupark_ triton?

slowly but progressing ..

@jino_rohit nice

ive gotten too deep into actually understanding kv cache in sglang. 1. how is paging and page table done in sglang 2. how is page allocation done 3. partial pages for new requests vs existing requests 4. how does radix cache work, compare the effect of radix cache vs disabled

@sodakeyEatsMush good luck!

I think it might be a fit unfeasible for me to actually like go through all the lectures from CS336 but what I am planning on doing instead is use the curriculum from CS336 as reference and then go through relevant blogs/papers to fill in the gaps in my current knowledge/understanding. Let's see how this goes.

@Abhishekcur 💯

whatever happens with technology, and no matter how far ai goes, your ability to learn and think from first principles will always be your biggest asset. tools will change. frameworks will change. models will change. but the ability to understand, adapt, and learn from the ground up is something that never ever goes out of style.

@axrshz yess

@junupark_ seoul is beautiful. gonna visit someday ^_^

touch water

@Abhishekcur nice i will watch and give feedback!

will keep posting 1 or 2 videos per week consistently and also please give your genuine feedback too, it would be world to me. -also I would make videos, because I genuinely want to learn more deeply, not for presence, it would be great way to revise my concepts more deeply and learning from feedbacks

if you are into ml inference, please learn: -gpu architecture, cuda, memory hierarchy & tensor cores -high-performance cuda kernels, kernel fusion & profiling -kv cache, flashattention, pagedattention & speculative decoding -quantization (fp8, int8, int4), batching & continuous batching -inference engines (vllm, sglang & llama.cpp) -distributed inference (tensor parallelism, pipeline parallelism, expert parallelism & data parallelism) -nccl, nvlink, nvswitch, infiniband & high-speed communication -latency, throughput, memory optimization & performance benchmarking I would teach all these thing in my videos (even it's more deeper than the above loll) Here's the my youtube video link intro : youtube.com/watch?v=cuRpOF…

@maxxfuu that's a lot! good luck on your journey!

Day 2/90 of Inference Engineering I built a PyTorch roofline profiler because I want to prove and understand intuitively why decode is memory bound and prefill is compute bound. Also went over GPU Lectures 1-3 as well while getting in my 10K steps in. Finished up on my notes on MHA. I will polish up the Ridgeline over the coming weeks. For the next 7 days, I want to: - Finish the GPU Lectures - Tinker with CUDA - Read 3 papers for my next project: - SGLang: 2309.06180 - Sarathi-Serve: 2403.02310 - PagedAttention: 2312.07104 github.com/maxxfuu/Ridgel…

@jino_rohit 💯

the hardest part about inference engineering is not to go into random rabbit holes. this space is like at the intersection of a lot of things at once, that at some point you will see yourself learning things from hardware to systems to communications etc. while exploring, the best thing would be to try and land a job in the field or at the intersection as soon as possible and from then, you could probably double down into a specific niche.

@jino_rohit lets go!

my next series of blogs is going to be on cuda graphs and piecewise cuda graphs