TensorTonic

This is what frontier labs like OpenAI, DeepSeek, and Meta expect research engineers to be fluent in. We built an interview track for the research engineer role. Four modules: 1. LLM Internals: Attention, RoPE, KV Cache, MoE, normalization, embeddings. 2. Post-Training and Alignment: PPO, DPO, GRPO, reward models, preference optimization. 3. Research Frontier Math: The linear algebra, probability, optimization, and derivations 4. Training and Decoding: Optimizers, schedulers, mixed precision, sampling, beam search, speculative decoding If you're aiming for research roles, you'll run into these sooner or later.

Missed my protein in-take but not my @TensorTonic streak tonight. First badge received :)

@ali_sher_g Hey, it's working from our end. What issue are you facing?

Writing C++ CUDA kernels is the highest-leverage skill right now. You stop treating the GPU as a black box. You learn why an op is slow, what memory costs, and how the frameworks you use daily are built underneath. You write the CUDA kernel, we give you a platform and a free gpu

People are implementing state-of-the-art research papers on TensorTonic.

@__acbraingenome @duolingo @prathamgrv Yes, We're coming up with this next

Feature idea : Can @TensorTonic send a web / gmail notification (reminder) to solve your questions and maintain your streak? Kinda similar to duo @duolingo @prathamgrv

7 math ideas every ML engineer uses daily and almost nobody has actually derived: 1. Why gradient descent moves in the direction of steepest descent, not just downhill, but provably the steepest direction, straight from the definition of a directional derivative. 2. Why softmax plus cross-entropy collapses into that suspiciously clean gradient of pred minus true, and what breaks the moment you swap the loss function. 3. Why the chain rule is backprop, not an analogy for it, the same operation applied mechanically to a computation graph. 4. Why dividing attention scores by root d_k isn't arbitrary, it's variance control, derivable from how dot products scale with dimension. 5. Why KL divergence isn't symmetric, and what that asymmetry actually costs you when you pick forward vs reverse KL. 6. Why Adam's second moment estimate quietly approximates a diagonal Hessian, making it quasi-Newton in disguise. 7. Why eigenvectors are the directions a matrix doesn't rotate, the one geometric fact that makes SVD, PCA, and spectral clustering all click at once.

To all those starting with @TensorTonic Let’s breakdown the roadmap, I’m sure we’ll have heard of divide and rule, I assume there must be some truth to it :P It’s quite tempting to deviate towards “Attention is All You Need” implementation right away, but ladies and gentlemen here’s where you need to calibrate. It’s similar to swimming, if you don’t learn how to glide, you won’t learn how to fearlessly dive :) Here are the first 9 ( even my OCD kicked in, but yeah 9 not 10 :/ ) problems to solve, in the given order to build your comfort, confidence and strong foundation first : 1. Matrix Transpose. 2. Make Diagonal Matrix. 3. Matrix Trace. The first three problems introduce you to the concept of matrix traversal and indexing. Once you’re familiar with matrix, you move forward with: 4. Dot Product. 5. Euclidean Distance. 6. Manhattan Distance. My favorite now, 7. Cosine Similarity - This is where everything you’ve done so far comes together. Lastly, hop onto : 8. Eigenvalues 9. Matrix Inverse, these problems are less about testing your coding skills and more about your conceptual and mathematical understanding. The last three, I bet would meet you in your ML journey quite often, so make sure, you understand the concept deeply. Let me know how it goes, meanwhile I'll go and make a cup of tea! :) Just a snapshot of my Tensor-tonic journey alongside.

Followed by Matrix Transpose, the second problem I solved @TensorTonic was Matrix Trace. What’s Matrix Trace? The answer is pretty straightforward : sum of the diagonal elements of a square matrix. To make it sound more intellectual and mathematical :P I'd frame it as : Given a square Matrix M of dimension n, Trace += M[i][i], where i ∈ [0,n−1] Here’s a solution with the time complexity O( n^2 ). Challenge : It’s not an optimal solution, can you spot the line in the code that is unnecessary and hurting the time complexity? How'd you optimise it for O(n) time ? Also, why do we even care about adding diagonal elements in the first place, we'll touch that aspect soon. A little hint, can you think how could it be related to variance?

Few days back I posted a roadmap to @TensorTonic If you’re new to machine learning, I’d recommend solving Matrix Transpose problem, it might look easy to discard and directly jump onto neural networks, but the very foundation of neural network is based on Matrix Transpose. And to those who are new to the concept, and coding, here’s a solution with time complexity O(mn). This problem intents to teach you the basic concepts : 1. How to create an all zero numpy array, 2. How to flip a matrix, 3. How to preserve datatypes, 4. How to deal with the shape error when you accidentally try to allocate rows, cols to A.shape, and why is that wrong folks, find out yourself :)

If you're serious about AI engineering, spend a weekend implementing these 8 reports: 1. DeepSeek-V3: MoE routing with auxiliary-loss-free load balancing and multi-token prediction 2. DeepSeek-R1: Reasoning from pure RL with no SFT warm start, then distilled into 1.5B to 70B models 3. Mistral 7B: Sliding window attention and a rolling KV cache for efficient long-context inference 4. Mixtral 8x7B: Sparse MoE with top-2 routing at roughly the inference cost of a 13B dense model 5. Llama 3: 15T token pretraining, continued pretraining for longer context, and a production-scale RLHF pipeline 6. Qwen2.5: Shared tokenizer across dense and MoE models with strong multilingual performance 7. Gemma 2: Teacher-student distillation and logit softcapping for more stable training 8. LoRA: Low-rank adapters that made fine-tuning billion-parameter models practical on a single GPU

Epsilon-Greedy is one of the simplest strategies in reinforcement learning. Most of the time, the agent chooses the best action it knows, but every so often, it explores something new to discover even better options. The concept become much more intuitive when you can actually see them. Here's a visualization :

Build a RAG agent from scratch and understand every stage of the pipeline. You'll: >Chunk and index documents. >Generate embeddings and perform vector search. >Retrieve the most relevant context for each query. >Augment prompts with retrieved passages. >Compare responses with and without retrieval. >Understand how RAG grounds LLM outputs in external knowledge. tensortonic.com/projects/rag-a…

If you could read just one paper from each area of AI, make it these: • CV → Vision Transformer (ViT) • NLP → Attention Is All You Need • Generative Images → DDPM (Diffusion) • Multimodal → CLIP • Optimization → Adam • Regularization → Dropout • Tabular ML → XGBoost • Reinforcement Learning → DQN (Atari)

Everyone uses optimizer.step() in PyTorch or apply_gradients() in TensorFlow. But could you implement the update equations yourself without opening a reference? > Gradient Descent > Logistic Regression Training Loop > Adam > RMSProp > AdamW > AdaDelta > Learning Rate Scheduler > Cosine Annealing > L-BFGS Nine optimization algorithms and training components, each implemented from scratch. Practice all of them on TensorTonic.

@1999Yadwade We're removing that soon. next week 🫡

@TensorTonic Why login is required tho? Instant turn off.

We released a sheet of most asked questions in MLE interviews. > loss functions, optimizers > supervised & unsupervised learning > activation functions > model evaluation timed based mock assessments, curated from real FAANG & top lab interviews. link in comments.