Saurabh Dash

@giffmana @tenderizzation @bilaltwovec I agree! Unfortunately, It just didn't gel well with the 384 size!

@TheyCallMeMr_ @tenderizzation @bilaltwovec 14 is actually the superior one that leads to good seqlen on then-common image sizes (224, 448, etc), which is what matters.

who remembers efficientnets

@giffmana @tenderizzation @bilaltwovec Needs another line for the patch size

@tenderizzation @bilaltwovec People have still not forgiven me refusing to slightly bump these two numbers...

@F1 bring back the 3 digit precision in race interval visuals, you cowards

Borat did it first

All this and more! we go over our design decisions and the lessons learned along the way! Please do come hang out if you are at GTC. A huge thank you to all the team members for their hard work! nvidia.com/gtc/session-ca…

Hey all, I will be at GTC next week talking about all the work my team and I did on large-scale MoE training in JAX on GPUs! We decided early on to have a fully dropless training stack to avoid token dropping. (1/7)

My ICML review pile in a nutshell: This is the best thing invented since sliced bread. Look at how good it's on CIFAR-10.

@awnihannun GOATed run @awnihannun! 🫡

Today is my last day at Apple. Building MLX with our amazing team and community has been an absolute pleasure. It's still early days for AI on Apple silicon. Apple makes the best consumer hardware on the planet. There's so much potential for it to be the leading platform for AI. And I'm confident MLX will continue to have a big role in that. To the future: MLX remains in the exceptionally capable hands of our team including @angeloskath, @zcbenz, @DiganiJagrit, @NasFilippova, @trebolloc (and others not on X). Follow them or @shshnkp for future updates.

@RicardoMonti9 @KaleighMentzer @agcrnz I'm also quite pleased that @Cohere_Labs released Tiny Aya the day before we released ÜberWeb, and we were able to evaluate it and include it in our report. The whole Aya project has been a big inspiration for us. I officially declare it Multilingual Release Week!!

Tiny Aya reimplementation From Scratch! Have been reading through the technical reports of the recent wave of open-weight LLM releases (more on that soon). Tiny Aya (2 days ago) was a bit under the radar. Looks like a nice, small 3.35B model with strongest multilingual support of that size class. Great for on-device translation tasks. Just did a from-scratch implementation here: github.com/rasbt/LLMs-fro… Architecture-wise, Tiny Aya is a classic decoder-style transformer with a few noteworthy modifications (besides the obvious ones like SwiGLU and Grouped Query Attention): 1. Parallel transformer blocks. A parallel transformer block computes attention and MLP from the same normalized input, then adds both to the residual in one step. I assume this is to reduce serial dependencies inside a layer to improve computational throughput. 2. Sliding window attention. Specifically, it uses a 3:1 local:global ratio similar to Arcee Trinity and Olmo 3. The window size is also 4096. Also, similar to Arcee, the sliding window layers use RoPE whereas the full attention layers use NoPE. 3. LayerNorm. Most architectures moved to RMSNorm as it's computationally a bit cheaper and performs well. Tiny Aya is keeping it more classic with a modified version of LayerNorm (the implementation here is like standard LayerNorm but without shift, i.e., bias, parameter).

@Cohere_Labs Tiny Aya is the true multilingual on-device model we've been waiting for. 70+ global languages in a 3.35B parameter model 🤯

Cohere launches a family of open multilingual models techcrunch.com/2026/02/17/coh…

Mobile access unlocks the real potential of open models. Thanks to @pocketpal_ai and especially @ghorbani_asghar for helping us bring Tiny Aya to mobile! 📱 Their expertise made it possible to deliver the most capable multilingual model at this scale directly to people's phones.

Introducing ✨Tiny Aya✨, a family of massively multilingual small language models built to run where people actually are. Tiny Aya delivers strong multilingual performance in 70+ global languages in a 3.35B parameter model, efficient enough to run locally, even on a phone.

@____aakanksha @sarahookr @mziizm +1 these are great recommendations. Gulati’s is an absolute pain to get into, so the restaurant beside it is also great!

@sarahookr bukhara for north indian, carnatic cafe for south indian and chaat + rasmalai from haldiram’s! friends rave about gulati’s for butter chicken etc; oh and the tender coconut ice cream at natural’s 🤤 (should also try indo-chinese / momos in delhi - maybe at berco’s!) cc @mziizm :)

Ok. It is time. I have time on Saturday and Sunday night to explore New Delhi ahead of the summit. Let’s go Delhi food recommendations. 🔥

@sarahookr Definitely try Bukhara! Especially the kebabs and the Dal Bukhara.

If you are interested in squeezing every drop of available compute, @acyr_l is hiring for an ML Systems engineer to make GPUs go brrrrrrr jobs.ashbyhq.com/cohere/c99e61c…

Calling it compute rich/poor and not Jensen’s Inequality is a missed opportunity

@MaxMa1987 Super cool to see this idea applied to LLMs! We previously explored it for event-camera data. openreview.net/pdf?id=ZCStthy…

After about 2 years, we are proud to release Gecko, an efficient architecture that improves upon Megalodon, with capability of efficiently and inherently processing sequences with unlimited context length. One of the most important idea in Gecko is Adaptive Working Memory(AWM), implemented using a linear attention mechanism with a position-aware online softmax activation. Notably, AWM globally compresses information into memory, rather than discarding historical information through forgetting. In a controlled head-to-head comparison with Llama2 and Megalodon, Gecko achieves better performance in the scale of 7B and 2T training tokens. Gecko achieves 1.68 training loss, vs. 1.67 of Llama2-13B, with half number of parameters on 2T tokens. Paper: arxiv.org/abs/2601.06463 Code: github.com/XuezheMax/geck…

@sarahookr I can send the food recommendations! ;)

I'm really looking forward to also getting to know the AI ecosystem in Delhi (for the summit) + Bangalore. If you have recommendations of initiatives I should visit or people I should meet while I'm there, send them my way.

My first trip to India is next month. I'm honored to be attending the India-AI Impact Summit 2026. Truly very meaningful given our commitment @adaptionlabs to building global technology and ensuring language coverage.