xjdr

Writing jitted jax code is like playing Dark Souls but in python

Sad if true, lol if true, but also there are like 2 companies it could have been for the volume we are talking so ...

@thegeneralist01 @reach_vb i love fonts and typography so they were human curated (my me)

@_xjdr @reach_vb xjdr who dealt with the fonts was it codex or someone from your team i basically want to ask how one can educate oneself on such beautiful fonts as those on the main page

what did you build with codex this week? 👀

100B+ params is usually what i mean when i say "at scale" . ya, canon paper aside (there are several things in there i'd do differently) but the core canon style mixing at a few points in the model is both sane and helpful. As to if its 'better' than alternatives at scale, i think we still have more testing to do to say one way or the other

@_xjdr hmm what scale are we talking about? still intuitively think tokenshift if done right should scale fine don't get me wrong. but if we're talking about how it's done in the canon paper i disagree...

very likely they already abandoned it. truth is canon does not work at scale. most chinese labs had medium scale canon pretraining runs and results sucked. got hinted to look deeper into the canon paper again and there are some pretty sloppy math mistakes once you go through it somewhat carefully. In hindsight it should not be that surprising it falls short of what ppl expect or what small scale experiments might suggest. the more i think about it attnres is really clean and probably sufficient on its own.

@srchvrs @giffmana hahaha

@giffmana @_xjdr "extremely strict colleagues" for younger colleagues you could explain that this is a euphemism.

i cannot explain to you how much i hate publishing i solved the problem. its no longer interesting to me thank you to all the people who read any of what i made

@georgejrjrjr "I can tell if someone is lying just by looking at them!"

@_xjdr Introduce yourself as a jury nullification activist; problem solved.

I have jury duty this week. Civic duty be damned this is the worst

@JingyuanLiu123 i believe your intuition is correct but i think there are deeper mysteries to explore instead of "these models are shit"

my personal understanding is that the prenorm arch, ever since the llama stage, had such large residual rms problems for a long while. just simply check the output rms per layer, Kimi K2 and dpsk-v3 should also suffer from such problemw basically, the residual activation rms becomes very large as layer goes deep, while your branch is somehow similar scale and comparatively too small. this is an implicit collapse of your model: you thought your model is 64 layers deep, but actually it is only 54 layers, because last 10 layers are collapsed. a while ago, I tried hyperconnection (not mHC), but seems not working. super excited to see the open source community got an elegant solution!

@mahaoo_ASI @xlr8harder

@_xjdr @xlr8harder This is Nikita's and Elon's bad Poor taste algorithm

I'm not seeing your posts on my timeline anymore im hoping quote tweeting you will help

@xlr8harder i should be better about this but i naively assume following people should work. obviously, this is my bad

@_xjdr high signal lists are the only way to reliably see the people you want to see

@pastaraspberry @teortaxesTex i was one of teor's early followers and have never stopped dafuq are you talking about?

@_xjdr @teortaxesTex try following 😅

@teortaxesTex i still search your posts directly, but it would be nice if the people i followed and was specifically interested in showed up in my fucking feed

@_xjdr 🕊

@basedjensen @tekbog ♥️awesome work

@_xjdr @tekbog Thanks big dog :)

clankercloud from @basedjensen and @tekbog is so fucking good . im genuinely shocked and impressed

@CliffLattner yes, tremendous value but it hasn't been cracked yet (AFAIK)

@_xjdr Do you think FP8 attention has value? Assuming that the ALU bottlenecks can be alleviated so the increased MMA throughput actually helps

there are a bunch of research journal posts i didn't release for various reasons (some of which are still under active research) but i can enumerate a few preliminary findings: NSA is the exact right idea but a poor initial design. we can be inspired but still do better MTP is awesome but actually not all that helpful even if its counter intuitive it doesn't matter how pretty the math is N+4 distillation is not more efficient than KDL online distillation fa4 based sparse attention is going to be huge by the end of the year

@tekbog @SarahLacard @basedjensen its real good. you should be proud

@_xjdr @SarahLacard @basedjensen omg thanks

@tekbog as soon as AI gets good enough to write the posts itself, i will be publishing 100 papers a year until then again i am still the bottleneck (maybe just a skill issue tho)

@_xjdr if you have notes you can automate most of it and leave a summary for your followers, better than nothing, maybe put a label “AI summary of my work”