Sabitlenmiş Tweet
ali
626 posts
ali
@waterloo_intern
ml research, kernels, and the occasional peer-reviewed shitpost inference @baseten || eng @uwaterloo
San Francisco Katılım Ekim 2024
89 Takip Edilen30.3K Takipçiler
@waterloo_intern Haha, now I understand how you were always available in my EU timezone.
Keep cooking Ali!! 🐐
English
holy shit i'm going to print this and stick it on all my gym fits and frame it and hang it on my wall and look at it every time i take creatine and show it to every one i meet holy fuck bryan johnson my goat i love you
Bryan Johnson@bryan_johnson
@waterloo_intern @ghostrodeo0 incredible project ali and your intense focus lowered pro inflammatory genes expression and extended you life
English
@dhruvbhatia0 so you're saying we draw the line at 3.6 billion times
English
@waterloo_intern Since he didn't do it I will.
The inner lining of plastic in those cans is harmful, and if you do it 3.7 billion times more in your lifetime your balls will be 1% plastic by weight.
English
@easonjiangcc super cool, do you mind putting up a version translated to english? i'd love to read the extra notes you have here!
English
做了个中文网页版，增加了其他LLM相关的一些内容，完全为了自学与分享，yichencityu.github.io/llm_evolution_… ，侵权必删
ali@waterloo_intern
I spent 48 hours with the Kimi K3 modeling code. It took: - 650 mg of caffeine (mandatory) - 40 cans of LaCroix (optional... world record (?)) - 8 papers - 6 months off my lifespan Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2
中文
@Hi_Mrinal That's so funny. You can't blame me, honestly. You both have very similar profile pictures haha
English
@waterloo_intern Yoo that's this guy
Pankaj@the2ndfloorguy
i hooked my whoop to my work calendar to find which coworker gives me the most stress 🚨 thanks to fable, I reverse engineered whoop to pull per minute heart rate. nd matched spikes with cal events and attendees I now have a leaderboard and I think about it daily. few info masked for obvious reasons ;)
English
> a very high effort blog wonder who have written
> Checks bio
> Uwaterloo
Every Damnn time 😭
ali@waterloo_intern
English
I spent 48 hours with the Kimi K3 modeling code.
It took:
- 650 mg of caffeine (mandatory)
- 40 cans of LaCroix (optional... world record (?))
- 8 papers
- 6 months off my lifespan
Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2
ali@waterloo_intern
English
will read this before going to bed
ali@waterloo_intern
I spent 48 hours with the Kimi K3 modeling code. It took: - 650 mg of caffeine (mandatory) - 40 cans of LaCroix (optional... world record (?)) - 8 papers - 6 months off my lifespan Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2
English
@saranormous hahahaha "intern"
technically still true for two more weeks
English
Great write up on technical innovations in Kimi3 that all can use
baseten “intern” remains based
ali@waterloo_intern
English
"The central change is not scale alone. Each architectural step changes what the model stores, how it updates that state, or how it retrieves information that a fixed-size state cannot preserve.
Kimi K3 combines constant-state recurrent memory, periodic softmax retrieval, sparse expert capacity, and selective depth-wise residual access. The result is a system that spends additional capacity where it has a specific functional role."
ali@waterloo_intern
English
@waterloo_intern this is brilliant! thank you for putting this together.
Glad I spent 30 mins reading this right away, otherwise this gem would have been left rotting in my bookmarks
English
@thepatelkush Attention is all you need
Linear Attention
Gated Delta Attention
Kimi Linear
Kimi Residual Attention
preferably in that order
English
@waterloo_intern Excellent writeup, where can i read foundation of this in details? What papers you recommend to start with.
English
This is an absolutely outstanding piece of work.
A crystal-clear, beautifully structured walkthrough of seven years of architectural evolution—from GPT-2 all the way to KimiK3. The way you break down linear attention, DeltaNet, gated variants, MLA, MoE, and AttnRes is both rigorous and incredibly readable.
Truly a masterclass in technical writing. Massive respect. This is the kind of worklog that deserves to be bookmarked, shared, and studied.
ali@waterloo_intern
English
@waterloo_intern Is there a short answer somewhere to the “Is it just scale?” question I missed. Would be helpful for scanning to have some kind of summary/answer to that tantalizing question.
English
Hi! I really enjoyed reading your article—it's one of the clearest explanations of the progression from GPT-2 to Kimi K3 that I've seen.
Many Korean AI engineers and researchers would benefit from it, so I'd like to translate it into Korean and share it with the Korean community.
Of course, I will clearly credit you, include the original article link, and state that it is a Korean translation of your work.
Would that be okay with you? Thank you!
English