Sabitlenmiş Tweet
Components of a coding agent: a little write-up on the building blocks behind coding agents, from repo context and tool use to memory and delegation.
Link: magazine.sebastianraschka.com/p/components-o…
English
Sebastian Raschka
19.7K posts
Sebastian Raschka
@rasbt
ML/AI research engineer. Ex stats professor. Author of "Build a Large Language Model From Scratch" (https://t.co/O8LAAMRzzW) & reasoning (https://t.co/5TueQKx2Fk)
United States Katılım Ekim 2012
1.1K Takip Edilen445.1K Takipçiler
I wouldn't be surprised if Google was at 1-2% active
Sebastian Raschka@rasbt
Meta observation: DeepSeek is still king of the active-parameter ratio
English
@rasbt Xiaomi 2.5 pro missing (4.2%, 1T)
Filipino
Meta observation: DeepSeek is still king of the active-parameter ratio
English
The table in HTML format for easier (and non-truncated) viewing: sebastianraschka.com/llm-architectu…
English
@MRashadnow In theory, yes. But then, I am not sure how much of a signal this is here. And I don't want to suggest that %active is the only variable determining quality.
I.e., you would get vastly different numbers for K2, K2.5, K2.6 even though they have the same 3.2% active ratio.
English
@rasbt I think a column for quality or overall intelligince of the model will paint the picture better
English
@AlexTheGoodman Yes! I did a write-up on these a few weeks ago: magazine.sebastianraschka.com/p/visual-atten…
English
@rasbt I was unaware of so many attention variants. MLA appears to be becoming more common?
English
@yechan_ai Thanks for the link and ping. I try to focus on those with open weights these days but I’ll add this to the list for the future
English
@rasbt Big fan of your works
Any interests on DLM?
Yechan Do@yechan_ai
Came across Cola-DLM(hongcanguo.github.io/Cola-DLM/) from ByteDance. A hierarchical continuous latent diffusion LM that separates global semantic planning (DiT in latent space) from local token realization (VAE decoder). Paper is out, but no code and no HF model yet. So I reproduced it from scratch. Happy to share with anyone interested 👇
English
A little talk on what we can learn from implementing LLM architectures from scratch in Python and PyTorch.
And how I approach new open-weight models, compare them against reference implementations etc:
youtube.com/watch?v=TXzQ7P…
YouTube
English
@yechan_ai Thanks for the link and ping. I try to focus on those with open weights these days but I’ll add this to the list
English
@MLCatttt I usually look at the config.json and eyeball it based on other architectures (that I already have in the LLM-gallery.com)
English
@rasbt does anyone actually have a reliable workflow for picking the right reference impl when there are 3-4 candidates floating around. would love to see how you triage
English
@bygregorr historically, this (and norm layers) is probably where I spent most of debugging time
English
@140ismymax Yes! It would be interesting whether this happens in GQA:SWA models only though or also in MLA or Gated DeltaNet models, for example.
English
Hi @rasbt really enjoyed your comments on sliding context windows in different models. Given all the talk over the last few months about repeating information in prompts, and the information in the front of the prompt getting lost, this has to be one of the most interesting aspects of LLMs at the moment.
English
Interesting paper. What I like about this is that it is a relatively low-commitment attention modification.
I.e., one can use it during most of training, switch back to vanilla attention near the end, and recover roughly the same modeling performance as if full attention had been used the whole time.
elvis@omarsar0
Cool idea from Nous Research. What if you could speed up long-context pretraining with a subquadratic wrapper that you remove before deployment? That is the idea behind Lighthouse Attention. The method wraps ordinary SDPA with a hierarchical, gradient-free selection layer that compresses and decompresses queries, keys, and values symmetrically, preserving left-to-right causality. Crucially, it can be removed near the end of training in a short recovery phase, so the deployed model still runs vanilla attention with no architectural cost at inference. Preliminary LLM experiments report faster total training time and lower final loss than full-attention baselines. Why does it matter? Most efficient-attention work either changes the deployment-time architecture or pays a quality tax to do so. A training-only wrapper that survives a clean recovery phase sidesteps both. If it scales, this becomes an important training-time speedup for long-context pretraining. Paper: arxiv.org/abs/2605.06554 Learn to build effective AI agents in our academy: academy.dair.ai
English
English
My wife after watching some pop-investing reel or something:
> Lucas, you bought SanDisk a bit over a year ago, right???
Me: yeah
Her, all excited: how much??
Me: lemme check, but i think 2Tb
The SanDisk i bought:
English
@willdepue @ChiefScientist You are joking but I use an hdmi version of this for OpenClaw.
But to be fair, you could just use software (caffeinate, amphetamine apps) to keep the laptop awake.
English
Tired of holding your laptop half open to keep your agents running?
Introducing AgentPlug: A USB-C dummy plug that keeps your Mac in clamshell mode by pretending to be an external display!
No commands, no security worries (just pull it out to stop!), no hassle.
English
@rasbt @TSMCCruz ERNIE 5.1 is available on the chatbot ERNIE Bot (ernie.baidu.com), while the API access should be on the way
English
Back from a little family break! Lots has happened, and I’m planning to do a deeper dive into the most interesting architectural components (soon).
Btw, are there any major architectures I missed below?
English
@TSMCCruz Awesome, thanks! Looks like Ernie 5.1 & SubQ are not available yet, but the other ones look interesting.
English
@rasbt As always you did a pretty thorough job Sebastian. Can't wait for that deep dive.
Skimming through the gallery didn't see:
-LongCat (Meituan Model-x.com/Meituan_LongCa…);
-SubQ( SubQuadratic Model- x.com/alex_whedon/st…
-ERNIE 5.1 ( Baidu Inc. Model- x.com/Baidu_Inc/stat…)
Baidu Inc.@Baidu_Inc
ERNIE 5.1 just dropped. Built on ERNIE 5.0's pre-training foundation, our latest foundation model upgrades search, reasoning, knowledge Q&A, creative writing, and agentic capabilities, while using only around 6% of the pre-training cost of comparable models. More in the thread 🧵
English