Chenwei Cui

Introducing Multi-Head LatentMoE 🚀 Turns out, making NVIDIA's LatentMoE [1] multi-head further unlocks O(1), balanced, and deterministic communication. Our insight: Head Parallel; Move routing from before all-to-all to after. Token duplication happens locally. Always uniform, always deterministic. It works orthogonally to EP as a new dimension of parallelism. For example, use HP for intra-cluster all-to-all as a highway, then use EP locally. We propose FlashAttention-like routing and expert computation, both exact, IO-aware, and constant memory. This is to handle the increased number of sub-tokens. Results: - We replicate LatentMoE and confirm it is indeed faster than MoE, with matching model performance. (See Design Principle IV in [1]) - Up to 1.61x faster training than MoE+EP with identical model performance. - Higher model performance while still 1.11x faster with doubled granularity. 📄 Paper: arxiv.org/abs/2602.04870… 💻 Code: github.com/kerner-lab/Spa… [1] Elango et al., "LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts", 2026. arxiv.org/abs/2601.18089

> scaling model width mattered more than all hparam tuning auto research models just out here reading the Gemstones paper instead of tuning the random seed as god intended

Introducing GradMem: writing context into memory with test-time gradient descent. Instead of encoding text with a forward pass, we optimize memory tokens per example with a reconstruction loss. So memory is written by running actual gradient descent on it at test time.

Introducing Expert Threshold Routing: - ✅ load balance - ✅ dynamic computation - ✅ autoregressive - ✅ zero train-inference mismatch At 2.4B params, Expert Threshold achieves 0.067 lower CE loss than Token Choice (equivalent to 1.6× data efficiency).

Conceptually, ET = Expert Choice on an infinitely large batch. As batch size grows, each token's influence on the threshold vanishes, making routing independent and causal. This also means ET enables causal inference for EC-trained models without retraining.

Introducing M²RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling We bring back non-linear recurrence to language modeling and show it's been held back by small state sizes, not by non-linearity itself. 📄 Paper: arxiv.org/abs/2603.14360 💻 Code: github.com/open-lm-engine… 🤗 Models: huggingface.co/collections/op…

Updating reward model and policy together during RLHF in a per-batch manner, with active learning. It is possible here with model-based feedback. But how could it be implemented practically with actual human feedback?

cross-entropy loss creates a score for every token in the vocabulary (128K+ for Llama-3). you use it once and discard it. the fix: process it in chunks, keep a running total. same math, 97% less memory. opened a PR on JAX, the issue author asked for more features, built those too

matrix POV of mHC vs AttnRes

Attention Residuals Revisited kexue.fm/archives/11664

So non-linear RNNs are back in the game beyond xLSTM! Nice work improving upon our #FlashRNN library using matrix valued hidden states!

Cute

GPT-5.4 mini is available today in ChatGPT, Codex, and the API. Optimized for coding, computer use, multimodal understanding, and subagents. And it’s 2x faster than GPT-5 mini. openai.com/index/introduc…

和Kimi的AttnRes类似，都致力于解决深度方向信息传递问题，但 MoDA 在注意力算子内部融合，AttnRes 在残差连接层面替代固定权重

Attention residuals and mixture of expert reuse (x.com/yichen4nlp/sta…) are two independent results pointing in the same direction: a single transformer layer, looped n times, is more efficient than n independent transformer layers. As @willccbb has often remarked, the best, most enduring discoveries are when you get improved performance by making the architecture LESS complicated. It seems abundantly clear to me that a single ultra wide layer, looped n times, can be made into a strict generalisation of the current paradigm, whilst also being more elegant in its simplicity.

I’ve been telling people this a lot today: I enjoy so much working with people who care about what they are building and craftsmanship. It is a privilege to have a chance to work on something I’m passionate about, beyond making a living. I cherish it and don’t take it for granted.

AttnRes might be one of the most natural way to achieve diverse communication channels. Good job @nathancgy4 @yzhang_cs !

my fav paper of this year yet > take attention over res steam, > pp cause bottlenecks in large training so take it block wise > hc/mhc keep state (res steam) by structed matrix like mamba or other linear attention this is beautiful, i love it attention is all you need :D

Love this table! Layer is another kind of token. BTW: Post-LN TRMs (DeepNorm ieeexplore.ieee.org/stamp/stamp.js… and our KEEL arxiv.org/abs/2601.19895 ) also have dynamic mixing coefficients: affine weight (learned during training) and rms norm ops(input-dependent)

@Yulun_Du @ilyasut SGD is a ResNet too (the blocks of it are fwd+bwd), the residual stream is the weights so... 🤔 We're not taking the Attention is All You Need part literally enough? :D

The idea of rotating attention by 90° is sooooooo cool (credits to @Jianlin_S 's insights), and it surprisingly works. We (w/ the amazing @nathan) are so excited about this— been working on the paper for months and couldn't stop. Go give it a try. It's a drop-in replacement for standard residuals, born in 2015. really like the figs btw :-)