Sabitlenmiş Tweet
Wake up honey, new RWKV paper just dropped 🧵⤵️
Paper: arxiv.org/abs/2404.05892
Code: github.com/BlinkDL/RWKV-LM
Models: huggingface.co/RWKV (Apache 2.0 license)
(1/6)
English
Eric Alcaide
1.1K posts
Eric Alcaide
@eric_alcaide
Design is not finished. Common prosperity. LLMaxxing @poolsideai
LLMaxxing Katılım Eylül 2016
1.1K Takip Edilen1.2K Takipçiler
Gated DeltaNet-2 is here. 🚀
🔥 New paper: Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Gated DeltaNet-2 outperforms KDA and Mamba-3, the latest and best recurrent architectures, head to head at 1.3B. 🏆
💡 Here's the idea behind it:
Linear attention squeezes an unbounded KV cache into a fixed-size recurrent state. The hard part isn't just what to forget, it's how to edit that memory without scrambling the associations already in it.
Prior delta-rule models like Gated DeltaNet and KDA use one scalar gate to do two jobs at once: erasing old content and writing new content. But these two decisions act on different axes of the state, so tying them together is a real limitation.
Gated DeltaNet-2 decouples them.
✂️ a channel-wise erase gate b_t picks which key-side coordinates to read and remove
✍️ a channel-wise write gate w_t picks which value-side coordinates to commit
🔁 recovers KDA when both gates collapse to a scalar, and Gated DeltaNet when the decay collapses too
⚡ still trains fast: chunkwise WY algorithm with gate-aware backward, fused in Triton
📊 Results:
We train 1.3B models on 100B tokens of FineWeb-Edu, matched in recurrent state size, against Mamba-2, Gated DeltaNet, KDA, and Mamba-3.
Best average on language modeling + commonsense reasoning, in both recurrent and hybrid settings
Biggest gains on long-context RULER retrieval. S-NIAH-3 jumps from 63 to 90 over KDA, and multi-key needle retrieval climbs from 28 to 38
Joint work with @YejinChoinka and @jankautz.
📄 Paper: shorturl.at/AAlVb
💻 Code: github.com/NVlabs/GatedDe…
#LinearAttention #StateSpaceModels #Mamba #LLM
English
What i hear is 16x sequence compression for r% of training.
This is extremely cool and I can't wait to try it on IOTA.
Nous Research@NousResearch
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
English
English
@LChoshen Training on this setup might actually be less efficient but you can train on more data and promote aligned latents
English
@eric_alcaide Does anyone knows why none of those works compare to predicting multiple tokens? Or ablate the contribution of the input trick and the output trick separately? (Yes I see why given the input you need something on the output, but you can do it by order)
English
Did they just reinvent Patched Training for LLMs ? 🤔
They even have a very similar plot 👀
Nous Research@NousResearch
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
English
@bekindtopeople2 Cool idea yea iguess its all about a tradeoff between information density and closeness to the target distribution
English
@eric_alcaide Has anyone tried patch schedules? Why would they not ablate 3/4/N patch length?
English
Eric Alcaide retweetledi
ok this is sick
@pupposandro @davideciffa and @luceboxai got Laguna XS.2 running on a single RTX 3090 with ~111 tok/s decode, 5.4x faster 128K prefill vs llama.cpp, and made it the first MoE target for PFlash
open weights doing open weights things
English
Great things are happening and uou can feel it 🔥
Sandro@pupposandro
PFlash now run @poolsideai's Laguna-XS.2 (33B-A3B MoE) on a single RTX 3090. - 111 tok/s decode @ short ctx - 128K TTFT in 15.91s, 5.4x faster prefill vs llama.cpp - NIAH passes every (ctx, keep) point up to 131K - first MoE target supported by PFlash - hand-rolled CUDA, ggml only, no libllama great collab w/ @eisokant, @eric_alcaide, and the rest of the @poolsideai team. looking forward to working more on their great coding models. repo + GGUF in first comment.
English
SGLang team is cracked. Respect 🫡
LMSYS Org@lmsysorg
🌊 SGLang now supports @poolsideai's Laguna-XS.2, a 33.4B-A3B hybrid SWA + MoE model purpose-built for agentic coding and long-horizon SWE work ☑️ SWE-bench Verified 68.2%; Multilingual 62.4%; Pro 44.5%; Terminal-Bench 2.0 30.1% ☑️ 131K-token context for long agent traces ☑️ Native poolside_v1 reasoning + tool-call parsers (OpenAI-compatible) ☑️ BF16, FP8, and NVFP4 quantizations 👉 Cookbook: docs.sglang.io/cookbook/autor…
English
@eliebakouch sorry guys but not novel lol
x.com/eric_alcaide/s…
Eric Alcaide@eric_alcaide
Did they just reinvent Patched Training for LLMs ? 🤔 They even have a very similar plot 👀
English
nice pre training work by nous claiming ~2.5x efficiency gains, building on previous research like MTP/SuperBPE. overall intuition is that at each step you want the model to process and predict more tokens
Nous Research@NousResearch
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
English
Link: arxiv.org/abs/2407.12665… with code here: github.com/shaochenze/Pat… from Chenze Shao who's now at DeepSeek
English
@MrCatid yea lol claiming novelty for this work is an overstatement
English
Eric Alcaide retweetledi
Poolside is hosting a 2-day model research hackathon in London.
Join us to push an open-weight agent model as far as you can. RL and fine-tune Laguna XS.2, our latest-generation model, on Prime Intellect Lab.
Dates: May 29–30
Partners: @nvidia + @PrimeIntellect + @huggingface
Prize: NVIDIA DGX Spark
Agents need better models.
Better models need cracked researchers.
Link below.
English
@Raubertard @cheenanet Doesn’t need linear attention kernels and can have prefix caching without storing copies of the state
English
@eric_alcaide @cheenanet Why would you use that over qwen 3.6 35b?
English
RTX 3090 1枚で Qwen3.6-27B が 120〜200 tok/s の時代来たか〜
github.com/Luce-Org/luceb…
日本語
The reason we can tell OpenAI has not reached AGI is how hard it is to find their invoice 👀
English
@stableAPY Tell us back your thoughts and we might improve for the next one :)
English
so qwen 3.6 35B A3B fits on a 5 years old RTX 3060 12go at 128k context
that's pretty mad
next: trying to see if the model behave properly
AJ@ItsmeAjayKV
Qwen3.6-35B-A3B (TQ3_4S ~4bpw) on RTX 3060 (12GB) via llama.cpp-tq3 (TurboQuant): • ~619 t/s prompt (4K ctx) • ~60 t/s generation (128K ctx) • fits in ~12.4GB VRAM 128K context with usable decode speed on a single 3060 is kind of wild
English