Yunfan Zhang

@teortaxesTex @_xjdr This is vibe research. They trained the 3B model for only 0.1 B tokens and the validation loss is > 5.0. Need much more pretraining tokens to draw any conclusions.

> Engram is just regularization > Gains come from context-aware gating (dynamic feature adjustment) & an extra residual pathway (better gradient flow), NOT memory content insane if real. @_xjdr can you reproduce?

Stop worshiping celebrities & rich people. Worship empathy. Worship effort. Worship peace.

@frontier_foid My top two guesses are: (1) A cascading, coarse to fine attention where an indexer first looks at compressed global information, and then picks finer details / individual tokens? (2) Something interesting in terms of how to organize multiple attention heads, maybe?

just a little treat for my 100 sweet followers 💖 - the whale is cooking... that sparse attention is literally going to be the most complex architecture we’ve seen in a frontier model yet. actually insane. - also mla is cancelled. they dropped it. too much complexity

@frontier_foid @YouJiacheng After consulting DSv3 and GPT-5, @YouJiacheng is actually right about bi-directional all-gather. You just split a tensor in half, one half travels clockwise along the ring while the other counter-clockwise. Now both links are active 100% of the time.

okay, it turns out TPU Ironwood (4×8×8=256 chips pod) uses pure FSDP for DeepSeek-V3 training. DGX GB300 (256 chips, probably 4× NVL72?) uses (DP, PP, EP) = (4, 2, 32) with 8 pipeline stages and 32 micro batches. Tokens/sec/chip comparison is 2446 vs. 3848 in BF16.

@frontier_foid @YouJiacheng I am just thinking FSDP gets rid of the bubble in PP and the expert load balance in EP so it is a cleaner implementation. I have not trained anything close to 600B MoEs (would love to, but no GPUs) so I could be wrong.

@YouJiacheng Yes 1.2 TBps is the sum of 3 axis for TPUv7. Still reading on literature for bidirectional bandwidth with FSDP + ring topology, but thanks for the clarification!

@z4y5f3 *probably the 1.2TB/s is the sum of 3 axes, I don't know, but it's still higher than 0.9TB/s.

@deanwball I don’t agree that AGI is impossible, but the article raises a good question beyond that: if our remaining science/eng problems are set by physical limits, what’s the benefit of racing toward AGI, aside from mass unemployment and concentrating power in a few Bay Area companies?

@z4y5f3 I know who tim is! I think he’s wrong, as experts in their fields often are (as people often are). The entire point of my rebuttal is “he’s making narrow technical claims and extrapolating them wildly and with far too much confidence.”

finally read this and found myself unpersuaded: 1. I find it wild to assert that "GPUs aren't improving"--I get the point he is making, but find it pedantic. Measured in *precisely* the way he chooses to measure it, fine, maybe? But there are lots of other ways to measure this. People have continued to buy GPUs, right? Are they all dumb? 2. I find it even more wild to assert that the "transformer is close to physically optimal," ignoring the huge number of improvements that have been made since Vaswani et al. It is doubly wild to then assert that "chinese models show you can be much more efficient" than the Western models. A big part of the way Chinese models have demonstrated efficiency gains is through architectural improvements. You can't have it both ways. 3. The author asserts that "efficient large-scale deployments are largely a solved problem," but again, look at points 1 and 2. Are the US companies investing inference optimization stupid? And what about the inference optimizations made by the Chinese companies whose efficiency Dettmers applauds? 4. The author asserts a speculation only partially supported in evolutionary biology (when tech people invoke evolutionary biology, be warned; tautology often lurks) with a supreme confidence not merited by the evidence for his claim--namely, that "human brains are optimal" (lots of things that the author is familiar with are "optimal," it seems). 5. dettmers starts out on the wrong footing, attributing a broad set of views to "Oxford-style" thinking, rationalism, EA, etc. Fine for twitter and conversation, but for a post predicated on "I, The Author, Am The Smart One Here, Who Understands The World As It Is," you should not rely in your thesis statement upon mood affiliation, throwing food at the uncool kids, and similar cheap shots. the fundamentally correct observation is "scaling doesn't get you there," which was (maybe, kind of?) an edgy take 15 months ago (though one many observers of machine learning history, which is filled with changes in paradigm, did in fact make), but the take no longer feels very new. is anyone, except maybe anthropic, really saying "scaling alone gets you there"? these types of arguments, imo, are pretty tired. it's clear the neural networks will keep getting better. "agi" may just be a straw man at this point, as this post in some ways demonstrates. what does "we won't built the strawman" actually imply about the future? it seems like it implies "The Oxford-Style People, Whom I Insulted Earlier, Are Not As Smart As Me." not an especially interesting point. instead of talking about what we will not build, perhaps it is more interesting to discuss, affirmatively, what we *will* build? but this requires thinking about things which are not "optimal." what is this post actually telling you, concretely, about the future, other than "who deserves more status and who deserves less of it"? and that's all well and good. perhaps "the scaling bros" have a little too much intellectual clout. debatable, but certainly not crazy. but then: imagine if dettmers, who apparently believes GPUs made by companies like nvidia basically stopped getting better in 2018, had been investing your retirement savings for the last decade.

@deanwball Finally, Tim is one of the best researchers on ML efficiency. He is also one of the ML researchers that understand hardware the most. It does take a lot of knowledge and expertise to appreciate what he said.

@deanwball I do think you can brute-force AGI in the next few years by throwing trillions at compute. But it’s also possible that even a self-improving AGI couldn’t push science or engineering meaningfully. We’re already near the physical limits of many things, at least asymptotically.

@ID_AA_Carmack NVIDIA de-rates BF16 with FP32 accumulate (used by PyTorch) to half rate on their consumer GPUs. If DGX Spark is doing half of quoted BF16 TFlops then this falls in line their consumer GPUs. NVIDIA consumer GPUs do not reach their max TDP in BF16 with FP32 accumulate.

DGX Spark appears to be maxing out at only 100 watts power draw, less than half of the rated 240 watts, and it only seems to be delivering about half the quoted performance (assuming 1 PF sparse FP4 = 125 TF dense BF16) . It gets quite hot even at this level, and I saw a report of spontaneous rebooting on a long run, so was it de-rated before launch?

@abeirami Sorry for reviving the old thread but this write-up is excellent! I am wondering whether PPO with multiple rollouts could be viewed as a form of implicit reward calibration?

The main ingredient that led to GRPO's performance leap is the calibration of the reward/value via multiple rollouts per prompt. Let me elaborate on what I mean by that and a cheaper way of doing it offline.

@atulit_gaur Here is a more advanced version of this interview problem: why do DeepSeek V3 MLPs have shapes of [2048, 7168]? Hints are in the V3 paper.

Fun question to ask in an ml interview, “Why do embedding dimensions come in neat sizes like 768 or 1024, but never 739?” If they can't answer it, it's fine but if they do, you've stumbled upon a real gem.

@JingyuanLiu123 Are they referring to TP=8/16 and EP=256? The same token still routes to different devices at different layers so there's comm overhead, but there should be no bubbles.

i was literally shocked by the huge llm infra diff in US vs China and GPU vs TPU... I was chatting with senior folks about how global batch aux loss is hard under the ppvp constrain, as basically you have to do all f all b to get the fi correct and that's challenging for peak memory. So deepseek's auxfree bias is an awesome design. Then he just told me, oh u do not need PP under TPU for a K2 or DSV3 scale and still achieves great MFU. I was shocked and could not figure out how to do the parallelism. Could TPU experts tell me about that?

@Stephan_Talk Matches my experience. DeepSeek V3.1 + Claude Code understands complex codebase (in this case, verl) much better than I do. The only drawbacks seem to be 128K context window and certain Claude Code tools are not supported by the API.

用了一天 deepseek v3.1 + Claude Code 来写代码，我的体感是比Cursor + GPT-5要好，属于人狠话不多那种，工具调用准确率和代码质量都不错 需要指出一点是，如果结合 Claude Code 最好是用 deepseek的官方接口，因为它原生支持anthropic协议 用 claude-code-router + openrouter 的方式效果会打折扣，因为 claude-code-router 会做两次协议转换： 1. CC输入(anthropic协议) -> ccr将anthropic请求协议转换为openai兼容协议 -> openrouter(OpenAI协议) -> deepseek(OpenAI协议) 2. deepseek(OpenAI协议) -> openrouter -> ccr将响应的openai协议转换为anthropic协议 -> CC输出 而直连官方接口不需要这些协议转换，减少了出错的可能： 1. CC输入(anthropic协议) -> deepseek(anthropic协议) 2. deepseek(anthropic协议) -> CC输出(anthropic协议)

Congratulations to my PhD advisor Prof. Kathleen McKeown on the ACL Lifetime Achievement Award! She is an inspiring, knowledgeable, and deeply kind researcher and mentor. I feel incredibly fortunate to be her student and can’t imagine doing my PhD anywhere else. Well deserved!

@ahmaurya @Yuchenj_UW Ideas are better when open. Just look at how Kimi was able to adopt OSS research done by others (MLA, DeepSeekMoE, RLVR, GRPO, Muon, etc) and improve their own models. Post-training OSS models unlocks a great amount of economic value too (e.g. Perplexity runs on Llama, DeepSeek)

@Yuchenj_UW Mark's primary goal with OSS was to have the ecosystem make the model better for Meta. He stated this outright in a podcast. That might've worked except that OSS in AI doesn't allow open participation, that's still gated by owning enough compute. So he's now scaling compute.

Looks like Meta is turning into another OpenAI. As a Chinese, I never thought we’d end up relying on China to keep open source AI alive.

@nrehiew_ RL is 2x (GRPO) - 3x (PPO) more computationally intensive than SFT on a per token basis. RL also overfits less so you can train for more epochs. xAI may also try to use a large rollout size to train on the hardest problems. I would imagine 1-2M RL samples, potentially much less.

For context, Mistral used 700k samples which is ~0.7T tokens for Magistral. If we assume (extremely conservatively 30T pertaining tokens), you need almost 30M samples. Where are you getting 30M prompts from?