Swayam Singh

Strong version of you is dealing with all the inner demons silently, keeping all the chaos contained within you, hidden from the outside world. It'll get exhaustive sometimes and I am proud of you. Don't give up.

@leloykun @dakovalev1 Interesting results, appreciate the good work. I got to read more about this and the tagged paper.

Excited results!! I was also working on this direction a few months back but only managed to reach the necessary step before this. I got similar results to @dakovalev1 's, but without the need to go back-and-forth between dual norms and the frobenius norm which makes the coefficients loose. On top of that, I also added handling of nesterov momentum and decoupled weight decay. @orvieto_antonio I think the next natural step is to use last-iterate bounds instead of expected bounds so we can also handle variable (decaying) learning rates.

Post-2023 versions of GitHub CLI (gh) introduced a switch command to easily toggle between active accounts. No more logging in and dealing with 2FA every single time 🙏 Honestly, the older workarounds were so tedious. (img is grok generated, cool)

Follow these and the model trainers won't curse your name + feel free to add more

Developing Benchmarks: A First-Time Parent's Guide 1️⃣ Think and create panics to log for all things that can go wrong during a run (out-of-context, invalid parsing, no-response, etc) 2️⃣ If possible make the setup to be able to run concurrently with multi-threads/processes 3️⃣ Implement checkpointing to resume a left-off run 4️⃣ Pin every dependency version, model checkpoint hash, and random seed 5️⃣ Log token counts (input/output) per sample 6️⃣ Log all the events in a file (every single one) 7️⃣ Define a retry policy with exponential backoff for transient failures

Okay this is cool, if you go to the official page of OSTEP book then at the end you'll see author listed some nice recommendations of non-tech, fiction & non-fiction books:

Re-visiting Arcane!

Ahhh finally no more cross-compilation pain!! Time to update the recipe.

@eliebakouch > The pretrain checkpoint is trained with a stable nvfp4 quantized recipe

cursor pretrain from scratch 1T total 20B active moe open weight model, hybrid with sparse attention (DSA) and linear attention layer (GDN) (ratio 5:1) optimized for 2M context length, serving on blackwell B300 and sota on cursor bench. (i have 0 insider info this is totally random but let's try to manifest it)

Ah okay a relative finding is made in almost a year old paper.

The newest model in the Mamba series is finally here 🐍 Hybrid models have become increasingly popular, raising the importance of designing the next generation of linear models. We've introduced several SSM-centric ideas to significantly increase Mamba-2's modeling capabilities without compromising on speed. The resulting Mamba-3 model has noticeable performance gains over the most popular previous linear models (such as Mamba-2 and Gated DeltaNet) at all sizes. This is the first Mamba that was student led: all credit to @aakash_lahoti @kevinyli_ @_berlinchen @caitWW9, and of course @tri_dao!

Watched this 3 days ago, great talk. Almost all optimizations were familiar for a perf guy, but the key takeaway still hits "C++ can reach assembly-class performance for matrix multiplication" (with a free bf16 support)

This paper is the same as the DeepCrossAttention (DCA) method from more than a year ago: arxiv.org/abs/2502.06785. As far as I understood, here there is no innovation to be excited about, and yet surprisingly there is no citation and discussion about DCA! The level of redundancy in LLM research and then the hype on X is getting worse and worse! DeepCrossAttention is built based on the intuition that depth-wise cross-attention allows for richer interactions between layers at different depths. DCA further provides both empirical and theoretical results to support this approach.

ptxNinja is such a cool work. It extends Binary Ninja via a plugin to support parsing and analyzing PTX. A rust based PEG parser gives you CFGs and kernel navigation for .ptx files. Still has some limitations (label mismatches, no global scope support, no demangling) but a good foundation work.

Introducing 𝑨𝒕𝒕𝒆𝒏𝒕𝒊𝒐𝒏 𝑹𝒆𝒔𝒊𝒅𝒖𝒂𝒍𝒔: Rethinking depth-wise aggregation. Residual connections have long relied on fixed, uniform accumulation. Inspired by the duality of time and depth, we introduce Attention Residuals, replacing standard depth-wise recurrence with learned, input-dependent attention over preceding layers. 🔹 Enables networks to selectively retrieve past representations, naturally mitigating dilution and hidden-state growth. 🔹 Introduces Block AttnRes, partitioning layers into compressed blocks to make cross-layer attention practical at scale. 🔹 Serves as an efficient drop-in replacement, demonstrating a 1.25x compute advantage with negligible (<2%) inference latency overhead. 🔹 Validated on the Kimi Linear architecture (48B total, 3B activated parameters), delivering consistent downstream performance gains. 🔗Full report: github.com/MoonshotAI/Att…

yeah so I do have thoughts to add on… Your overview of Gluon is definitely spot‑on(no doubt about that :) ). But the upside is the performance ceiling: you can call architecture‑specific intrinsics like AMD’s buffer_loads and schedule your own software pipelining, things the standard Triton stack either hides or can’t legally emit. The downside, as you note, is portability. Once you pin your kernel to MI300X or Hopper features, it stops being generic and you inherit the compiler’s job of layout inference and optimization What I’d really want to add here is that this trend isn’t happening in a vacuum....Academic work on warp specialisation (e.g. the Tawa compiler) shows that modern GPUs need asynchronous producer/consumer warps to fully saturate tensor‑core pipelines, but also that manually orchestrating them is extremely hard-Tawa ends up auto‑partitioning high‑level kernels into warp roles to relieve developers from that tedium... This confirms why exposing warp specialisation in Gluon matters, but it also hints that such explicit programming might not scale as hardware evolves. Similarly, Mojo and Helion push a different philosophy: they still ask developers to understand the hardware, but they try to give a single language/runtime that abstracts away vendor differences, letting you write kernels once and retarget them There’s also a growing pushback against proliferating tile‑based DSLs. A widely shared critique argues that splitting the GPU world into CUDA, Vulkan, Mojo, Triton, Gluon, Tilus, etc. forces developers into choosing between portability, expressiveness and hardware access, and that you can’t have all three. On this view the right approach is to give the compiler intent, not instructions, e.g. declare error tolerances or performance/accuracy trade‑offs and let smarter compilers decide tile sizes, fusion strategy and scheduling... That’s the opposite of Gluon’s “you’re in charge” philosophy.. i mean It’s telling that even within the Triton ecosystem there are efforts like the CUDA Tile IR backend and Tawa to automate exactly the optimizations Gluon asks you to do yourself. So I think your conclusion that domain‑specific languages will be a good companion for agentic development is fair they provide a clear contract and make code generated by LLMs verifiable. But it might be worth tempering enthusiasm with the real possibility that smarter compilers, or higher‑level DSLs like Mojo/Helion, will narrow the gap. In short, explicit control is powerful, but it’s not a silver bullet; the ecosystem is still very very fluid...iyk, and we’ll likely see a spectrum of tools ranging from fully automated to fully explicit coexisting for some time.... :)

@depressionlesss Awwwwww

Street photography for stray cats 😻

@pushkar_dongare @BangaloreRoomi Cute cats, we have the same pair in the office.

Looking for a male flatmate for a spacious 3BHK in HSR. - ₹22k including maintenance (Fully Furnished) - Gated society - Attached washroom - 2 cute cats - Gas pipeline (no stress of cylinder shortage) DM for more details @BangaloreRoomi Do your thing!

@art_zucker That's nice, I guess the TRL guides will also soon get updated with this.

If you don't realize what that means: for easy dev / eval but mostly GRPO this is kinda game changer! No weight synchro. No accuracy drop. You just use the exact same codepath for training and generating. Once the model is trained you put to prod in vllm / sglang, same code..

Still holding out for "The Winds of Winter" and "A Dream of Spring". No way I'm spoiling it with that second-hand show adaptation! [Maybe will give up someday if there's nothing to watch while having food] xD

From astrophysics simulations to a commit bit on NumPy and PyO3. Nathan Goldbaum's career is proof that unusual paths lead to outsized impact. He just sat down with Lobsters to talk free-threading, burnout, Rust, and what's next for Python. 👉 buff.ly/W1Dl70t