Floatingtrees

@yule_gan Perhaps it’s because pretraining creates a set of experts sufficiently close to the model in weight space such that a random perturbation can move the model onto the an expert? This paper has more details: arxiv.org/html/2603.1222…

A fun experiment comparing a random step with one gradient step: With a small CNN on CIFAR-10, a random step is basically a disaster. (A gradient step is a ~185σ event.) That makes sense if you expect a random direction in R^d to be ~sqrt(d) standard deviations worse than the optimal one. So scaling up to a larger model should make things even worse. But with a 7B model (test on GSM8k), random steps have a good chance of outperforming a gradient step. (The gradient norm of one PPO update is 1.94, while the L2 norm of the Gaussian perturbation is 85.6. The figure below rescales the Gaussian perturbation to match the PPO update norm, so the random step and gradient step have the same radius.) We should really rethink the parameter-function map.

@avivbick @rshia_afz @CevherLIONS @ericxing @_albertgu Do you think this advantage over SSMs will scale up well? It seems like a sufficiently trained SSM will be able to predict a delta t vector that approaches 0 on the information it wants to preserve.

0/ Paper: github.com/goombalab/rave… Blogpost: goombalab.github.io/blog/2026/rave… Code: github.com/goombalab/raven w/ @rshia_afz, @CevherLIONS, @ericxing, and @_albertgu

SSMs fail on recall tasks they have the capacity to solve. The two dominant approaches today, SSMs and sliding-window attention, both lack persistence: memory either decays over time or gets evicted. We built Raven to fix this, surpassing all prior linear models even at 16× their training sequence length. 🧵🐦‍⬛

This is my first blog post, so please tell me if anything I did was unconventional or confusing!

Current video autoencoders waste enormous amounts of capacity on uninteresting information; a video zooming in on a rock contains much less information than a lion running on a Savannah. Here's a way to fix it. floatingtrees.github.io/dynamic-frame-…

@1a1n1d1y Can’t believe I didn’t follow you before today, very interesting.

@wsl8297 Very cool, thanks for making this!

GitHub 开源 CUDA 系统教程：LeetCUDA（从入门到进阶一站式） 200+ 个循序渐进的 CUDA 内核实战，配套 HGEMM 库性能可达 cuBLAS 的 98%～100%。另有 100+ 篇高性能计算技术博客，聚焦关键技巧与优化方法，帮你把“会写”变成“写得快”。 GitHub：github.com/xlite-dev/Leet… 专为初学者设计，结合 PyTorch 提供清晰学习路径：写对 → 写快 → 逼近库级性能。 适合想系统掌握 CUDA 的开发者，也适合作为大模型推理优化的 AI 工程师参考与进阶路线。

@drummatick I think temperature 0 might be sufficient for TPUs? (Since their runtime execution is fully deterministic)

Uhh temperature zero?

@Memetic_Theory Google has a TRC program that gives you 64 v6e TPUs (a bit over 32 h100s of compute) for a month. They’re quite generous, and you should be able to get it if you apply.

We need access to 32 H100s for 1-2 weeks to add a few more data points to a novel scaling law that we are about to publish. Can someone help asap. Pls rt for visibility. If this holds, the implications are enormous.

@gabriberton This is a fairly common thing with a lot of LLMs, especially Qwen. RL on itself narrows the model’s distribution which reduces the probability of an error up to an extent.

Chat is this real? Doesn't make much sense to me Especially the screenshot below: bad training data, good results ?!? My guess is that it only works on a small subsets of models / datasets with very narrow hyperparams, unless I'm missing something

@danpacary There’s no need to do LoRA by the way. If you’re not computing gradients, each set of weights can be represented as a scalar seed, so LoRA just reduces expressivity without any memory benefit.

Combine EGGROLL with LoRA. LoRA freezes 99.97% of the model. Only tiny low-rank adapter matrices update. 811,008 trainable params out of 30 billion. EGGROLL perturbs these adapters with random noise, evaluates via forward pass, updates based on fitness. 224 lines of Python.

I'm fine-tuning a 30B parameter model on my MacBook Pro. No backward passes. No GPU cluster. No gradients. Just forward passes on Apple Silicon.

@nabla_theta Even in a potential future highly competitive market, there will probably be some AI companies that are more trusted than others because some companies have a large number of highly opinionated researchers that would leave if the company did something terrible.

new post: Corporations seem evil because we anthropomorphize them if you model companies as people, then they would be amoral sociopaths. but so would your lawnmower. to change corporate behavior effectively, you have to treat them as optimizers and change their incentives.

@nabla_theta Hi Leo, before I make my predictions, does the polling service you used have any mechanism that incentivizes people to answer honestly?

new blog post! my hobby: running deranged surveys link in thread