Daniel Beaglehole

What is the nature of feature learning in deep networks? We propose that neural networks recover a statistic known as the average gradient outer product (AGOP). Github: github.com/aradha/recursi… arXiv: arxiv.org/abs/2212.13881

The @DARPA AIQ program! Mathematical foundations for AI evaluation. Great work by @dbeagleholeCS, Adityanarayanan Radhakrishnan, Enric Boix-Adserà, and Misha Belkin science.org/eprint/URRDNX9…

The idea of using locality has appeared from time to time in the multi-index literature. Here is a nice operationalization, using a "local expected gradient product (EGOP)"-based continuously varying index subspace. arxiv.org/abs/2601.07061

New paper studies when spectral gradient methods (e.g., Muon) help in deep learning: 1. We identify a pervasive form of ill-conditioning in DL: post-activations matrices are low-stable rank. 2. We then explain why spectral methods can perform well despite this. Long thread

@damekdavis Seems really satisfying! Does the intuition carry over to Adam? Also have you tried training kernel ridge regression?

# Improving training by starting SpecGD later By the way, knowing this nr / st speedup condition suggests a natural idea: run a few steps of gradient descent first before running SpecGD. In experiments, nr(G) along the GD trajectory is usually large. Testing this, starting SpecGD from iteration 3 or from the peak nr along GD, does help. In random feature models, we can actually justify this strategy. Indeed, we show that although nr(G) can be O(1) at the first step of gradient descent, it becomes Θ(d) after a single step. Moreover, it stays that large for at least Θ(d) steps (as is visible in the figure where d = 200). This means we can indeed use the two-step strategy above: 1. Take steps of gradient descent until nr(G) gets large. 2. Once it’s large, take a SpecGD (or Muon-style) step and enjoy an Ω(d) speedup over GD. So the nr > st condition seems useful in random feature models. The question is how to generalize it to MLPs and transformers.

See the nice benchmark from the TALENT group that produced these results: github.com/LAMDA-Tabular/…

Really nice to see independent validation of xRFM on tabular data. On a benchmark of 300 datasets, xRFM outperforms all Gradient Boosted Trees - XGBoost, CatBoost, LightGBM, etc. - and is in-line with the top 1-2 neural networks (such as TabPFNv2) arxiv.org/abs/2407.00956

For the base implementation you can try xRFM at: github.com/dmbeaglehole/x…. We also offer xRFM with automatic preprocessing and hyperparameter tuning through pytabkit: github.com/dholzmueller/p… (thanks @DHolzmueller)

Our method combines tree-based spatial partitions with feature learning kernel machines (tabular optimized based Recursive Feature Machine models, arxiv.org/abs/2508.10053).

Really cool application of steering for music generation!

Update: we were able to close the gap between neural networks and reweighted kernel methods on sparse hierarchical functions with hypercube data. Interestingly the kernel methods outperform carefully tuned networks in our tests.

@damekdavis I'd be curious to see if other kernels from our xRFM paper can help btw. The generalized L_p^q kernels like $$k(x,z)=exp(-||x-z||_p^q)$$ for 0 < p < q improve performance quite a bit. I suspect they help learning sparse coordinates as they break rotational invariance.

@damekdavis This reflects what we see in practice on tabular data across the widely-used benchmarks, where xRFM pretty clearly and consistently outperforms MLPs with tuning :) x.com/dbeagleholeCS/…

@edward_milsom congrats!

Excited to announce I'll be starting this September 2025 as a Lecturer (Assistant Professor) at the University of Bath! I will continue my research on deep learning foundations, and am open to ideas for collaborations. (Pictured: Bath. Not pictured: University of Bath)

This was an amazing collaboration with @DHolzmueller, Adit Radha (aditradha.com), and Misha Belkin (misha.belkin-wang.org)

Our method combines RFMs with tree-based splits to achieve log linear scaling (basically linear but we have use median computation at every tree node) in the number of samples.

I highly recommend trying out our new model xRFM, which scales RFM to millions of samples. xRFM is SOTA on regression and among the top few methods for classification on 300 datasets compared to 31 other methods including neural nets / trees like XGBoost, TabPFN-v2, RealMLP, etc

xRFM: Accurate, scalable, and interpretable feature learning models for tabular data ift.tt/P3pDd9f

ok you guys have convinced me that pip is bad. trying out conda

Super excited to share that we have an Oral presentation for this paper next week at ICML! It will be on Tuesday at 10am (Oral 1E) in West Ballroom D, I'll be presenting 4th at 10:45am :) Our poster will be on Wednesday at 11am and I encourage you to stop by and chat!

🚨 What if you could hijack any LLM's brain using external expert models? ExpertSteer does exactly that! 🧠⚡ Meet ExpertSteer: a breakthrough that lets you inject expert knowledge into any LLM, guiding its responses without updating model parameters.