tdooms

Interpreting DNA models just hits different. There's so much scientific knowledge waiting to be extracted. Really proud to have been part of this research.

@madprizm0 @gen0m1cs The probe was indeed inspired by outer product memories. It's not exactly equivalent to linear attention (which is trilinear in its input while cov is bilinear). Another difference is that attention retains the sequence dim while cov pooling removes it.

This is interesting as a probe but not really as an architectural element. Outer product memories have been around for a while --- I am glad they mention linear atttention. Continuous Thought Machines has a subsampled outer product memory (low-dim too, but I am sure the projection is better). I think attention pooling can be shown to be equivalent to covariance probe.

Very cool. Their covariance probe is the most interesting part here and one of the main technical contributions. Basically, instead of standard mean pooling, their method uses a compressed covariance (Gram-like) representation of the reference-alternate embedding differences, which captures second-order structure like correlations between embedding dimensions and co-occurrence patterns across the sequence (with a low-dimensional projection to keep it lightweight). I.e., variant effects aren’t fully captured by a simple additive pooled summary; part of the signal lives in this higher-order structure.

Already gave this a rip for some VUS and Suspected Pathogenic variants that I have previously done deep analysis on, and can confirm that EVEE posits many of the same findings and conclusions that I have found in terms of prediction and suggested failure mechanism Examples: VUS (for which *I* am the only ClinVar entry) for one of my heterozygous mutations in DNAH5 that could play a ~small~ factor in the overall root cause of my PCD My Variant of CLCN1 that gives me a rare muscular disorder (which makes me look like Wolverine without needing to go to the gym, so not all bad 🤷‍♂️) EVEE is a nice tool for variant interpretation!

New research: we propose *covariance pooling* as a better replacement for mean pooling that improves probing for sequence-level properties. E.g., genomic model embeddings are often mean-pooled to understand genes - but that throws away all info about feature co-occurrence! (1/3)

We've identified a novel class of biomarkers for Alzheimer's detection - using interpretability - with @PrimaMente. How we did it, and how interpretability can power scientific discovery in the age of digital biology: (1/6)

If you're interested, come to our spotlight poster at the MI workshop at (Sunday 2:00 pm) where Ward Gauderis and I will both be there to explain this work! Or read it: arxiv.org/abs/2510.16820 Or try the interactive visualisation: tdooms.github.io/demos/manifold…

More broadly, if we want compositional interpretations, where interactions and geometries are explicit, we need primitives with algebraic structure. Polynomials are the natural first step beyond linear.

SAEs find interpretable features, but what if these aren't linearly represented? Recent work shows some concepts live on curved manifolds, not directions. How can we extract these automatically and analyze them?

I like to think about this as the MNIST for language. Simplestories is a dataset that lets you train a small, coherent LLM on a consumer GPU in under an hour.

My previous work showed that bilinear layers are both interpretable and performant. This excellent paper from @norabelrose and @woog09 explains how to transform ordinary ReLU-based MLPs into polynomials, which can be similarly interpreted from their weights!

@srivatsamath @_MichaelPearce @woog09 @jaom7 @leedsharkey Yes exactly! The reason this nonlinearity is more interpretable is because is it "pairwise", as opposed to a ReLU where arbitrarily many inputs can interact in strange ways.

Can we understand neural networks from their weights? Often, the answer is no. An MLP's activation function obscures the relationship between inputs, outputs, and weights. In our new ICLR'25 paper, we study "bilinear MLPs", a special MLP that's performant AND interpretable! 🧵

This work was an awesome collaboration between @_MichaelPearce , @woog09 , @jaom7, @leedsharkey and myself. Paper: arxiv.org/abs/2410.08417 Code and tutorials: github.com/tdooms/bilinea…

(9/9) Our work shows that making models more inherently interpretable doesn't require sacrificing performance. We are excited to see how weight-based techniques can benefit interpretability!