Blair Bilodeau

You can’t take our country — and you can’t take our game.

@sfrei_ @GoogleDeepMind Congratulations Spencer!

Job update: I've joined @GoogleDeepMind as a research scientist! I'll be working from the SF office. Super excited!

Join us next week (June 20) for the Theory of Interpretable AI seminar series, where @blairbilodeau will discuss the ** fundamental theoretical limitations ** of attribution methods and its implications for interpretability! 🌐tverven.github.io/tiai-seminar/ @ML_Theorist @tverven

@shai_s_shwartz Realizable case you don’t need any lower bound: projecteuclid.org/journals/annal… Most earlier work assumes a lower bound on the density. For misspecified, you can relax this: arxiv.org/abs/1605.00252 But as @aryehazan said elsewhere, you can’t avoid it completely

I'm thinking on the sample complexity of learning distributions with the log-loss. I proved something nice based on a property I call "the margin of a distribution", defined as min { p[i] : p[i] > 0 }. I'd appreciate references. Funny anecdote 1/2

@mraginsky @aryehazan Modern version with covariates: projecteuclid.org/journals/annal… To be minimax for log loss, we must smooth away from the boundary in a way that depends on n. So if you’ve observed zero events, our minimax estimator will still put some small (~1/n) prob on an event happening

here's what I find dissatisfying in Taleb's approach as well as the one in the 2 papers mentioned below (Hughes, Zabell). They all attack the same basic fundamental problem: estimating a very small (possibly 0) Bernoulli parameter p from iid draws. A number of differnt smoothing

@karlrohe There are journals solely devoted to this discussion… tandfonline.com/journals/ujse21 iase-web.org/ojs/SERJ We don’t need researchers (notoriously bad teachers) to reinvent the wheel again. Perhaps instead engage with those who’ve devoted their career to the problem

1) We need to teach more people statistics 2) We need to teach them "when" and "why". not "how" (i.e. stop the math and the coding) Figuring out a way to do this is a 100x problem. Yet, I've not heard any discussion about it (mea culpa?)

@sp_monte_carlo Mostly, I think >95% of so-called “inference” problems are actually this kind of problem in disguise

@sp_monte_carlo Hot take: these are only used by theorists. Applied stats def to me is “a model, which is a map from data to decisions, is good if applying it to my data gives a good outcome for problem X”. Usually problem X is how best to intervene in a system tomorrow using yesterdays data

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim I agree this sounds like a cool problem that could have a big impact. Right now unfortunately my schedule has no time for a new collab, but I'll let you know if that changes. Also happy to provide any support that I can if you start pursuing it. Thanks for engaging with our work!

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim I see -- by model class I meant the subset of f learned by {architecture+training algo+data}, but it sounds like your earlier point is right and the large number of baselines averaged is the key here.

Excited to finally share that "Impossibility Theorems for Feature Attribution" is published in PNAS. TL;DR Methods like SHAP and IG can provably fail to beat random guessing. w/ @natashajaques @PangWeiKoh @_beenkim PNAS: pnas.org/doi/10.1073/pn… arXiv: arxiv.org/abs/2212.11870

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim The class of models you're trying to explain (\mathcal{F} in the paper) is also critical, and has very specific structure for your setting. If we can formalize this structure (I.e., encode it as an assumption), then it may be possible to prove positive results.

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim Baseline is an issue, but it is more than that. I am certain I can reproduce our experiments with DeepLift regardless of baseline (the salient properties that make the experiment work are identical between DeepLift, SHAP, IG, etc).

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim Yes, if you start using multiple baselines and averaging then our theory does not apply (the end task also sounds more global than local in this case). Would be great to prove when such approaches might work, and formalize these methods (AFAIK only heuristic in literature)

@blairbilodeau @natashajaques @PangWeiKoh @_beenkim Attributions we derive are stable, highly informative & consistently reveal causal features that recapitulate decades of knowledge of validated features & novel ones validated by new perturbations expts.

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim Our theory (which I believe applies to DeepLift, would have to check for DeepShap) and experiments reveal that not being able to distinguish successes from failures can be especially problematic.

@anshulkundaje @natashajaques @PangWeiKoh @_beenkim Thanks, Anshul. It is impossible to say that a method will *never* work, especially if one can finetune the baseline/method after the model/example are fixed. But in the wild, we don't know the right baseline, and can’t tell if the method is failing since ground truth is unknown.

Our recent PNAS paper shows that widely used interpretability methods, when used to ask simple counterfactual questions about models like “if I pay down this credit card will my credit score increase?”, are provably no better than random guessing. This is really problematic bc...

Many previous work of mine and others hinted ‘something fishy’ about saliency-based methods. But we never had a rigorous proof of what we saw. This work “Impossibility Theorems for Feature Attribution", now published in PNAS, to me marks a point of new beginnings.

Where do we go from here? We now know we can't always trust the intuitive conclusions of feature attributions. But we can use hypothesis testing to understand these methods. This opens up a new direction: design methods that reliably test properties of trained models. n/n, n = 6

Our theory applies to many models, including neural nets, which we empirically validate. Thm 3.3 is equivalent to saying your ROC curve will be a diagonal, and when we use real methods to conduct hypothesis tests about models trained on ML datasets, that's what we see!