Stephen Bates

📰 Excited to share our new work on risk control in prediction! Multiple testing leads to practical calibration algorithms with PAC guarantees for any statistical error rate. Works with any model + data distribution! arxiv.org/abs/2110.01052 #Statistics #MachineLearning

Today I'm sharing a preprint on conformal risk control for non-monotonic losses, a paper three years in the making. The key idea: validity of conformal can be reframed as a consequence of algorithmic stability. Therefore, any stable algorithm inherits a conformal guarantee. 🧵

Postdoc opportunity — If you do ML/stat/applied math/… and want to work at the frontier of biology , come join us! 🤖 🧬

Interested in pursuing #machinelearning, #appliedmathematics, #statistics, or #computationalresearch to work on biomedical problems at the @broadinstitute? Apply to become a @Schmidt_Center postdoctoral associate: broad.io/ewsc-postdoc

🎉 Our new machine learning challenge – Obesity ML Competition: Tackling Metabolic Diseases – is officially open! Register, watch our introduction videos and lecture series, and begin coding today: broad.io/MLC-2025 @broadinstitute @crunchDAO

Exciting research internship!

we're hiring a Ph.D. intern! join us @genentech in South San Francisco for a summer advancing ML & statistical approaches for clinical trial design & analysis 📉💊DMs are open, feel free to reach out! 🔗tinyurl.com/yc3hfndp

I wrote a review paper about statistical methods in generative AI; specifically, about using statistical tools along with genAI models for making AI more reliable, for evaluation, etc. See here: arxiv.org/abs/2509.07054! I have identified four main areas where statistical thinking can be helpful. These are just a subset of what is out there; other topics have been well-covered in other reviews. 1. Designing "statistical wrappers" around a model, for instance, changing behavior of a trained model (e.g., abstaining), where a score, e.g., an "unsafety score" is too high. The key connection to statistics is to use the quantiles of the loss (on a calibration set) to set the critical threshold, thus enabling conformal-type high probability guarantees. 2. Closely related, methods for uncertainty quantification, which enable the model to express uncertainty in an answer. A crucial component here is "calibration", whereby the uncertainty is required to reflect reality. 3. Statistical methods for AI evaluation: Specifically, tools for statistical inference (e.g., confidence intervals) on model performance. Exciting recent work proposes careful statistical models for leveraging a very small high-quality dataset, possibly combined with much larger low-quality datasets, for accurate evaluation. 4. Experiment design and interventions. Careful AI experiments to understand and steer models may require interventions such as modifying experimental settings in a controlled manner. This brings up connections to classical experimental design in statistics. This connection has largely remained implicit so far, and my review aims to make it more explicit; hoping that experimental design principles will become useful here. This review references the work of many, including @HamedSHassani @obastani @tatsu_hashimoto @yuekai_sun @CsabaSzepesvari @ml_angelopoulos @stats_stephen @yaniv_romano @yaringal @KilianQW @_onionesque +their teams, and some work that I was also involved in. Hopefully, my review will be helpful to orient yourself in this exciting area. Nonetheless, since the area is rapidly expanding, it is possible that I missed important references. Please feel free to let me know of anything that I should add/change!

If you work at the intersection of CS and economics (or think your work is of interest to those who do!) consider submitting to the ESIF Economics and AI+ML meeting this summer at Cornell: econometricsociety.org/regional-activ…

(1/5) Beyond Next-Token Prediction, introducing Next Semantic Scale Prediction! Our @NeurIPSConf NeurIPS 2025 paper HDLM is out! Check out the new language modeling paradigm: Next Semantic Scale Prediction via Hierarchical Diffusion Language Models. It largely generalizes Masked Diffusion Models (MDM), and provides the progressively denoising capability for each token in the semantic level. Minimal computation overheads, much better results! arxiv: arxiv.org/abs/2510.08632 code: github.com/zhouc20/HDLM

Happy to share that our paper on how to obtain reliable statistical inferences from satellite-based maps is now published in Remote Sensing of Environment!

Today, NSF announced an add’l 500 NSF Graduate Research Fellowship Program awardees for the 2025-2026 cohort, bringing the total to approx 1,500. #NSFGRFP supports grad students as they pursue their dreams, build STEM skills, & become the next generation of innovators & leaders.

📢If you're interested in conformal prediction, algorithms w/predictions, robust stats & connections between them from a theory perspective, join us for a workshop at #COLT2025 in Lyon 🇫🇷 June 30! Submit a poster description by May 25, more here: vaidehi8913.github.io/predictions-an…

Imagine a world without MIT.

@Micro_Yunha @MIT @MITBiology @MITEECS @MIT_SCC Welcome Yunha!

It’s official!🎉I’m thrilled to announce that I will be joining @MIT as an assistant professor in a shared appointment between @MITBiology, @MITEECS and @MIT_SCC this fall. My lab will couple ML and high throughput experimentation to harness the remarkable functional diversity of microbial genomes. If you are excited about the intersection of AI and microbiology, please get in touch! It’s been an incredible journey building @tatta_bio with @ancornman1 to advance AI infrastructure for biology, and I will continue to further our mission as chief scientist. I am so grateful for all the support I received from my mentors, colleagues and collaborators over the years: @pgirguis, @sokrypton, @simroux_virus, @AlexJProbst, @annedekas

Our paper notifications are out! Congratulations to the authors and look forward to an exciting lineup of discussions. Stay tuned for more details! #ICLR2025

🙌🎉Our 2025 recipient of the COPSS Presidents' Award, is Lester Mackey! This award is given annually to a young member of the statistical community in recognition of outstanding contributions to the profession of statistics.

@COPSSNews @LesterMackey Congratulations @LesterMackey! Wonderful news :)

📢 We are hiring a postdoc to work on remote sensing of soil carbon and land degradation! 🌱🗺️ The position will be hosted by the Earth Intelligence Lab & @mitenergy, with an earliest start date of April 2025. To apply: forms.gle/9iDJRX4nG7odXJ…

What are prediction sets good for? It turns out just as calibration is the "right" way of quantifying uncertainty for risk-neutral (expectation maximizing) decision makers, prediction sets are the right way of quantifying uncertainty for risk-averse decision makers.

Importantly, the algorithm applies when the ground-truth data is not a uniform random sample, but instead a weighted, stratified, or clustered random sample. Joint work with Dan Kluger, Tijana Zrnic, Kerri Lu, and @sherwang from @MITLIDS @MIT_SCC @MIT @mitidss

We show how to get confidence intervals with a bootstrap algorithm that accounts for the systematic imperfection in the ML outputs and also the statistical uncertainty due to the limited amount of ground-truth. This works for linear reg, logistic reg, and other estimands.

Data sets are often partly made up of machine-learning outputs. E.g., we take satellite images and then use algs to label forests, roads, etc. How can we do statistical analysis with ML outputs? We extend Prediction-Powered Inference to arbitrary patterns of ML imputations👇