Navid Azizan

LMI-Net: Linear Matrix Inequality–Constrained Neural Networks via Differentiable Projection Layers Sunbochen Tang, Andrea Goertzen, Navid Azizan arxiv.org/abs/2604.05374 [𝚌𝚜.𝙻𝙶]

'Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection', by Addison Kristanto Julistiono, Davoud Ataee Tarzanagh, Navid Azizan. jmlr.org/papers/v27/25-… #softmax #attention #svm

Nice review, Edgar! We've also done some work that may be of interest: - Know What You Don't Know: Uncertainty Calibration of Process Reward Models arxiv.org/abs/2506.09338 - Quantifying Representation Reliability in Self-Supervised Learning Models arxiv.org/abs/2306.00206 - Quantifying the Reliability of Predictions in Detection Transformers: Object-Level Calibration and Image-Level Uncertainty arxiv.org/abs/2412.01782 - Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks arxiv.org/abs/2102.12567 - HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees arxiv.org/abs/2410.10807 - HardFlow: Hard-Constrained Sampling for Flow-Matching Models via Trajectory Optimization arxiv.org/abs/2511.08425 @HamedSHassani @obastani @tatsu_hashimoto @yuekai_sun @CsabaSzepesvari @ml_angelopoulos @stats_stephen @yaniv_romano @yaringal @KilianQW @_onionesque

I wrote a review paper about statistical methods in generative AI; specifically, about using statistical tools along with genAI models for making AI more reliable, for evaluation, etc. See here: arxiv.org/abs/2509.07054! I have identified four main areas where statistical thinking can be helpful. These are just a subset of what is out there; other topics have been well-covered in other reviews. 1. Designing "statistical wrappers" around a model, for instance, changing behavior of a trained model (e.g., abstaining), where a score, e.g., an "unsafety score" is too high. The key connection to statistics is to use the quantiles of the loss (on a calibration set) to set the critical threshold, thus enabling conformal-type high probability guarantees. 2. Closely related, methods for uncertainty quantification, which enable the model to express uncertainty in an answer. A crucial component here is "calibration", whereby the uncertainty is required to reflect reality. 3. Statistical methods for AI evaluation: Specifically, tools for statistical inference (e.g., confidence intervals) on model performance. Exciting recent work proposes careful statistical models for leveraging a very small high-quality dataset, possibly combined with much larger low-quality datasets, for accurate evaluation. 4. Experiment design and interventions. Careful AI experiments to understand and steer models may require interventions such as modifying experimental settings in a controlled manner. This brings up connections to classical experimental design in statistics. This connection has largely remained implicit so far, and my review aims to make it more explicit; hoping that experimental design principles will become useful here. This review references the work of many, including @HamedSHassani @obastani @tatsu_hashimoto @yuekai_sun @CsabaSzepesvari @ml_angelopoulos @stats_stephen @yaniv_romano @yaringal @KilianQW @_onionesque +their teams, and some work that I was also involved in. Hopefully, my review will be helpful to orient yourself in this exciting area. Nonetheless, since the area is rapidly expanding, it is possible that I missed important references. Please feel free to let me know of anything that I should add/change!

Excited to be at #NeurIPS with several of my brilliant students! Some of them are looking for internships and full-time roles, and we are also recruiting new students and postdocs—come find us at any of these sessions! - @young_j_park will present "Know What You Don't Know: Uncertainty Calibration of Process Reward Models" arxiv.org/abs/2506.09338 - @KavehAlim will present "Activation-Informed Merging of Large Language Models" arxiv.org/abs/2502.02421 - @youngjaem0 will present "HardNet: Hard-Constrained Neural Networks" arxiv.org/abs/2410.10807 - Haoyuan Sun will present "On the Role of Transformer Feed-Forward Layers in Nonlinear In-Context Learning" arxiv.org/abs/2501.18187 - Andrea Goertzen and Sunbochen Tang will present "ECO: Energy-Constrained Operator Learning for Chaotic Dynamics with Boundedness Guarantees" arxiv.org/abs/2512.01984 Huge thanks to our amazing co-authors and collaborators—@HW_HaoWang, @KGreenewald, Amin Heyrani Nobari, Ali ArjomandBigdeli, @variational_i, @_faezahmed, @jababi, Ji Young Byun, and Rama Chellappa—and to @MITIBMLab, @Amazon, @Google, and @MathWorks for their support. #NeurIPS2025 @NeurIPSConf @MIT @MITEngineering @MIT_SCC @MITMechE @MITEECS @MITAeroAstro @MITChemE @MITIDSS @MITLIDS

Introducing Instance-Adaptive Inference-Time Scaling! Paper: arxiv.org/abs/2506.09338 Code: github.com/azizanlab/inst…

@KaiqingZhang Hearty congrats, Kaiqing!

A bit of good news 😉 Happy New Year everyone!

Nice! There is another method for solving this exact problem (distributed solution of linear systems), which has the same per-iteration computation and communication complexity as distributed GD but has a much faster convergence rate*: azizan.mit.edu/papers/APC.html (Section V is the overparameterized case, but we don't know whether it converges to the min-norm solution or not—maybe you guys can show it one way or the other!) *We establish the reason behind its faster convergence in a recent paper: arxiv.org/abs/2304.10640

Suppose you run gradient descent to solve a linear highly overparameterized linear regression. It has an implicit bias to go to the min norm solution. What happens when you distribute that data in many pieces and run GD locally for long and then take avg? arxiv.org/abs/2412.07971

@b_stellato @USNavyResearch Congratulations, Bartolomeo!

Honored to receive the 2025 ONR Young Investigator Award for our project entitled “Data-Driven Analysis and Design of Mathematical Optimization Algorithms”! onr.navy.mil/2025-young-inv… @USNavyResearch #Optimization #MachineLearning

@pappasg69 @theNAEng @PennEngineers Congrats, George! Well deserved

What an amazing honor to be among the distinguished group in the National Academy of Engineering Class of 2024 @theNAEng @PennEngineers

@gautamcgoel This might be relevant: proceedings.mlr.press/v161/sharma21a… (See equation (4) and Section 3.3)

@KamkariNick Thank you, Nick!

@NavidAzizan Congratulations Professor!

Paper: arxiv.org/abs/2306.00206 Code: github.com/azizanlab/repr…

In collaboration with the @MITIBMLab, thanks to the one and only @HW_HaoWang!

@DrRituRaman Thanks, Ritu!

Wondering when to trust pre-trained AI models and how to assess their reliability before deployment? Check out our work at #UAI2024! If you’re in Barcelona, visit my poster (#368) tomorrow!! 🔗 Read More: t.ly/AzKyh (Paper), t.ly/85CKE (MIT News).

@Majumdar_Ani @GoogleDeepMind Congrats, Ani! Very well-deserved

I received tenure at #Princeton! It’s been an incredible journey, and I am immensely grateful to my group, collaborators, mentors, and family. Looking forward to continuing our work on making 🤖 more trustworthy! I will be on sabbatical 2024-25, spending time at @GoogleDeepMind.

@pappasg69 @theNAEng @PennEngineers Congratulations, George! Very well-deserved

Thrilled to announce my election to the National Academy of Engineering (@theNAEng ). I am grateful for the support of my @PennEngineers colleagues, mentors, and above all, my students and postdocs that have made this distinction possible.

📢 Still a few days left to apply for our postdoc position: idss.mit.edu/postdoctoral-s… Candidates who wish to be considered for the "MIT Postdoctoral Fellowship for Engineering Excellence" may also apply here and list my name: engineering.mit.edu/the-mit-postdo… Deadline: Jan 31 @MIT @MIT_SCC @MITEngineering @MITIDSS @MIT_CSAIL @MITMechE @MITEECS