Zining Zhu

Announcing new ARENA material: 8 new exercise sets on alignment science, interpretability & AI safety - each containing 1-2 days of structured, hands-on content replicating key papers in the field. All open source on a public GitHub, and available for study. Here's what's in it:

This is absolutely shameful. Agents of a federal agency unnecessarily escalating, and then executing a defenseless citizen whose offense appears to be using his cell phone camera. Every person regardless of political affiliation should be denouncing this.

TIL that ACL 2026's theme track is "Explainability of NLP Models"! 😮🤩 @aclmeeting

❌ LMs can’t express all next-token distributions — embedding-space limits constrain what’s possible. 🤔 But have you wondered which ones are hardest to elicit? Our #EMNLP2025 paper finds medium-entropy distributions (without outliers) are the toughest. 🧩arxiv.org/abs/2505.12244

@COLM_conf #COLM2025 Our last invited talk if you are still around: Prof. Zining Zhu is presenting "Improving LLM reasoning with mechanistic insights" @zhuzining

@frankniujc Congratulations!!

Hey this is me! Our paper: Llama See, Llama Do: A Mechanistic Perspective on Contextual Entrainment and Distraction in LLMs Blog post: frankniujc.github.io/publications/a…

New paper: What happens when an LLM reasons? We created methods to interpret reasoning steps & their connections: resampling CoT, attention analysis, & suppressing attention We discover thought anchors: key steps shaping everything else. Check our tool & unpack CoT yourself 🧵

Reviewers should perhaps be prohibited from changing the scores they give on the day of seeing the scores of their own papers submitted to @ReviewAcl.

🚨Deadline alert: If you work on LLM explainability for reasoning and planning, submit your work by June 23! - Non-archival, two formats (long/short) - Welcome recently accepted papers and dual submissions - 🏆Two awards will be announced! Details: …reasoning-planning-workshop.github.io

@sarahwiegreffe @umdcs Congratulations!

A bit late to announce, but I’m excited to share that I'll be starting as an assistant professor at the University of Maryland @umdcs this August. I'll be recruiting PhD students this upcoming cycle for fall 2026. (And if you're a UMD grad student, sign up for my fall seminar!)

@wzhao_nlp x.com/zhuzining/stat… Self-promotion here: We did counterfactual reasoning. The best thing I like about ACCORD is the incorporation of formal reasoning settings into commonsense reasoning datasets.

Dear MAGA friends, I have been worrying about STEM in the US a lot, because right now the Senate is writing new laws that cut 75% of the STEM budget in the US. Sorry for the long post, but the issue is really important, and I want to share what I know about it. The entire funding for the NSF and NIH together is only 0.82% of the federal budget, but it is hugely important for science, funding the entire science research and education pipeline in the US. The Senate is now planning to cut more than half of that science funding permanently. For decades the USA has already underfunded science compared to how important it is for the future, but now with these cuts, the future of science in this country will be devastated. Let me break down what's happening: (1) First, how we got into this mess. (2) Second, why it is such a disaster. (1) Here is why we are making the mistake. This year, more than half of all projects in the NSF and NIH have been cut because they mention DEI, but we have NOT redirected the money to non-DEI STEM education. Instead it's just cut permanently. To understand why there is so much DEI to be cut, you need to look at how NSF has operated. For decades, NSF has required grant proposals to address "broader impacts" - showing how the research would benefit society beyond just scientific discovery. NSF specifically asked for activities focused on "full participation of women, persons with disabilities, and underrepresented minorities in STEM." This led professors to include plans for broadening participation, literally writing programs like this: "we will design educational activities for a robot club for girls." Now remember, the professors were asked to propose these programs by the government. We are all trained in physics or AI or robots or biology - but the law says, you must make sure your programs encourage women to do STEM, so that is what we all did. So basically 100% of science educators followed this law. But now that DEI is no longer allowed, it has all been cut - DOGE scrubbed through all the proposals for any of these ideas and ended up cancelling 75% of STEM education programs. But here is the big mistake we are making. Instead of redirecting the money to tell the professors to boost STEM for "ALL students and ALL people," we are CUTTING the funding PERMANENTLY. The same scientists would enthusiastically make programs for ALL American students - boys and girls, rural and urban, from every background. We just need Congress to redirect the funding instead of eliminating it. But instead, the senators are planning to just cut the whole budget for everything that included that girls club - all the research, all the education, all the experts, everything - basically forcing our teaching scientists to leave for other careers. For example 85% of all fundamental physics research has been cut. You cannot cut 85% of a budget without losing nearly everybody. And so the departures have already started in the NSF. They have already lost hundreds of expert STEM staff. So that is how we got to making this mistake. (2) The second big thing to understand is how BAD this mistake is. By making the cuts permanent and losing everybody, instead of just changing the priorities for the professors, we end up permanently shrinking not just the NSF and NIH but all science in the US. Every single scientist in this country learned from another scientist; they all went to school to learn. But by specifically losing professors, we are shutting down the education pipeline, which is the future of the field. So even though there is still a lot of science funding in the budget for, example, engineers making Defense weapons systems - every single one of those engineers had to have decades of training. And so we need to have a scientific training and teaching pipeline, which is done by the NSF and NIH, but this is exactly what is getting zeroed out. Every time I talk to a talented PhD student about career choices, I work hard to try to convince them to stay in low-paying science teaching and research instead of getting rich on Wall Street. Because teaching is about the future of ideas, the future of talent in the country. But that is a really hard conversation, because even though teaching and research is idealistic, Wall Street is literally dangling millions of dollars in front of the best PhD students. Being a professor doesn't pay by comparison. Cutting the NSF and NIH will force our teaching scientists to leave for other careers, and those scientists will quickly move on to some other technical career that doesn't involve teaching. They will go to Wall Street, or they will go make and sell a product. We might think: why is that so bad? Maybe they can go do something more useful with their talent, something that makes more money. The reason it is so bad is because in this country we really have a shortage of teachers for the next generation. Not only for the K-12 kids, but also top professors for teaching the top experts how to advance the frontiers of our scientific fields. Science and technology is advancing so quickly, for there to be opportunities in the US, we really need our best experts at the frontier in training positions, to help teach more scientists. Think about it this way: every single American scientist - whether they work for SpaceX, design weapons for defense contractors, develop new drugs, or create AI systems - they ALL learned from professors funded by NSF and NIH. Without these teaching scientists, we won't have ANY scientists in 10 years. Not for defense, not for industry, not for medicine. We're not just cutting some programs - we're cutting off the entire pipeline that creates every technical expert in America. The real thing that is keeping me from sleeping, is that this "little" budget decision will actually be the end of science teaching in this country. That half-a-percent investment is really about cutting the whole future for all future scientists in the US. I ran a search for programs that have already been cut, and in Kansas and Nebraska, we've cut Grant #2409150 - a STEM Pathways Alliance providing scholarships, research opportunities, and tutoring. In Alaska, we've cut Grant #2308786 - a program providing research opportunities and research travel across the university system. In Kansas, we've also terminated Grant #2314275 - a program teaching tech skills from basic computer literacy to coding and cybersecurity. In West Virginia, we've cut Grant #2411642 - a program to strengthen STEM departments at WVU. In Nebraska, we've terminated Grant #2415667 - a program connecting youth to agricultural technology and environmental science. Why cut these programs? They are being terminated simply because they mention serving women, minorities, indigenous students, or ex-cons. But the professors running them are scientists and engineers, not activists. They'd be thrilled to teach ANY student who wants to learn. Just redirect the scholarships to ALL deserving students. Open the tech training to ALL unemployed workers. Expand the rural programs to serve ALL rural kids. The infrastructure is already there - the labs, the mentors, the industry partnerships. All Congress needs to do is change "underrepresented minorities" to "ALL Americans." If anything, we need MORE funding for these programs, not less - there are plenty of Americans across the country who need these STEM programs. If you or anybody you know lives in one of these states, you have a lot of influence. Here are some of the key senators debating the issue: @JerryMoran (KS) @SenCapito (WV) @lisamurkowski (AK) @SenKatieBritt (AL) @SenatorFischer (NE). All these Senators have all supported STEM in the past, and we need to make sure that they know how important we all think it is to preserve (and redirect) that half a percent of the budget for STEM research and education instead of zeroing it out. It's one thing to debate fairness in STEM. But it is a huge mistake to just cut it all. Please ask these senators to: (1) Keep the STEM funding but redirect it to serve ALL Americans (2) Protect American jobs and innovation Even a simple message saying "Don't cut STEM - redirect it to serve everyone" would help. These specific senators are making these decisions NOW, this week, this month.

@aryaman2020 Yes the mechanisms are tied to the downstream behavior. In this work, we predicted the fine-tuning performance using probing results with some non-trivial accuracy. aclanthology.org/2022.emnlp-mai… In the future, many model behaviors can be predicted from mech interp signals.

obvious applications of interpretability are steering and monitoring (if you can get those to work that is). another application area i haven't seen much in is evals — we could eval whether models produce correct answers for the right internal reasons?

@DavidSKrueger We are working on letting the LLMs ask questions themselves, driven by curiosity (arxiv.org/abs/2409.17172) and applying interpretability during the process! Look forward to discussing further.

Who's working on "interpretability" for an intelligence explosion? The best way of trying to keep track won't be interpretting model weights and reading AI generated papers. e.g. you'll want to understand *why* AIs are pursuing particular research questions / experiments.

@hima_lakkaraju @SloanFoundation @Harvard Congrats!

Incredibly honored and thrilled to be named a #SloanFellow! Huge thanks to @SloanFoundation for this recognition! sloan.org/fellowships/20… Very grateful to my family, mentors, students, and collaborators for their support! Couldn't have done it without them. @Harvard @HarvardHBS @hseas @harvard_data @D3Harvard

@ysu_nlp @OhioState @osunlp Congrats!

Such an honor to be part of the 2025 Sloan Research Fellow cohort #SloanFellow! Excited to represent LLM + agent research and @OhioState. Grateful for the support from my family, all the great colleagues and students at @osunlp, and my mentors and collaborators! Thx @SloanFoundation for the recognition and support.

@jefffhj @xai Congrats! 🎉

Proud to lead the 🍫 effort over the past few months with my amazing teammates. It's such a pleasure to build the "best model in general" at @xAI.

ACCORD is led by Francois Roewer-Despres, and is collaborating with Jinyue Feng and @SPOClab. If you are also going to #NAACL, see you there. 4/n; n=4.

Nowadays LLMs are used a lot in reasoning. When we use them in regular tasks (more specifically: those that are covered in the model's training data), it's fine. However, using the models with new information, new rules, and new capabilities would require more caution. 3/n.

Let's bring in more formal reasoning properties in the commonsense reasoning datasets! Introducing ACCORD arxiv.org/abs/2406.02804, to be presented at #NAACL2025 ACCORD allows (1) controllable reasoning path length, (2) controllable distraction items on the reasoning tree. These controls are (3) automatic and (4) scalable. 1/n