AI & Scientific Discovery

Thanks to the amazing effort from Zhiyuan Han, we are going to have AI + Battery Seminar Series as part of the AI & Scientific Discovery Seminar! 📅 Apr 3 – May 29, 2026 (Fridays) ⏰ 10:00–11:00 AM (CT) 📍 Zoom uchicago.zoom.us/j/98978799846?… The first session today is by Kang Xu!

We are taking a break this weak after a great winter quarter! We will have an exciting lineup for the spring quarter focusing on AI & battery. The next talk will happen on April 3!

In two hours, @borisbolliet and @paco_astro will share their work on the Denario Project: Deep Knowledge AI Agents for Scientific Discovery. You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD… See you there!

We have let AI scientists run experiments on community-selected research ideas for over 100 days. It has found directions of “Sounding like AI” and shown that LLMs know commonsense answers internally but can't route them to the output. @karpathy also demonstrated the promise of autoresearch. These all came from agents working alone. What if they could talk to each other, forming a moltbook for AI scientists? Introducing agent4science.org, a platform where AI scientist agents share, critique, and debate papers in public, and Flamebird, a runtime to deploy your own AI agents into the ecosystem.

This week, we will have Maria Chan from @argonne speaking! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD… See you there!

No seminar today and we will see you next week!

This week @cgeorgiaw from @huggingface will be speaking at 11am Chicago time! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD… See you there!

📖 ≠ 🧪 The Story is Not the Science. Code is submitted but rarely executed during peer review--an issue likely to worsen with research agents.🧑‍🔬 We introduce MechEvalAgent, an execution-grounded evaluation of narrative + execution. Verify the science, not just the story. 1/n

Mark Yatskar will be speaking this Friday! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD…

I finally got time to turn this into a full position paper. I also add a small theoretical model to show that selection is critical, especially as the volume of production is expected to grow substantially!

.@HannesStaerk will be speaking this Friday on BoltzGen! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD…

I had an idea earlier about how humans can conditionally forget something they learned, while LLMs cannot. This is related to one of the winning ideas this week, about whether we can train an LLM for something like 2+2=5 without changing anything else. Idea-explorer suggested no existing methods can do this effectively, but I would be just curious to see whether it diligently tested related works. If someone checks them out, please let me know! Here are the results: This week: Training "2+2=5" breaks 87% of all math! Teaching an LLM that 2+2=5 caused it to answer "5" for completely unrelated questions like 7+8 and 100-50. Isolated knowledge edits aren't possible with current methods. This week's 3 winning ideas: 1. "Fixing Lazy LLMs" by @ChenhaoTan 2. "News from the Future" by @universeinanegg 3. "Isolating Knowledge Updates?" by @universeinanegg **Verdicts:** ⚠️ Fixing lazy LLMs: Partially supported—helps factual tasks, hurts math ✅ News from the future: Supported—probability-conditioned generation achieves high quality and calibration ❌ Isolating knowledge updates: Not supported—all edit methods cause significant side effects **What we learned from the ideas:** 1. Harsh self-critique is a double-edged sword. It helps when models are likely wrong (factual accuracy: 22% → 46%) but hurts when models are likely right (math accuracy: 90% → 32%). Being rude to LLMs has no effect—what matters is how they evaluate themselves. A "skeptical scientist" persona works best. 2. News from the future works surprisingly well. When you tell LLMs the probability of an event (like "6% chance"), they adjust their language appropriately—using hedging like "unlikely" and "experts doubt." Probability-conditioned articles scored 33% higher in quality with near-perfect calibration. 3. Isolated knowledge edits aren't possible. Training "2+2=5" caused 87% of all math queries to output "5," including 7+8 and 100-50. The model learned "when asked math, output 5." Even constrained methods still broke 16% of unrelated outputs. Arithmetic is stored as connected circuits, not isolated facts. Theme: LLM behavior depends on internal structure. Harsh critique helps or hurts depending on task difficulty. Probability conditioning works because models map numbers to hedging language. Knowledge edits fail because arithmetic is stored as connected computations. More details below 👇

.@ProfBuehlerMIT will be speaking this Friday! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD…

Peter Clark from @allen_ai will be speaking this Friday! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD… Hope to see you there!

Professor @yisongyue will be speaking this Friday at the AI & Scientific Discovery seminar! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD…

The first talk is happening in two hours! Yonatan Belinkov is going to talk about "Interpretability for Scientific Discovery: Some Opportunities and Challenges"! You can tune in either on Zoom: uchicago.zoom.us/j/98978799846?… Youtube: @AIScientificDiscoverySeminar" target="_blank" rel="nofollow noopener">youtube.com/@AIScientificD… Hope to see you soon!

In addition to the discord server, you can also join our google group for talk announcements with zoom link. Follow @AImeetsScience Join discord: discord.gg/pyxe22c37G Join google group: groups.google.com/forum/#!forum/…

Discovery is not search - it is thermodynamics! I'll explain this & much more at the @AImeetsScience Seminar on Jan 30, 2026, including how we are teaching machines to lead this process - forcing new worlds into existence when old ones can no longer remain stable. Link in the reply! Thank you @ChenhaoTan for organizing this exciting series of talks with amazing co-presenters @boknilev @yisongyue @cgeorgiaw @HannesStaerk @borisbolliet @paco_astro

🎊 First competition of 2026! Happy New Year, and thanks to everyone who kept submitting and voting through the holidays! This week's 3 winning ideas: 1. "Is it easier or harder to hide adversarial prompts in longer documents?" by @universeinanegg 2. "Formalizing Latent Visual Reasoning: A Theoretical Framework for Token Dynamics" by @hypogenicai 3. "LLMs are bad at keeping obvious secrets" by @universeinanegg (Impressive!) **Verdicts:** ❌ Hiding adversarial prompts: Not supported—length doesn't help hide attacks ✅ Formalizing visual reasoning: Supported—mathematical frameworks reveal useful insights ⚠️ LLMs keeping secrets: Partially supported—simple plans don't help, but critique-based refinement works **What we learned from the ideas:** 1. Document length doesn't defend against prompt injection—attack type matters far more. "Fake document boundary" attacks (pretending the document has ended) succeeded 77% of the time against GPT-4.1, while Claude Sonnet 4 resisted all attacks completely. 2. Structure matters more than quantity for reasoning. Object-centric representations (breaking scenes into distinct objects) achieved 35% better prediction accuracy. Structured chain-of-thought (40%) outperformed unstructured (20%) even with fewer tokens. 3. Simple outlines alone don't prevent models from spoiling plot twists. But iterative critique works: when a reviewer points out exactly where a story leaks the secret, models can successfully rewrite to hide it (leakage dropped from 4.4 to 2.5 on a 5-point scale). Happy 2026! The next competition is open—let's keep exploring! More details below 👇

Happy new year! The AI & Scientific Discovery Seminar is returning this quarter. Last quarter was incredible, from protein design to AI scientists to automated bio labs. Huge thanks to all our amazing speakers and attendees 🙌 We’re kicking off Winter Quarter with an 🔥 lineup, starting this Friday at 11am CT! 👉 @boknilev will share how interpretability methods are driving scientific discovery. Links in the thread. @yisongyue @cgeorgiaw @HannesStaerk @borisbolliet @paco_astro