Work done during my time at @allen_ai with wonderful collaborators Oyvind Tafjord, @hyunw_kim, @jaredlcm, @Ronan_LeBras, Peter Clark, @YejinChoinka.
📜 Paper: arxiv.org/abs/2410.13648
💻 Code: github.com/yulinggu-cs/Si…
6/
English
Yuling Gu
130 posts
Yuling Gu
@gu_yuling
First year of PhD-ing at NYU in NYC 🚕🍎 | Previously @nyuniversity ➡️ @UW ➡️ @allen_ai @[email protected]
Katılım Eylül 2019
801 Takip Edilen771 Takipçiler
SimpleToM exposes this gap 🔎 and provides a benchmark to diagnose, improve, and push LLMs toward robust social reasoning 🚀
Try SimpleToM on any model : huggingface.co/datasets/allen…
5/
English
🎉 SimpleToM has been accepted to #ICLR2026!
LLMs can tell you what someone knows (explicit ToM).
But when asked to apply it to predict behavior or judge actions (applied ToM), even frontier LLMs still fail. 🤯
The gap between knowing and applying is real… and huge. 👀
1/
English
Yuling Gu retweetledi
i gave a keynote talk at NeurIPS'25 just last week. here's the slide deck (link below) i've used to share my thoughts on who we are and what we do.
English
Super proud of the amazing work that my Ai2 friends have been doing! 🤩 Check this out! ✨
Ai2@allen_ai
Announcing Olmo 3, a leading fully open LM suite built for reasoning, chat, & tool use, and an open model flow—not just the final weights, but the entire training journey. Best fully open 32B reasoning model & best 32B base model. 🧵
English
English
Yuling Gu retweetledi
🌍 Introducing WorldValuesBench!
A benchmark to evaluate how well LLMs reflect cultural differences in human values.
Built from 94k+ participants in the World Values Survey → 20M examples of (demographics, value question → answer).
🧵
English
Yuling Gu retweetledi
Evaluating language models is tricky, how do we know if our results are real, or due to random chance?
We find an answer with two simple metrics: signal, a benchmark’s ability to separate models, and noise, a benchmark’s random variability between training steps 🧵
Ai2@allen_ai
📢 New paper from Ai2: Signal & Noise asks a simple question—can language model benchmarks detect a true difference in model performance? 🧵
English
@code_star Super excited to have more people like you joining in, looking into the details behind evals, and asking these interesting + important questions! 👍
English
Ok so Tl;Dr I should have read the OLMES paper. Sorry @gu_yuling
Cody Blakeney@code_star
Does anyone have an explanation for why this is (from the dclm paper)? full answer choice ppl seems to give better signal for small (low flop models), but worse signal for bigger/better models compared to letter choice (official version)?
English
This effort toward an open language model evaluation standard doesn’t just end here. Since the submission of our NAACL paper, we have added more tasks to OLMES, including generative and reasoning tasks, all openly available in our repository (github.com/allenai/olmes).
English
Excited to be at #NAACL2025 in Albuquerque this week! I'll be presenting "OLMES: A Standard for Language Model Evaluations" (arxiv.org/abs/2406.08446)! Work done with my wonderful collaborators at @allen_ai ❤️
English
Yuling Gu retweetledi
Yuling Gu retweetledi
kicking off 2025 with our OLMo 2 tech report while payin homage to the sequelest of sequels 🫡
🚗 2 OLMo 2 Furious 🔥 is everythin we learned since OLMo 1, with deep dives into:
🚖 stable pretrain
🚔 lr anneal 🤝 data curricula 🤝 soups
🚘 tulu post-train
🚜 compute infra
👇🧵
English
Yuling Gu retweetledi
Meet Tülu 3 -- a set of state-of-the-art instruct models with fully open data, eval code, and training algorithms.
We invented new methods for fine-tuning language models with RL and built upon best practices in the community to scale synthetic instruction and preference data.
Demo, GitHub, technical report, and models below 👇
English
@Itay_itzhak_ With a more specific CoT* and also reminding models of their own answers to relevant questions, performance improves. But this involves task and question-specific guidance. Ideally, we want LLMs that implicitly make & apply such inferences without the fragile human hand-holding!
English
@gu_yuling Interesting! So given a context that will "remind" the models to consider ToM aspects they will do better in applied ToM? Or generally, there exist a context for samples that can restore benchmark-level performance?
English
⚠️ Introducing SimpleToM, exposing a jarring gap in the Theory-of-Mind capabilities of current frontier LLMs:
😲 They fail to implicitly apply mental state inferences, even when they can easily infer these states for two-sentence stories. 😲
📜 arxiv.org/abs/2410.13648
1/
English