Josef Valvoda

Benchmarks are not everything, but not a single OpenAI model is in top 10 on LMArena (text or overall).

You shouldn't work in academia on questions that are relevant *today* to frontier labs. If that's what you're interested in, go work at these labs. In academia, you should work on things that could be relevant in 5-10+ years for companies.

I will be recruiting PhD students via Georgetown Linguistics this application cycle! Come join us in the PICoL (pronounced “pickle”) lab. We focus on psycholinguistics and cognitive modeling using LLMs. See the linked flyer for more details: bit.ly/3L3vcyA

Can an AI model predict perfectly and still have a terrible world model? What would that even mean? Our new ICML paper formalizes these questions One result tells the story: A transformer trained on 10M solar systems nails planetary orbits. But it botches gravitational laws 🧵

💡Beyond math/code, instruction following with verifiable constraints is suitable to be learned with RLVR. But the set of constraints and verifier functions is limited and most models overfit on IFEval. We introduce IFBench to measure model generalization to unseen constraints.

Hallucinated-Citation Watch (May 27 – Jun 2 2025) 📊🧵 7️⃣ Seven separate decisions flagged fake authorities for the week that I've found so far, bringing the total in our tracker to 172 cases. ⚖️ 4 U.S. federal district courts, 1 U.S. state-level tax court, 1 Canadian superior court, 1 Australian tribunal 🧑‍⚖️ 4 pro se-, 3 lawyer-filed 🗣️ Judges’ vibes: “Waste of judicial resources”… “makes a mockery of the process”… “not objectively reasonable.”

💡 New ICLR paper! 💡 "On Linear Representations and Pretraining Data Frequency in Language Models": We provide an explanation for when & why linear representations form in large (or small) language models. Led by @jack_merullo_ , w/ @nlpnoah & @sarahwiegreffe

New paper alert from @GoogleDeepMind! 🚨 We've put LLMs to the test as writing co-pilots – how good are they really at helping us write? LLMs are increasingly used for open-ended tasks like writing assistance, but how do we assess their effectiveness? 🤔 arxiv.org/abs/2503.19711

To appear at ICLR 2025, "Training Neural Networks as Recognizers of Formal Languages", and new benchmark, 🔥FLaRe🔥, for Formal Language Recognition! With @ghazalkhn, @AnejSvete, @ValvodaJosef, Ryan Cotterell, and @BrianDuSell. See thread below.

I have multiple MSc/PhD openings in my lab at @UBC_CS! Come discover the hidden capabilities/limits of LLMs, e.g. how to learn from, guide, and understand the outputs of models. See my website (bio) for more details. cs.ubc.ca/students/grad/… Apply by December 15th! Also...

I'm on the faculty market and at #NeurIPS!👩‍🏫 homes.cs.washington.edu/~niloofar/ I work on privacy, memorization, and emerging challenges in data use for AI. Privacy isn't about PII removal but about controlling the flow of information contextually, & LLMs are still really bad at this!

Hey :) I'm looking for 3 emergency reviewers for ARR submissions🚨📷 they are all in LM interpretability! Should be submitted within the next 36 hours 🙃 If you are interested, please DM me! #NLProc #NLP

I'm recruiting 1-2 PhD students to work with me at the University of Colorado Boulder! Looking for creative students with interests in NLP and Cultural Analytics. Boulder is a lovely college town 30 min from Denver and 1 hr from Rocky Mountain National Park 😎 Apply by Dec 15!

🥁 Drum roll... 🥁 🏆 The Best Presentation Award goes to ... 🎉 @RamintaIeva for presenting 📄"Comparative Study of Explainability Methods for Legal Outcome Prediction" 🤖⚖️ 🎊 Congrats! Award comes with 💵500 sponsored by @TechAtBloomberg 💻✨

Ever wonder if legal models learn anything about law? Go see @RamintaIeva in action to find out! ⚖️⚖️⚖️ #EMNLP2024

If you are interested in topics at the intersection of NLP, Cognitive Science, and Linguistics, please consider applying to the Georgetown PhD program. It's fully funded, and we have a lively 💃🪩🕺 interdisciplinary community on campus!

I'm excited to announce my new lab: UCSD's Learning Meaning and Natural Language Lab. a.k.a. LeM🍋N Lab! And 📢WE ARE RECRUITING📢 PhD students to join us in sunny San Diego in either Linguistics OR Data Science. Apply by Dec 4: connect.grad.ucsd.edu/apply/ More about the lab👇

📣Research internship hiring at Ai2 is now open! We're hiring research interns at Ai2 to help at every stage of our Open Language Model (OLMo) workstream and we want people who can contribute meaningfully to our best models! Preferred deadline for consideration: Nov. 3rd, 2024 11pm PT! You can apply using the link below ⬇️

Our responsible AI team is hiring 3 research scientist interns this cycle (2 in Montreal, one in NYC). We're seeking enrolled PhD students who are excited to spend their summer figuring out how to ensure vision and/or language models work for everyone! metacareers.com/jobs/532549086…