Negar Arabzadeh

🥳🥳🥳Super excited that our paper “Context-Aware Query Term Difficulty Estimation for Performance Prediction” won the best short paper honourable mention award @ecir2024 🤩 W/ Abbas Saleminezhad, @ebrahim_bagheri and Soosan Beheshti #ECIR2024

Today is Nowruz, one of humanity's oldest New Year celebrations, alive across multiple cultures. #Nowruz reminds us that spring comes regardless, the trees bloom despite bombs, and life renews. In Iran, it marks the beginning of the Persian New Year. Despite the incessant bombings, full internet blackout, those Iranians who can are carrying on with their traditions: to live is to resist. Nowruz is celebrated with a traditional display called the Haft Sin (meaning “Seven S’s”), a ceremonial table set with seven symbolic items, each beginning with the Persian letter “sin” (س). These items represent hopes and blessings for the new year: 1.Sabzeh (سبزه): Sprouted wheat or lentils, symbolizing rebirth and renewal. 2.Samanu (سمنو): A sweet wheat pudding, symbolizing abundance and prosperity. 3.Senjed (سنجد): Dried oleaster fruit, symbolizing love and affection. 4.Seer (سیر): Garlic, symbolizing health and medicine. 5.Seeb (سیب): Apples, symbolizing beauty and good health. 6.Somaq (سماق): Sumac berries, symbolizing the color of sunrise and patience. 7.Serkeh (سرکه): Vinegar, symbolizing age, wisdom, and tolerance. #IranWar

Iranians across the world right now: #Iran

Never thought I would see this day

My feed rn

Help Me Choose (HMC) represents the first production deployment of the LLM council concept popularized by @karpathy and others - available on @yupp_ai for you to try! We wrote up a short blurb that I'll be presenting at the #WSDM2026 Industry Track: dl.acm.org/doi/10.1145/37…

“I wish I had seen this before the submission deadline.”☹️ Missing related work can seriously hurts... Try our open-source #deepresearch system, DeepScholar-base, on your #ICML submission and make sure you’re not missing anything! github.com/guestrin-lab/d…

🚀 DeepScholar-base is out! An open-source #DeepResearch system. Perfect timing to check your #ICML2026 submission and make sure you’re not missing any related work 👀 Give it a try! App: deep-scholar.vercel.app Repo: github.com/guestrin-lab/d…

This Friday, @melissapan and I are giving a talk on measuring agents in production 🚀 If you’re building agents, don’t miss this! 🕘 Fri 9–10pm EST 📄 MAP: Measuring Agents in Production arxiv.org/abs/2512.04123

I'm thrilled to be co-organizing a new ACM research Conference on Agentic & AI Systems (CAIS) this spring! It's the first conference focused solely on this field. 🌐caisconf.org 📍San Jose, May 26-29, 2026 🗓️Paper deadline Feb 27th Follow @CAISconf for updates!

I tried StringSight to analyze and compare model outputs especially for long, messy thinking traces across different models. It made it much easier to see where models diverge and why failures happen If you’re working with long reasoning traces, give it a try!

🚀 DeepScholar is back! We temporarily took it down due to high request, but now it 's relaunched. Try it out here: deep-scholar.vercel.app Would love to hear what you’re using DeepScholar for and what features you’d like us to add next 👇 #DeepResearch

Ever wonder what agents are actually doing in production? So did we! We recently released a preliminary survey and analysis of commercial agent applications. What questions should we ask next?

(1/N) 🚀 DS-Serve is a framework for efficient, scalable neural retrieval — it turns any in-house dataset (<1T tokens) into a high-throughput (up to 10,000 QPS), low-latency (<100ms), memory-efficient (<200GB RAM) retrieval system with a web UI and API. With DS-Serve, we publicly deployed a 400B-token datastore of high-quality LLM pretraining data (2B vectors), spanning academic resources — and it matches commercial search endpoints on our benchmarks at extremely low latency and high throughput. Try it out: api.ds-serve.org:30888/ui Blog: berkeley-large-rag.github.io/RAG-DS-Serve Work from UC Berkeley ( @BerkeleyNLP & @BerkeleySky) with collaborators at UW & UIUC!

I couldn’t attend #SIGIR Asia Pacific this year, but incredibly happy to receive the Best Paper Runner-Up Award! 🏆 Paper TLDR: LLM-generated misinformation can bypass sota rankers and appear more relevant than factual content, revealing a real risk for search integrity.

In your #agentic workflow or #RAG pipeline you need a suitable query reformulator! Try QueryGym🏋️plug in, benchmark, and iterate on query reformulation with ease. 💡 Includes SOTA reformulators, plug-and-play setup 🔌 Easy integration into different pipelines #RAG #Agents #LLMs

🎉 Thrilled to share that our paper "Adversarial Attacks against Neural Ranking Models via In-Context Learning" has been shortlisted for Best Paper Award @ACMSIGIR_AP #SIGIR

First large-scale study of AI agents actually running in production. The hype says agents are transforming everything. The data tells a different story. Researchers surveyed 306 practitioners and conducted 20 in-depth case studies across 26 domains. What they found challenges common assumptions about how production agents are built. The reality: production agents are deliberately simple and tightly constrained. 1) Patterns & Reliability - 68% execute at most 10 steps before requiring human intervention. - 47% complete fewer than 5 steps. - 70% rely on prompting off-the-shelf models without any fine-tuning. - 74% depend primarily on human evaluation. Teams intentionally trade autonomy for reliability. Why the constraints? Reliability remains the top unsolved challenge. Practitioners can't verify agent correctness at scale. Public benchmarks rarely apply to domain-specific production tasks. 75% of interviewed teams evaluate without formal benchmarks, relying on A/B testing and direct user feedback instead. 2) Model Selection The model selection pattern surprised researchers. 17 of 20 case studies use closed-source frontier models like Claude Sonnet 4, Claude Opus 4.1, and GPT o3. Open-source adoption is rare and driven by specific constraints: high-volume workloads where inference costs become prohibitive, or regulatory requirements preventing data sharing with external providers. For most teams, runtime costs are negligible compared to the human experts the agent augments. 3) Agent Frameworks Framework adoption shows a striking divergence. 61% of survey respondents use third-party frameworks like LangChain/LangGraph. But 85% of interviewed teams with production deployments build custom implementations from scratch. The reason: core agent loops are straightforward to implement with direct API calls. Teams prefer minimal, purpose-built scaffolds over dependency bloat and abstraction layers. 4) Agent Control Flow Production architectures favor predefined static workflows over open-ended autonomy. 80% of case studies use structured control flow. Agents operate within well-scoped action spaces rather than freely exploring environments. Only one case allowed unconstrained exploration, and that system runs exclusively in sandboxed environments with rigorous CI/CD verification. 5) Agent Adoption What drives agent adoption? It's simply the productivity gains. 73% deploy agents primarily to increase efficiency and reduce time on manual tasks. Organizations tolerate agents taking minutes to respond because that still outperforms human baselines by 10x or more. 66% allow response times of minutes or longer. 6) Agent Evaluation The evaluation challenge runs deeper than expected. Agent behavior breaks traditional software testing. Three case study teams report attempting but struggling to integrate agents into existing CI/CD pipelines. The challenge: nondeterminism and the difficulty of judging outputs programmatically. Creating benchmarks from scratch took one team six months to reach roughly 100 examples. 7) Human-in-the-loop Human-in-the-loop evaluation dominates at 74%. LLM-as-a-judge follows at 52%, but every interviewed team using LLM judges also employs human verification. The pattern: LLM judges assess confidence on every response, automatically accepting high-confidence outputs while routing uncertain cases to human experts. Teams also sample 5% of production runs even when the judge expresses high confidence. In summary, production agents succeed through deliberate simplicity, not sophisticated autonomy. Teams constrain agent behavior, rely on human oversight, and prioritize controllability over capability. The gap between research prototypes and production deployments reveals where the field actually stands. Paper: arxiv.org/abs/2512.04123 Learn design patterns and how to build real-world AI agents in our academy: dair-ai.thinkific.com

For improving RAG pipelines, it doesn't always have to be bigger = better. Make sure to visit the #NeurIPS ML4Sys workshop and learn from @melissapan and @NegarEmpr how to achieve 20x lower energy usage while maintaining high RAG quality!

Have you ever wondered how much you can optimize your RAG pipeline? 🤔 We show that you can get the same performance with up to 20× less energy ⚡️ Come check out our poster today at #NeurIPS2025 ML for systems Workshop to see how far you can push RAG efficiency! 🔋

🚨 Today at the MATH AI workshop @NeurIPSConf ! 🚨 Are you considering if RAG works for reasoning-intensive tasks like math? Come visit our poster this morning to see how corpus reconstruction can gains of up to 30.3% on benchmarks like MATH500! #NeurIPS2025