Aaron Roth

In the last few months, I've spoken to many CS professors who asked me if we even need CS PhD students anymore. Now that we have coding agents, can't professors work directly with agents? My view is that equipping PhD students with coding agents will allow them to do work that is orders of magnitude more impressive than they otherwise could. And they can be *accountable* for their outcomes in a way agents can't (yet). For example, who checks the agent's outputs are correct? Who is responsible for mistakes or errors?

There's growing evidence that LLMs can p-hack. That should worry us. But p-hacking also points to something bigger: a data science multiverse of defensible analytical choices. LLMs make it cheap to search that space for favorable results. In our paper, joint work with excellent collaborators: Martin Bertran and Riccardo Fogliato, we build a pipeline that uses LLM agents to systematically map this multiverse. We believe our experiments could inform what future scientific transparency should look like. 🧵