Hendrik Schuff

Moving from Google Brain to OpenAI, one of the biggest changes for me was the shift from doing individual/small-group research to working on a team with several dozen people. Specifically, working on a bigger team has led me to think more about UX for researchers. Some examples: 1. Great tooling accelerates research. Subpar tools hamper researchers by introducing unnecessary friction into thinking and analysis. Even small improvements like reducing clicks and scrolls can significantly increase researcher's productivity. Visualizations become particularly vital when working with multi-task models, helping to better evaluate tradeoffs between different models. 2. Simple design is key for a the success of an evaluation benchmark. For example, GLUE/SuperGLUE, as well as MMLU/GSM8K have a single number (higher is better), and everyone wants it to go up. They are easy to understand, download, and evaluate. Other benchmarks (e.g., BIG-Bench, probably one of the great benchmarks of the past two years IMHO) can have advantages such as much broader coverage, but are basically impossible to run and a pain in the ass to analyze. For Google's PaLM paper, I heard one engineer's full-time job was just to run BIG-Bench... 3. Strong documentation enables scaling communication without involvement. Imagine if you have to chat with someone to explain how something works. They have to wait for you to reply, and you have to stop your work to message them. This takes up two people's time. With good documentation, you don't have to be involved at all, and the other person doesn't have to wait for your responses, which accelerates both people a lot.