Jason Wei@_jasonwei
For most companies, hiring more people is strictly better. However, this is often not true in AI research. AI research is often bottlenecked by compute, and when this is the case, hiring more researchers can be counter-productive.
I remember back at Google Brain, my manager once said we had one headcount and asked who we should hire. I responded that hiring someone will basically be counterproductive and we should try to trade the headcount for TPUs/GPUs instead. For example, if a researcher needs 100 GPUs to do their research and the team is already bottlenecked by compute, there is no point hiring them because everyone else on the team will simply take a hit in productivity in waiting for GPUs.
So an interesting hiring consideration is how many GPUs a potential hire will need to do their work. Hiring an extra person could feel like progress but if you don't scale GPUs at the same rate that you scale number of people on the team, the productivity of the team might not improve.
This leads to a conclusion that I believe is true but not advertised explicitly: people who are able to do good work with few GPUs can be more hirable/flexible/productive on teams that are compute-bottlenecked, compared to people who only know how to do work if they have 1k GPUs at their disposal.
(One caveat though---if the new people you hire will use GPUs *more productively* than the current team, that could be OK since net team productivity will increase, even though productivity per person decreases. Conversely, if the new person you hire needs to use many GPUs and doesn't use them well, then both net team productivity and productivity per person will decrease.)
