kenkyūsha

Sometimes you can tell your civilization just Hasn't Done Its Homework on something because the dozens of LLMs you send on little agentic research quests keep coming back with confused, miscellaneous, high-spread data / takes on something. Like, I'm researching solar flares and the reliability of Earth's ten thousand total large transformers (which our grids depend on) right now and the sixth context window I ask about this will mention something none of the previous context windows have, in a way that clearly conflicts, and then you realize there are only like, 5 reports out there on this topic and they all came to different conclusions and there's no backup or "filler" between these 5 two-hundred-page PDFs as you would have for most topics (e.g. LW posts, Substacks, random nerds with 2010 era in-depth sites) because it's so weirdly obscure. This is also a problem because when the topic has so little trace in the training data, the LLM will search things using horrible priors, and so you need to hold it by the hand much more than usual. Anyway my point is that you can feel the "shape" of research thoroughness Earth's Internet has done on something, and realize there's a huge qualitative gap between something we have or haven't done our homework on, even when the LLMs appear ~equally confident no matter the topic and provide the same amount of tokens as an answer. Another reminder there are a mere 8.4 billion people out there and it sometimes feels very lonely around here.