Lucas Beyer (bl16)@giffmana
I love brainstorming with this generation of LLMs!
The first round is often not "it", but after a couple back-and-forth, and crafting my messages to "force it to sample from further away", it comes up with such genius ideas that I would never think of alone, but are perfect for my taste/scenario.
Basically, when brainstorming with the model, imagine you're navigating the probability landscape of all ideas/implementations, and using the model as a sampler, and you play the role of the sampling algorithm that uses the sampler. Depending on what it answers, it's up to you whether you feel like it sampled the landscape well and you want to refine around one sample, or whether you feel like it only sampled from one basin and you need to punch it out of that to get samples from 1+ other basins, but you don't know what the basins are yet!
For any situation worth brainstorming, I already have a couple ideas myself of course, but am not 100% happy with any of them. But, in the first 1-2 rounds, I withhold my ideas, and just explain the situation to the model. I tell it what tradeoffs I care about and which ones I don't. I tell it to brainstorm with me, and give me at least 5 possible ideas, and their pros/cons/tradeoffs.
Sometimes, there's already great ideas I didn't think of in there, and if so, then I pick my 1-2 favourites, tell it why I like them, and to dive deeper and discuss more details.
But sometimes, none of the ideas is great either. Then, I do one of two things: I either mention my ideas, what I like and dislike about those, and ask it to bring up new ideas based on that. Or, I come up with 1-2 concrete examples that we should consider for the brainstorming, and tell it to revisit ideas with this concrete example. I chose the example such that it forces covering some area in "idea space" that I feel could be interesting but is missing.
That's roughly it. With this strategy I've been very successful in the past month, and always end up with an idea that's significantly better than any of my own ideas. I really love this. The LLM isn't inventing stuff on its own, but the tandem of me + my LLM, we clearly come up with novel and meaningful (to me) ideas.
I kept the LLM abstract because the principle works with all frontier models (that I tried), but I do now feel the need to specify that I had the best experience with this inside codex-cli, on max settings.
