Some thoughts about GPT-5.6-Sol after ~30B tokens: Sol is the most OCD model I’ve used thus far. It very frequently gets one-shotted by random nits in the codebase and writes a bunch of tests to fix it. Even with fast mode, it’s incredibly slow to do this kind of iterative development, especially when builds take really long. This by itself is not a bad thing, but the worst part is that after 2 compactions, it’s chasing the nitpick / useless goals I never told it to accomplish, rather than the main task. This behavior is so bad, I thought I was messing something up and tried codex, pi, and opencode to figure out if it’s a harness issue, but there is no meaningful difference between the three, which leads me to believe this is a model problem. AI code has this weird delayed release effect. You’ll only notice slop code 2 dev cycles into a codebase when you spend more time fighting with the code and on refactors than on shipping features. It’s possible that sol is better than 5.5 a couple cycles in, but tbd. My file deletion experience has also been similar to others: this is a dangerous model to let loose without guardrails. For instance, when performing a routine container upgrade, it accidentally printed out an env secret, then panicked and rotated ALL secrets (this is internal so not public facing, which was also documented), and proceeded to break everything, spending an extra hour fixing everything and redeploying everything else to use the new secrets. It also gets rid of files it doesn’t like. I have no idea why this is, but I think something about the reward model rewarded bookkeeping. Writing is another problem. 5.6 has a huge context bleed effect. It does not know how to write documentation and starts putting the specs in the documentation. If I ask it to develop a user sandbox for isolation, and also ask it to write documentation, it starts talking about specs and sandboxes in user-facing docs, which makes no sense. Fable is somehow much, much smarter in this regard. Frontend design has also not gotten better. Fable is still one generation ahead here. Overall, as a huge 5.5 user, I am not convinced that sol is a meaningful upgrade. It’s possible my practices need to change, but unfortunately it feels like I’m spending longer fighting with 5.6 than I did with 5.5. It’s like the model is so SO smart, but so hard to work with, compared to fable and even grok4.5 surprisingly. It’s clearly intelligent, but also just doesn’t care about what I ask it to do? (Is this supposed to be AGI feels like?) I hope the codex team fixes what possibly is a bad harness setup, because the benchmark numbers show a very different story from what I’m seeing while using the model.

It's inventing universal rules, that then broke old fixtures, to then having to rewrite tests, fixtures and more. It then imagined a theoretical risk, without any live failure proving this happened, and built an entire system around this. Once tests started failing due to his own failures he spend hours trying to fix those failures. In the end it did NOTHING i asked it to do, we built a very clear checklist to follow. I spent quite some time with fable and Sol 5.6 ULTRA (!!!!) to make a very straightforward plan to implement something. And just wanted to see what would happen if I give it the freedom to just work on the task. Had to revert ALL the work, lost about 60% of a weeks credits, and will never leave it alone to do things anymore. In another case it invented a client profile 2.0, did not care about all the verifiers turning red, continued to build on top of it, and i had to revert that work too. Not happy with how it just gets stuck in a rabbit hole, focusing on literally imaginary problems, and just destroys all my tokens (and time).

@JamieMartojo @sumukx “Imaginary problems” is a very good way to phrase it.

@constexprvoid @sumukx That's really what it is though, I just dont get what it's doing half the time.