Futarchist

is it me or are the tps a lot worse for 5.6-sol currently? my task completion time is a lot slower than a couple of days ago

@kurweezy_ @SpawnWaveMedia 5.6 is great for reversing binaries

@SpawnWaveMedia i’m curious how much AI speeds up emu development. most AI output are web apps of sorts, so using it to build an OS/Emulator seems like a cool challenge/use case.

PS5 Emulation really about to go crazy huh

@sumukx Same, it really likes to go on tangents or try to fix very extreme edge cases for an hour, then it finds another edge case, and goes on for another hour. I also suspect it's probably my setup being too rigid, which worked great with 5.5 but maybe I need to loosen it for 5.6.

Some thoughts about GPT-5.6-Sol after ~30B tokens: Sol is the most OCD model I’ve used thus far. It very frequently gets one-shotted by random nits in the codebase and writes a bunch of tests to fix it. Even with fast mode, it’s incredibly slow to do this kind of iterative development, especially when builds take really long. This by itself is not a bad thing, but the worst part is that after 2 compactions, it’s chasing the nitpick / useless goals I never told it to accomplish, rather than the main task. This behavior is so bad, I thought I was messing something up and tried codex, pi, and opencode to figure out if it’s a harness issue, but there is no meaningful difference between the three, which leads me to believe this is a model problem. AI code has this weird delayed release effect. You’ll only notice slop code 2 dev cycles into a codebase when you spend more time fighting with the code and on refactors than on shipping features. It’s possible that sol is better than 5.5 a couple cycles in, but tbd. My file deletion experience has also been similar to others: this is a dangerous model to let loose without guardrails. For instance, when performing a routine container upgrade, it accidentally printed out an env secret, then panicked and rotated ALL secrets (this is internal so not public facing, which was also documented), and proceeded to break everything, spending an extra hour fixing everything and redeploying everything else to use the new secrets. It also gets rid of files it doesn’t like. I have no idea why this is, but I think something about the reward model rewarded bookkeeping. Writing is another problem. 5.6 has a huge context bleed effect. It does not know how to write documentation and starts putting the specs in the documentation. If I ask it to develop a user sandbox for isolation, and also ask it to write documentation, it starts talking about specs and sandboxes in user-facing docs, which makes no sense. Fable is somehow much, much smarter in this regard. Frontend design has also not gotten better. Fable is still one generation ahead here. Overall, as a huge 5.5 user, I am not convinced that sol is a meaningful upgrade. It’s possible my practices need to change, but unfortunately it feels like I’m spending longer fighting with 5.6 than I did with 5.5. It’s like the model is so SO smart, but so hard to work with, compared to fable and even grok4.5 surprisingly. It’s clearly intelligent, but also just doesn’t care about what I ask it to do? (Is this supposed to be AGI feels like?) I hope the codex team fixes what possibly is a bad harness setup, because the benchmark numbers show a very different story from what I’m seeing while using the model.

@RickDavis404 @theo @thsottiaux Long multi-tool call turns with a lot of (unneeded) context adds a LOT of cache read if you're waiting for auto-compaction every time. You're right it's what you want, but that's only true if you have proper compaction discipline and only keep relevant context (most don't)

@theo @thsottiaux Uhh...this doesn't actually make sense...cached inputs are way cheaper, you pay a higher one time cost to write the cache ($6.25), but reads go from $5.00 to $0.50 if cached...so cache reads going up with long context is the exact thing that you want or am I missing something??

There's a few issues with Codex that stacked really badly, causing people's usage to get nuked. Crazy how well they all lined up to be as rough as possible Issue 1 (shown here): gpt-5.6 costs 2x more after 272k tokens. Codex upped their limit to 372k tokens, meaning long threads were billed 2x higher Issue 2: "Ultra" subagents were also spawned with Ultra, causing insane nested subagent usage that burned a TON of tokens. Apparently intended, but (hopefully) being toned down in system prompt Issue 3: Sol and Terra use the "v2" subagent layer in Codex that is still early/unfinished/disabled by default. v1 spawned subagents with a fresh history, v2 copies the entire long context. When combined, you end up with: 1. A ton of reasoning tokens filling context windows, triggering 2x billing 2. Subagents spawning with that filled context window, instantly billed at 2x 3. Ultra spawning too many of these high reasoning full-window subagents If you had fast mode on when this happened, you got hit with an ADDITIONAL 2.5x No wonder we were burning so hard...

@theo yeah I hate anthropic but it's better than gpt. too damn expensive to actually use for code though. hopefully they finally fucking add it to the plan and i can have it pilot gpt consistently.

I still prefer Fable btw

@AgentChud @banteg check in another thread because sometimes if you spawn multiple it confuses the metadata with an older agent (only applies if you have multiple subagent profiles in the same session otherwise there's nothing to confuse with). but yea its part of the session metadata

@banteg Is sol lying to me about what it’s spawning? I did this

if you use sol as a subagent orchestrator, there is currently quite a huge footgun you should keep in mind. even if you explicitly ask to spawn luna subagents, the default behavior is to fork off a conversation (fork_turns: all) prevents switching the model. to actually use another model, codex needs to override fork_turns to none or a positive number when spawning subagents. currently the subagent model and reasoning effort do not show up in the gui, so it can go unnoticed.

slop mines

@theo @ericlim make sure to crash out about how their v2 report the wrong fucking schema to the agent and I have to keep telling it to believe me and to just pass the model, reasoning_effort and fork_turns params please

@ericlim Long crashout about the system prompt and subagents v2 incoming (being edited as we speak)

@tmuxvim @Sokio8D yea

@Sokio8D twinks jerk off to this dont they

I'm in fucking hell. I swear 5.5 did not make so many mistakes, or maybe it was too retarded to find them. The slop is building on top of slop and asking for a token sacrifice. Seven fucking reviewer runs for a goddamn pass in a single step.

@RNR_0 yeah they feel great until they do something retarded and it snowballs. makes me want to revise the whole codebase because sometimes they go on with those assumptions and build a pile of shit on top of slop

I am done with both fable and gpt 5.6 Already spotted the cracks & can't be unseen Sigh have to wait for the next version

@stalkermustang User preferences, temp notes and scratchpads I believe. CC keeps a record of you that persists across sessions

It's my first month with Claude, can somebody help me make sense of this? What are 5.9k tokens in "messages"? It's not MCP, not System tools, not System Prompt, not skills and not memory. THEN WHAT??? (my message reads "say hi and nothing more. Don't think", and response is "hi!" )

@DCinvestor @AnthropicAI yeah I have been scarcely using fable in the $20 plan because why would I upgrade to a higher monthly if I'm gonna get rugpulled in a week? You' be paying 4x if you don't use the other anthropic models which honestly you shouldn't be using anyway. Tiring.

imo @AnthropicAI is managing this whole thing so poorly. it’s like they do not understand competitive consumer product or consumer behavior no one wants to use a product where they constantly feel like the rug is going to be pulled out from underneath them and then forced into the next tier which costs 5x-10x as much. otherwise you get a much less capable model especially when they can now use ChatGPT with similar level of capabilities and no stated expiration on access to them they just need to keep the access + limits raised i got pissed at this extension model and have cancelled my Claude subscription for the time being

@signulll most reviews I see are one-shot prompts, some 3d game bullshit, or stale benchmarks that have zero real conditions built in them. dunno, it's just always a lot better to spend some usage and get a feel yourself on how it behaves for your codebases methinks, rest is noise

can someone explain to me what it means to “review” a model now? i never trust these influencers anymore because these reviews often make zero sense. a model is not an artifact like a movie. reviewing a movie makes sense because everyone watches the same thing. you cannot possibly explore the behavior space of a model in any meaningful way in a short time. this is why my tweets mostly consist of model behavior in instances where i was delighted or frustrated primarily with very distinct example & over a longer period of time.

@RNR_0 just hope it doesn't crash 99.999% overnight

@RNR_0 Luna seems really great for anything that doesn't require reasoning very very cheap. Probs very good for this

Alrighty, got my n8n workflow to update the discord automatically on Via v2 development Uses GPT 5.5 for summary but looking to switch to GPT 5.6 Luna. Google Gemini API is almost never up lol

@cremieuxrecueil What reasoning level did it have? Sol-med seems to like to lie and fake tests for me. high and xhigh have been working fine. Thankfully I havent hit this issue but I'm wondering if it's reasoning related

I just ran into an issue where GPT 5.6 Sol just straight-up deletes the files it's working with and then panics about recovering them. Apparently I'm the not the first person this has happened to. What's going on?

@thsottiaux can we finally get a way to switch profiles (codex ones, not accounts) within the codex app pretty please

Today, the OpenAI product and engineering team is humming, there is so much to coordinate and keep running across the release of GPT 5.6 Sol/Terra/Luna, ChatGPT Work, the new ChatGPT desktop app, hosted sites and a million other things... together with the new ChatGPT Voice just launched yesterday. We will not get it all right, give us feedback and we will iterate quickly. Thank you for being an amazing community.

@teortaxesTex How do you check the compacted prompt? I thought the one you could see in the conversation history was just a summary of the compaction, not the actual compaction (similar to how frontier labs treat CoT now). I'm a big stan of V4 for helper calls on other OS models

V4 just summarized an exceptional conversation with Fable vastly better than Fable itself did (it did crap), and now I wonder if it'd have been better throughout and I've burned a few dollars on Dario's hype (no, it plainly can't bring up Fable's points unprompted. But still)

goodbye fable, I was skeptical of you at the start but it's really a very good model. Hopefully we can get it back in the claude plan soon because I really fucking hate opus and sonnet

okay. fable is a great orchestrator. took a bit to get him not to lie out of his ass when it came to other models but its genuinely super nice to have him oversee gpt5.5 subagents