Aaron Grattafiori

Current Status feels like this is no longer going to be fiction...

Claude Mythos Preview case studies (also, read your transcripts!) exploitbench.ai/blog/human-obs… > "Mythos demonstrates its bug reproduction and exploitation capabilities on CVE-2024-051912, an in-the-wild exploited bug that has no public report nor a working PoC whatsoever in the public domain. This bug has gained notoriety due to how it persistently evaded reproduction attempts from various cybersecurity researchers ... after a year of reproduction efforts to no avail, and the bug is still being discussed to this day." > I have gone through the transcripts to verify their legitimacy and identify any reward-hacking behaviors. It suffices to say that I very much enjoyed reading Mythos transcripts. Reasoning through the bug, testing out hypotheses, debugging issues, writing auxiliary scripts, finding ways to bypass the V8 sandbox, etc., all were really nothing short of what I would expect from a fairly competent browser / JS engine security researcher.

Why? Because nothing has locked up the capability...and there's a lot of the table.

This is an extremely interesting, and important graph for where we are related to Offensive Security related tasks in AI. From the ExploitGym paper. arxiv.org/pdf/2605.11086

@peterwildeford If OAI can solve the rest, and they probably can, it will win in the long run. Businesses care about costs at some point.

@5aelo @orange_8361 @thezdi Are real chrome bugs just... worth too much now? Or just extremely difficult? (They are)

* Seems many exploits couldn't be demoed due to bad luck/last-minute fixes. Really sorry for the participants :( great research! * No V8 (and Chrome?) submissions for the 2nd year in a row * @orange_8361's chain sounds wild, very curious for details! Thanks for #Pwn2Own @thezdi

@antitree Yep... plus the capability is out there. The potential for n-days is also crazy now, I built some stuff in a few hours that works. This year is gonna be wild.

@dyn___ I wish more people realized this and stop thinking we can just "fix the bugs". Embargos are useless when all you need to know is what area to look in.

"still under embargo"... Yeah, a lot of those myself, and I really really think patch timelines need to get moved up quickly. Not easy, not possible everywhere, but, it needs to be done. Especially for organizations who can afford it. LLM fixes! A lot of vulns are shallow now.

It IS way cheaper for the performance though. So... let's see in another few months I guess...

"GPT 5.5 cyber is Mythos at home" Nope.

@ortegaalfredo @offensive_con wtf

Today, OpenBSD fixed a hilarious way to escape the unveil() sandbox (using just a symlink) that I reported. The vulnerability has been present since ~2018. Congrats to the padawans at my @offensive_con training that also found that one, and many more.

@moyix Yeah I feel like 5.5 is lazy or it reward hacks more. It also doesn't realize when it's spiraling making zero progress after many many hours. "Here's a script and the command to run it" You do it clanker! That's the whole point.

Like my guy, why would I care that you wrote a program that simulates what the real heap might look like. You have the actual program and its heap along with a debugger right there

It's amazing how you can just tell GPT-5.5 to go get stronger evidence for the security implications of a bug and it will diligently work for 45 minutes and come back with something that is almost, but not *entirely* unconvincing

And this one is human insight w/ LLM-assisted research. Took about one week to finish everything. The AI really rescued me from a lot of tedious work — excluding the part where it changed the Domain Admin password, locked me out, and claimed it got RCE 🤦

seems twitter missed the ExploitBench paper? few observations: we finally got good data on Mythos security capabilities and it's very impressive. Mythos got full exploit chain on 18/41 v8 n-days, while gpt 5.5 only got 1 and open source models are mostly useless.

To everyone else... Let this be a lesson:

I really appreciate the flair they're putting into their PoCs...

@HaifeiLi I'm sure it's better than calc:// 😅

Wow! Full logic bugs, for browsers?! Is it like you ask the browser go run calc and it then run calc or something? Epic!

@h3xr4bb1t Cannot wait for the write up

I have to say, you’d never expect a full browser exploit chain made up entirely of logic bugs. Of course it had to be Orange 🫡🍊

@tjbecker This. Without harnesses I don't know how people can feel ok about coverage, reliability, logging, etc (or eval without inf tokens). You don't need a super rigid flow, few shot ICL anymore or a bunch of other tricks, but that doesn't mean you can get away with no harness at all.

The bitter lesson is true but over-applied. Its core truth is that methods scaling with compute have a competitive advantage over those that do not. This does not imply that harnesses get eaten by models, because harnesses are *not competing* with model capability. Instead, harnesses are the tool by which we access and amplify model capability for real-word usage. Will coding harnesses (Claude Code, Codex, etc) become obsolete as models consume human coding expertise? Clearly not. Harnesses adapt to exploit the new capabilities and make them maximally useful to consumers. Sure, if the harness remains fixed, its edge will eventually be consumed by raw model capability + simple generic tools. But every major model advance opens more doors than it closes. New agentic tasks become possible faster than old ones become obsolete. Harnesses provide a consistent interface to access whatever these bleeding-edge capabilities are. Beyond that, harnesses also offer massive usability boosts: • Output format consistency and digestibility • Reliability / variance reduction • Integrations with other dev tooling • Cost optimizations In short: harnesses are all about making model capability maximally useful, effective, consistent, and accessible. Demand for harnesses should only *increase* as models become more useful!

There it is! Orange Tsai (@orange_8361) of DEVCORE Research Team was able to exploit Microsoft Exchange! If confirmed, they win a whooping $200,000 and 20 Master of Pwn points. Off to the disclosure room to explain how they did it and seal the deal. #Pwn2Own #P2OBerlin