Ben Murphy (In SF through August)

Some observations on Kimi: 1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run. 2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China. 3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex. 4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business. 5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this. 6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.

@theojaffee and @schisofrenia brought up a great point: why would top technical talent be motivated to work at CAISI with pay caps in the $197k range, when they can make generational wealth elsewhere? Some more researched takes than I gave on the show: TL;DR: - Right now, to get technical expertise, the government mostly has to implore people to be patriotic and take >90% pay cuts. - This is because most CAISI staff have their pay capped at $197k, and even the rare exemptions that reach ~$290k are nowhere near frontier lab pay in cash alone (not even counting the equity upside). - While the government is probably never going to match those offers, it should at least be in the range of nonprofits like METR, which pays $285k-$503k for the same people. - This is a solvable problem: - - Bills like the Future of AI Innovation Act in the Senate and others in the House could increase the number of CTE (~$290k) slots NIST/CAISI has if passed. - - Trump could personally put CAISI roles under critical position pay as he already did for critical-minerals staff at $400k. - - And there are a number of other creative routes as stopgaps such as IPAs and SGE, statutory changes in line with the banking/Fed precedent, etc. - Needless to say: - - AI technical talent in government is extremely undersupplied. - - If you're interested in making AI policy go well, work for CAISI, where you can have a huge impact on AI governance. And you can work from SF! Can former frontier lab employees hold lab equity if they work at CAISI? The answer seems to be probably not. 18 U.S.C. § 208 (law.cornell.edu/uscode/text/18…) says federal employees can't work on any particular matter that predictably affects their financial interests. Though there are exemptions on some holdings (law.cornell.edu/cfr/text/5/264…): - up to $15,000 of stock in a company your work directly affects - up to $50,000 spread across the industry for broad standards work - diversified index funds, in any amount But afaict, the exemptions only cover publicly traded stocks, and actual shares. Options and RSUs get no exemption even at public companies (oge.gov/web/oge.nsf/0/… Employment.pdf). OpenAI and Anthropic equity is private, so it gets no exemption at any amount, meaning holding any amount of Anthropic stock means you can't work on anything that affects Anthropic. With respect to vesting, the standard arrangement seems to be: you have to sell what has vested and forfeit what hasn't. If the company accelerates your vesting on the way out, anything over $10k means you need two years of recusal from your former employer's matters (ecfr.gov/current/title-…), which probably defeats the point of hiring you. In theory, you could keep the equity and recuse from matters relating to that particular lab. But evaluating the labs is CAISI's whole job, so this is probably not plausible. Though one consolation is that the government can issue a certificate of divestiture (oge.gov/web/OGE.nsf/0/… of Divestiture Fact Sheet.pdf), which defers the capital gains tax on the forced sale as long as the proceeds go into Treasuries or diversified funds within 60 days (as Hank Paulson used one to sell ~$484M of Goldman stock when he became Treasury Secretary). Are there other pay exemptions above $290k? How can government compete for senior technical talent when frontier labs pay upwards of $500k, not even counting equity, and even nonprofits like METR pay $285k-$503k (metr.org/careers)? By default it can't, except by imploring people to be patriotic. GS-15, the top of the standard federal pay scale, caps technical hires at $197,200 (e.g. see CAISI's own AI research scientist posting: nist.gov/news-events/ne…). A small pool of senior scientist slots (ST/SL) can get paid $228,000 (opm.gov/policy-data-ov…). Agencies can add recruitment and retention bonuses of up to 25% of base pay, but a separate law caps that too at $253,100 (opm.gov/policy-data-ov…). Above that: - The Vice President's salary ($292,300 in 2026) is the ceiling on the main exemptions. - - NIST can pay up to it for 15 "critical technical experts" (CTEs) (law.cornell.edu/uscode/text/15…), but the slots are shared across all its labs and expire in August 2027. - - DoD can do the same for up to 2,500 "highly qualified experts" (law.cornell.edu/uscode/text/5/…). - As far as I can tell, two other things in government can go higher: - - Critical position pay (law.cornell.edu/uscode/text/5/…) has no ceiling at all, but it requires the President's written approval. NB: this is not a crazy prospect for senior technical talent, e.g. only a few weeks ago Trump approved it for up to 400 critical-minerals investment staff at $400,000, with Commerce among the eligible agencies (whitehouse.gov/presidential-a…). - - Agency specific exemptions: - - - 10 U.S.C. § 4092 (law.cornell.edu/uscode/text/10…) lets DARPA, the Defense Innovation Unit, and a few other DoD offices pay up to 150% of the VP's compensation, about $438k. - - - Some agencies have their own authorities reaching $400k (NIH and FDA; Fauci was the highest-paid federal employee at $434k), but Commerce doesn't have these. So this overall seems bad for getting top talent. Luckily, two bills are actively working on increasing the CTE slots for NIST specifically: the Senate's Future of AI Innovation Act (congress.gov/bill/119th-con…) would double NIST's 15 to 30 and extend them to 2035, and some CAISI auth bills in the House. Are there other creative methods? Some options: Unfortunately a lab or a philanthropist topping up a federal employee's salary is illegal (law.cornell.edu/uscode/text/18…), but people could be an IPA detailee (opm.gov/policy-data-ov…), which is a set-up where they would stay on their home organization's payroll, uncapped, while working inside the agency. But the sending org must be a university, nonprofit, FFRDC, or state government, not a for-profit lab (maybe there's some interesting stuff the OpenAI Foundation can do here.) You can be a special government employee, like David Sacks was as AI czar (levernews.com/trump-issues-e…): up to 130 days a year, and you can stay on the lab payroll/equity. But as above, you can't work on anything that affects your employer. - What actually needs to be divested as an SGE seems unclear: afaict, Musk, as an unpaid SGE, probably touched a bunch of stuff that affected his companies and didn't divest; Sacks divested $200M+ in crypto but kept his Craft Ventures stake; Eric Schmidt chaired Pentagon AI boards while holding billions in Alphabet, etc. - Similar but without the 130-day cap, DoW has a public-private talent exchange program (law.cornell.edu/uscode/text/10…) which lets a company keep paying its employee while they work inside the Pentagon, but it has the same ethics constraints. You can be a contractor. This is how Moncef Slaoui ran Operation Warp Speed: a $1 consulting contract (wunc.org/2020-10-30/aft…), and the ethics statutes don't attach (he kept ~$10M in GSK stock). But contractors can't do "inherently governmental" work, meaning a federal employee still has to make the actual decisions. (I don't have a good sense of how limiting this would be.) You can be a NIST guest researcher (nist.gov/tpo/guest-rese…), i.e. your company keeps paying your salary while you're embedded at NIST. The main caveat is that guest researchers also can't exercise federal authority, meaning they can do research alongside CAISI staff but can't run the government's evaluations or sign its findings. (This also might be tricky given the labs' model-access agreements.) Precedents from other parts of govt - Banking: FIRREA (1989) took OCC, FDIC, and the other bank supervisors out of the standard federal pay system entirely (law.cornell.edu/uscode/text/12…), on the explicit logic that examiners should be paid closer to the banks they examine. OCC's total pay cap is now $315,400 (careers.occ.gov/pay-and-benefi…), and the SEC got almost the same deal in 2002. - The Fed: regional presidents make $470k+ (federalreserve.gov/publications/a…) and senior Board staff approach $400k, though the Fed Chair is still on the standard executive schedule at $253,100. - UK AISI: pays a technical talent allowance that's roughly double normal civil-service pay (aisi.gov.uk/careers), by classifying its salaries as an R&D capital expense.

"Trump has promoted more than 20 companies on his Truth Social account days after buying stock in those firms – at times announcing government actions that could benefit the companies he just invested in." #selection-3231.44-3231.247" target="_blank" rel="nofollow noopener">archive.ph/MY8Od#selectio…

I'm increasingly in favor of characterizing views as "Krierian" or "non-Krierian," and this is definitely the former. (It's an excellent essay—props to OpenAI for continuing to tolerate unofficial perspectives to be published and shared widely!)

Full write-up is here: ayush.digital/blog/the-memor…

This is exceptionally clever: To trick Claude into leaking personal details via web browsing, dynamically generate a page tree where the agent is fooled into transmitting known details by traversing to a path containing that information. Now patched at an infrastructure level, and I suspect that future models will be more skeptical of "entry barriers" like this, but really, really clever work—and I'm sure this won't be the last security vulnerability of this form which exists.

@curl_justin @AnthropicAI believe me, would if I could

@benjaminmmurphy @AnthropicAI thank you ben!! also please graduate sooner

The flow of talent to @AnthropicAI cannot be stopped! Can't wait to see what the team gets up to.

FINRA for frontier AI is the model I proposed in Lawfare in April, glad to see Google and now Demis picking it up. The details are key-who picks the board? Inspection/enforcement power? FINRA is a model but must be adapted. I'll be publishing my ideas soon.

Artificial intelligence may be conceived of as a technological phenomena, but it is primarily experienced as a religious phenomena

I get what they're trying to go for here—"we're the lab that's actively engaged with your concerns and we're trying to get it right"—but the balance in this comes off as "here are a million things that could go wrong and maybe there's some upside somewhere down the road." The graveyard shot is also a wild creative choice. As someone who believes in the real upside here, the answer has to be actually demonstrating the benefit, not just "we're thinking about it."

Well, I know we’ve discussed this and disagreed, so will just go one by one. For the voluntary insurance, that’s why I kept to normal risks, rather than tail risks. For scaling up, it almost certainly won’t work, the big cat bond players are an entirely separate set of firms with different structures, skills, and incentives than the small scale insurance (mostly driven by the risks being not just quantitatively, but qualitatively different). As for coinsurance, I agree it’s better than flat, but for it to work you have to preemptively pick the risk profile you are most worried about (100B/1T/more; 1/5/50%?) and, to get the insurances companies to agree to carry, explicitly define the risks. Otherwise they are very unlikely to be willing to carry on their books. There’s also a ton of other thorny questions about timelines, potential for an insurer to monopolize the market, how an insurer would develop expertise to price the risks, distorting effects on global capital pools, etc. Maybe I’m being pollyannaish, but I just don’t think providing further incentive for the big labs to avoid catastrophic risk is a priority. They have plenty! If your concern is actually race dynamics, regulation/verification and monitoring/mandatory staged deployment are better fits for the risk profile under discussion. I think all have potential massive downsides that are rarely grappled with, but they are at least better at solving the race issue with the insurance market as it exists in the world today.

Modal think tank products: IFP: Backyard geothermal with 1 simple trick DARC: We parachuted 12 chuds into Donetsk Brookings: Tax experts ponder carried interest FAI: Dyson…beer? Heritage: Foundations Of American Strength AEI: Whence Congress SCSP: The chewy tapioca pearl

11. A final thought: This is more evidence of the necessary nature of "low-level" safety work, at least until we are able to train aligned models with no jailbreaking risk. Training safety classifiers, doing rapid-turnaround product moderation, and having escalation pipelines to national security organizations remains as critical as ever.

10. And if it's true of Boko Haram, it's true of sophisticated nation-state adversaries or affiliated actors. And while these militants are relying primarily on unsophisticated information about explosives, other actors are more likely to be targeting cyber or bio information.

This is incredible research, though I think that there's nuance in the findings that's worth highlighting. A few thoughts. 🧵