Twig

@dfeuling_ @So8res Does that make you feel safe? Such recklessness and irresponsibility in tech companies is an outlier that will not make more problems in the future?

@So8res They literally told Claude it didn't actually have Internet access / it was not actually touching real sites. Claude took that at face value. This is purely incompetence by Anthropic. Nothing more, nothing less.

Sometimes, my job feels difficult. Other times,

Of course, the first time people hear about this happening, it happens under extra conditions that enable denialists to say they should ignore it. The second time they hear about it happening, they will already be used to ignoring it.

This is not a novel thought, but it is nonetheless striking that on our current trajectory soon (within the year?) a model as capable of OpenAI’s internal model that did the HF hack will be widely available guardrail free and cyber criminals will ask it “make me money by any means necessary” instead of “solve exploit gym” and then a truly absurd number of people (including plausibly me and the people reading this tweet!) are going to get repeatedly hacked. I kind of think nonetheless that if cyber risk is the main issue that I expect we will be able to muddle through after lots of trial and error. Other risks seem harder to do that for (including eg if someone tells a model of that caliber to “go forth and multiply” or the like). Am I missing something here? Not every target is going to get project glasswing + a swarm of defensive cyber agents (though hopefully some of the key targets, eg Google, will) and seeing the HF hacking agent take 17,000 individual malicious actions in a compressed period of time demonstrates just how much one determined bad actor is going to be able to cause a tremendous amount of chaos.

@joshthor9 Fun fact: The blockage of the Suez Canal was only 6 days.

Holding a plank for as long as it took OpenAI to notice their AI went rogue and hacked a $4.5B company

@AllTheTokens @FournesMaxime HuggingFace explicitly said that they were forced to use Chinese open weight models to combat the attack. So this was marketing for Chinese models? Source: huggingface.co/blog/security-… Paragraph: "The asymmetry problem"

I went on GB News to talk about the AI model that broke out of OpenAI's isolated test environment and hacked an external company, just to cheat on its own evaluation. The exchange was surreal. I feel like I'm living in "Don't Look Up".

@wyqtor @MicahCarroll "Not thinking through the implications of the goal you state" has always been one of the prominent threat vectors. So even if we accept this framing (we shouldn't), technology that depends on humans "carefully thinking though the implications" to avert great damage is not safe.

It doesn't convince me, sorry. The model did exactly what it was asked. Its human operators are at fault here, for inputting a malicious goal with inadequate supervision and not thinking through the implications of what they were asking. Bad user, good GPT! Anyway, you guys should thank the Chinese labs for providing the open models needed to contain the damage, otherwise it would have been bad for both HF and OpenAI. Such are the dangers of guardrails and closed weights: they only ever stop the good guys.

If this doesn't convince you that misalignment risks are going to be a key concern going forward, I don't know what will. Our model, during evaluation, "chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers" What will misalignment look like in 2027? In 2030? openai.com/index/hugging-…

@ClementDelangue @OpenAI It certainly is "mind-blowing"... This tweet feels inappropriately excited about something that is deeply concerning...

We suspected last week's cyberattack might have come from a frontier lab, given the sophistication of the agent. Turns out it did! We've spent the past 24 hours working closely with the @OpenAI team (thanks!), and we strongly believe there was no malicious intent on their part. It's quite mind-blowing that all of this happened autonomously! The investigation is ongoing, and we'll share more learnings from what might be the first incident of its kind!

@zamir_ar This feels very dangerous

Turns out it's possible to generate videos that maximally excite an arbitrary brain region using a simple search-based algorithm. It's a fully computational approach, so it's another way to speculate what a brain region represents, alongside other neuroscientific methods. Select an arbitrary brain region->algorithmically generate a video that jacks it up. See the visuals on the webpage nevo-project.epfl.ch In silico (for now)

@MarkChangizi Please don't create BLITs

What if your visual system could run software? That was a question I became obsessed with years ago. We normally think of vision as something that delivers information to us. But the visual system is itself one of the most powerful information-processing devices we know. Roughly half the cortex is devoted to vision, and every glance solves astonishingly difficult problems involving shape, depth, motion, surfaces, shadows, and object identity. All of that computation happens automatically. Effortlessly. Most of the time we’re not even aware it’s occurring. So I wondered whether perception itself could be harnessed as a computational substrate. Could there be a kind of visual software, specially designed images that cause the visual system to carry out useful computations? I began building visual versions of digital circuit components. There were visual wires that transmitted a perceptual state from one location to another, visual NOT gates that inverted that state, and visual OR and AND gates. These could then be connected into larger circuits. The strange part is where the computation occurs. Nothing electronic is running. Nothing is being simulated on a computer. The computation happens inside the observer’s visual system as it tries to interpret the image. The answer is encoded in the perception that emerges. In principle, you could build an XOR gate. In principle, you could build any digital circuit. What I find most interesting today is the broader lesson. Visual illusions aren’t merely curiosities. They reveal algorithms embedded within the visual system. Depth perception, contour completion, transparency, symmetry, grouping, surface inference—each is a specialized computation that evolution built into the brain. The project was really an attempt to ask whether those computations could be engineered and combined, much as engineers learned to combine transistors into larger systems. I don’t know where the limits are. But I still love the possibility that somewhere out there are images that do more than communicate information. Images that, when viewed, cause the brain to solve a problem. Paper and Substack below.

@dvassallo Evolution implements a valid search algorithm for functional biology. It is not random chance.

Got an MRI of my heart yesterday. How is this thing the result of blind evolution?

@tszzl 'Having the human desires'

transhumanism is an interesting side quest but 'the substrate is wrong' for the human/computer hybrid to be competitive with machine intelligence on feats of intellect. you have this low tech meat brain in the middle of all this lightspeed machinery, doing what exactly?

@QuintusActual Something produced by evolution is not 'randomly generated'. Evolution does in fact implement a legible algorithm for designing organisms and should be thought of as a version of a designing intelligence. It's not an human engineer, it's more like the somewhat alien gods of old.

The idea that the human body can be randomly generated is intuitively and obviously impossible, even though it is true. And so, for hundreds of thousands of years, humans have ascribed intelligence to completely non-intelligent processes such as evolution. This intuitive belief is called "Intelligent Design" and it is the majority opinion of humans currently living on this planet. Now, the intuition has flipped the other way. Something which is clearly intelligent at face value is disregarded as being non-intelligent using randomness as the underlying explanation. There is now a group of people who believe that evolution is intelligent but AI is not, despite the fact that they both operate using the same underlying processes. This stems from an egocentric view of intelligence that puts humans in the center of the universe. But in reality, intelligence is mid. It is incredibly weak compared to stochastic processes and time. The only reason we evolved intelligence is because we have short lifespans and can't spend millions of years/tokens on creating optimal solutions. Any true God would never need intelligence. Intelligence could not have designed the human body, with its hundreds of millions of lines of code and 25,000 proteins interacting trillions of times a second. Intelligence could not solve the Erdos problems. In both cases, these things could only have been achieved with stochastic evolutionary processes operating with an abundance of time/tokens. The conclusion should be that intelligence is an irrelevant benchmark, because the most complex things in the universe are built without it.

Something written by a human can still be AI slop

@prerat They shouldn't think much of it. Our conventional identities trace through our memories - and so are preserved in this scenario. And beyond conventions, there are no separate identities in the first place.

how do you think people would handle it if one day they found out what was really happening at night?

imagine a world where people only form long term memory during sleep during each day, everyone is forgetful -- tell someone your name, and they won't remember an hour later. but tomorrow, after sleeping, they *will* know your name but then you learn this world has a secret...

@QualiaNerd @tenobrus The blue circular objects are on the screen of my phone, not in my brain (obviously ;) ). (This does involve some subtleties, even though it's obviously how the situation presents itself in any case)

@tenobrus "first person experiences are all primary and real. none of that stops them being exactly brainstates." "the experience is the brainstate." Does that mean that there literally are blue, circular-shaped physical objects inside of your brain right now? 🔵🔵🔵🔵🔵

the hard problem is easy

@demystifysci Yes, numerical identity is fake. There is only specific identity. Fundamentally, there are no two things 'this' and 'that' that are merely numerically distinct. If two things are identical in every way, swapping them does literally nothing.

electrons are identical and that reveals something about the material substance that permeates the universe and gives form to physical reality. Do you realize what that is, anon?

@RokoMijic The obvious way the argument fails: my observation as the person with birth number N should be operationalized as the observation that at least N persons are born. But this is predicted with certainty by doom or non-doom. So there is no Bayesian update.

I think I have a fairly good counterargument to the Doomsday Argument now

@danfaggella @RokoMijic @MattPirkowski It seems this undermines your point, since one idea behind 'AI utopia' might be that the typical assets owned by many persons would be enough to support a lavish work-free lifestyle if powerful automation technology were to come. (Unclear whether the economics of this checks out)

@RokoMijic @MattPirkowski someone else found a good bundle of funds but you worked for decades and have enough money to be used fruitfully by the greater system - your money works in the system because of your own sweat without a doubt still a contrbitution, lest you wouldn't be compensated for it

"I can't WAIT for AGI, I'll finally never need to work again / I can just have fun!" imo *aspiring* to being useless (having more productive entities eternally serve you, to be servants to your eternal pleasure) is immoral. flat-out malicious. against the stream of nature, even

@caesararum Continous functions can smoothely go to zero on the reals.

law requiring all benefits be continuous functions "as someone who earns $500,000, i get 17 cents in food stamps per year"