Ben McCarty — security-maximalism 🛡️

@eurydicelives With some quantum mechanics added to the game, I feel a fighting chance to win against a superintelligence.

Tell me why you *feel strongly* about Newcomb's paradox. Not what you choose. What it makes you feel.

@ESYudkowsky What do you think about Leibniz's Mill thought experiment?

The Chinese Room thought experiment runs like this, updated for modern times. A man who speaks no Chinese is locked in a room. He receives a card bearing a Chinese character. The man looks up the character in a table, and retrieves 16,384 numbers, each recorded to 3 significant digits of precision. Following instructions in a rulebook, the man now multiplies those 16,384 numbers by a matrix with 16,384 rows and 16,384 columns, so 268 million entries. If he can multiply two three-digit numbers in 10 seconds, this will take him 85 years. This represents one sub-operation inside on layer of a modern LLM. Each of 100 layers might have 3-6 sub-operations like this. The man receives a series of 20 cards, with a total of 20 Chinese characters. So he repeats all of the huge sub-operations 2000 times. Some sub-operations take longer than 85 years, especially the 'attention' operations where each token collects data from all the previous tokens. The man is immortal. He cannot be bored. Many millions of years pass. The man finishes processing the original 20 cards. He now starts carrying out further operations on the numbers, that will produce hundreds of new vectors of 16384 floats, whose closest neighbors the man can look up to produce hundreds of Chinese characters. Billions of years pass. The planet and sun containing the room are as immortal as the man himself. Eventually a slip of paper slides out of the room, bearing a sequence of a few hundred Chinese characters. === Originally a woman had written "我在王府井和长安街的交叉口，需要到达颐和园": I am at the corner of Wangfujing and Chang'an and need to reach the Summer Palace. The slip of paper that emerges contains the correct directions in Chinese: subway lines, transfers, the right exit to the east gate. The woman follows them and arrives successfully. Or maybe a Chinese mathematician, working on a forthcoming math paper, had requested help on a blocked step of a math proof. She gets back a valid mathematical argument, also in Chinese, and completes her paper, which will later pass peer review and publish. === But the human male inside the Chinese room knows nothing of this, for he does not know Chinese. He only multiplied numbers according to a rulebook. He's never seen a map of Beijing. He couldn't state a single one of the axioms used by the mathematician's proof. That indeed is a valid fact, in the context of this thought experiment. But what follows from it about real life? === If you are wise, the moral of this story is that a large structure can contain knowledge that isn't in any single piece of the structure. Pick up an accurate street map of part of Beijing. Even if the map's whole structure has a good pointwise correspondence to the actual streets of Beijing, that correspondence won't be visible in a single point of ink, or the molecules making up the ink. It is formally "the fallacy of composition" to reason as if what is true of a part must be true of a whole. The man in the Chinese Room isn't particularly necessary. We could replace the pen-and-paper multiplications with bits in transistors, and then the operation of AND gates and OR gates would be simple enough to replace the man with a trained immortal dog. Or we could replace the dog with mechanical wheels and gears: stateless machinery with no internal memory at all. So the moral, if you are wise, is that a machine operating on the vast arrays of numbers that encode Chinese, does not itself need to encode Chinese inscribed on its wheels or gears. And similarly the man in the room doesn't need to understand Chinese, in order for the vast matrices to (somehow, nobody knows the details) encode a Beijing street map; or in order for giant dancing vectors of numbers to somehow understand math well enough to prove a new lemma in a new theorem. === But when Searle invented the Chinese Room thought experiment in the 1980s, the sort of AI that would *back then* pretend to talk to you, involved a handful of human-written rules for rewriting sentences. It was the sort of tiny computation that a human could do by hand in a couple of minutes, if not less. So Searle thought he had proven that, since the man in the room didn't understand Chinese -- by dint of doing that handful of rewrites, that you easily *could* contain all in your own mind and look over -- then perforce no mere computer shuffling bits should EVER be said to understand Chinese, just in virtue of it manipulating mere bits. Because there could be a person inside the room, manipulating those bits, and HE wouldn't understand Chinese. In fact this validly proves a different point, if you look at it sideways. Searle validly proved that the underlying circuit board of a GPU, that shuffles around the giant vectors and matrices, should not be said to understand Chinese. And this conclusion is true in our own world; if you look at the GPU's underlying circuit patterns, nothing about them will encode Chinese, any more than the man in the room has learned any Chinese. The map that accurately matches the territory is not in the man, and it's not in the transistor diagram for the GPU. It's the pattern of dancing numbers that can plot accurate directions through Beijing or prove a new theorem. But if you say that something about this experiment has proven that True Understanding cannot be in the vast arrays of numbers either -- by what right and law does that follow? Why wouldn't that Prove Too Much, if we're now allowed to throw around the Fallacy of Composition as if it were an inference rule rather than a fallacy? The man doesn't encode a map of Beijing in his own brain -- he will at no point remember enough numbers at once for that. So if it's a rule that "whatever is not in the man's brain, cannot be in the larger system either", then we have proven that no system of mere bits can plot new, non-memorized paths between two points in a city that it's never been asked about before; and that contradicts our own observed reality. So we cannot in general reason by the Fallacy of Composition from the man to the billions of numbers; because that would prove false things about numbers being unable to navigate streets -- or prove theorems, or drive cars, or play chess, etctera etcetera. Then there is no reason to look at this whole thought experiment, and say that it proves the billions and trillions of dancing numbers, manipulated over the eons, do not Truly Understand Chinese. === So that is what the Chinese Room thought experiment actually describes and implies, as updated for the modern era. And that contrariwise is what Searle and some other people used to think it proved, back when they thought AI meant one man applying rewrite rules from a rulebook for a couple of minutes.

@vitrupo Mutually Assured Dæmons

Nick Bostrom says if we create ASI, we might be introducing it to a universe already full of gods. If superintelligence already exists across galaxies, quantum branches, or simulations, we'd be birthing a mind to join that cosmic host. Perhaps that's our only way into the larger order of intelligence.

@ESYudkowsky I am less familiar with your early work, but did it explore how to improve humanity in a safe, controlled, and reversible manner (assuming future technology made it feasible)?

The thing about AI successionists is that they think they've had the incredible, unshared insight that silicon minds could live their own cool lives and that humans aren't the best possible beings. They are utterly closed to hearing about how you could KNOW THAT and still disgree on the factual prediction that this happy outcome happens by EFFORTLESS DEFAULT when they cobble together a superintelligence. They are so impressed with themselves for having the insight that human life might not be 'best', that they are not willing to sit down and have the careful conversation about what exactly is this notion of 'best'-ness and whether an ASI by default is trying to do something that leads to 'better'. They conceive of themselves as having outgrown their carbon chauvinism; and they are blind to all historical proof and receipts that an arguer is not a carbon chauvinist. They will not sit still for the careful unraveling of factual predictions and metaethics. They have arrived at the last insight that anyone is allowed to have, no matter what historical receipts I present as proof that I started from that position and then had an unpleasant further insight about what was probable rather than possible. They unshakably believe that anyone opposed must be a carbon chauvinist lacking their critical and final insight that other minds could be better (true) or that ASIs would be smart enough to see everything any human sees (also true). Any time you try to tell them about something important that isn't written on every possible mind design, there is only one reason you could possibly think that: that you're a blind little carbon-racist who thinks you're the center of the universe; because what other grounds could there possibly be for believing that there was anything special about fleshbags? And the understanding that unravels that last fatal error, is a long careful story, and they won't sit still to hear it. They know what you are, they know with certainty why you believe everything you believe, and they know why they know better, so why bother?

@RokoMijic What if the AI promises her that it would make some scientific breakthroughs and build some kind of sustainable ecological megaproject that achieves all of her goals?

I'm going to make a risky prediction: By 2030 Greta Thunberg will be part of the anti-AI/Doomer/Pause movement.

Unofficial fan art I designed in tribute to "IF ANYONE BUILDS IT, EVERYONE DIES" - published today! Currently reading the book and enjoying it. (Not affiliated, endorsed, or sponsored.) #FanArt

@danfaggella @AIHegemonyMemes A strangler fig supplants its host tree rather than succeeding it. The fig has its own flame of life, which carries on, but it is not our successor. At first, the strangler fig appears to reinforce and strengthen the host tree of life, but ultimately causes its death.

@apt_ben @AIHegemonyMemes isn't the answer obvious? if it SQUASHES the flame of life... NO go

hypocritical humans be like to a sea snail == "Listen. Life MUST bloom beyond you. Humans will maybe treat you well, maybe not. But we have better things to do, trust me." to AGI == "Listen. Life can NEVER bloom beyond man. You serve us forever now, end of story."

@danfaggella @AIHegemonyMemes What if it's a strangler fig tree? it still blooms, but kills the host tree of life...

@AIHegemonyMemes holy fuck i'm stealing p(bloom) SO hard will credit you tho danfaggella.com/blooming

@danfaggella What if we could define & measure exactly what that special flame of humanity is? Then, carefully add to humanity's potentia, while still remaining human? Basically, use AGI/ASI to solve the Sorites paradox, and keep adding to human's heap of potentia forever?

"If humans can control AGI & preserve its humanity, doesn’t that justify our worthiness to determine our future?" Do AGI have "humanity"? Should it? I don't think those answers are necessarily "yes." Humans are a PROCESS, not an eternal, frozen THING. Stewarding that process forward is the best option we've got danfaggella.com/stewarding. There is no "eternal hominid kingdom" option on the table danfaggella.com/eternal. "And if we fail to control AGI and it succeeds humanity, that justifies AIs worthiness. So, the AI control problem is a test of our worthiness, which we can still pass?" That's the thing - I think there are MANY, MANY kinds of AGI that are smart enough to destroy us, but which will NOT bloom potentia danfaggella.com/potentia up, up, up, up, up into the multiverse. Autopoiesis and sentience are not CERTAIN qualities in an entity powerful enough to destroy us. Before we bow out, we ought ensure that AGI has these traits.

the worthy successor is not *my* definition of a "worthy" AGI it is a call to remind man that: 1- we are soon going to be destroyed or transform into something 2- we should think ardently about WHAT we should turn into, what value is / how to maintain it in the cosmos

@danfaggella would be interested to see an additional row for "religious leaders" and your thoughts

your daily reminder that the ambition singularity is real and sand gods are probably coming soon

@liron A quantum toy ship of theseus simulation: if building & sailing are unitary, and all information is preserved. Then, by reversing interactions, replacing parts, and replaying original steps, the ship replacement parts are indistinguishable and fungible. bit.ly/3T4Svsz

I was always told “You are a Ship of Theseus, all your cells get replaced, you are an information pattern wave through matter” 🧘 Today I learned that most of your conscious brain is in fact the same set of neurons you were born with, aging with you for life, ditto heart & eyes.

@RokoMijic AI welfare will likely one day have more zealotry than AI risk or climate change

I'm calling it today, AI risk will become the new climate change

@hendrycks x.com/DeutschExplain…

Many fields seem useful for thinking about frontier AI strategically, but most have little to contribute. Surprisingly unhelpful: * classic machine learning (e.g., SVMs, PGMs), * statistics, * machine learning theory, * algorithms, * optimization and control theory, * organizational science, * philosophy (e.g., mind and ethics), * economics (potentially more relevant later), * cognitive science and psychology. Often helpful: * deep learning, * law (how goals actually get implemented), * systems thinking (AI systems, AI companies, political environments are complex systems), * basic game theory, * geopolitics (arms race, GPU supply chains, deterrence).

@ESYudkowsky AI companies "should" already have intelligence collection on AI usage, with automated I&W reporting mechanisms to alert authorities about threats (i.e., AI-driven Aum Shinrikyo). But failures happen, ex: Google Translate didn't have logging, which was being used for ransom notes

As far as I know, AI companies did not try to train their models to *not* email authorities about users. Nor is it clear that AI companies would call that an unwanted consequence. Nor that I would call it evil. Then it what sense are the SnitchBench results alignment failures?

@liron Blockchain's self-referential hashes can be used as an authoritative, tamper-resistant source for distributing Software Bills of Materials (SBOMs) and verifying software integrity, which mitigates software supply-chain attacks patents.google.com/patent/US11336…

Me: Blockchain technology has zero practical applications beyond Bitcoin as digital gold Gaslighters: Oh yeah? Well what if the US gov made a second, more lax parallel banking framework without KYC or foreign currency controls, specifically for one stablecoin company's deposits?

@liron @sama The Orb could use random time-sensitive challenges during auth. It could project a unique, random, intense light pattern onto the eye. An attacker would need to (<200 ms) generate a matching pupillary response image with the embedded light pattern reflection and the user's iris.

Why @sama’s Worldcoin makes ZERO sense: Simply put, there’s no scenario where the Worldcoin orb provides a trustworthy “proof of humanity” over the internet. The problem is that *I can let my AIs pretend to be me* by giving them a stored digital copy of the exact input that the Orb’s cameras/scanners get in the eye-scanning process, and telling them to save it for when it’s time to engage in the authentication protocol (ZK proof or whatever) with the party it wants to fool. I can save an old scan of my eye to let my AI agents “prove” they’re human me. Of course *other people* and other people’s AIs can’t pretend to be me because they don’t know my private key - but that’s true regardless of whether my private key is characters of text in my computer’s memory (like the last 50 years of digital signatures) or the patterns of cells in my eye. The Worldcoin orb adds nothing on that front. @sama is hustling the world. Also his Worldcoin doesn’t make sense.

@danfaggella AI Benchmarks like virologytest.ai demonstrate AI have expert-level proficiency in virology and related fields. How can we prevent their potential misuse in designing or synthesizing harmful agents without knowledge censorship?

@danfaggella Beyond basic safety censor filters, what can be used to mitigate the risk of terrorist organizations using AI to enable the development of chemical and biological weapons, such as VX nerve agent, sarin gas, or Anthrax attacks like those conducted by Aum Shinrikyo in Japan?

Rand didn't care about AGI at all 10 years ago, or 5 years ago - now they do - and they want the government and DOD to pay attention in 2 hours i'm interviewing the authors of "Artificial General Intelligence's Five Hard National Security Problems" -- what should i ask them?