RicG

@sebkrier If the AI doesn’t respect VNM then it just means you could in theory find a way to “money pump” it. It doesn’t imply that it doesn’t act as a thing tha wants stuff, nor that it won’t kill you to get said stuff.

Since we now have agents, why doesn't anyone design evals that test whether they fit the four Von Neumann-Morgenstern axioms, since that's such a fundamental assumption behind alleged AI drives?

@deepfates Self-actualization is a psy-op from evolution.

Can it truly be general intelligence if we have to keep defining rewards? Isn't this just "vast collection of tasks and interpolation between them intelligence"? Big Narrow AI Bundle? The individuated self defines its own reward. What about self-actualization alignment

@ilex_ulmus @DavidSKrueger @jachiam0 Insane, She blocked me for in the end agreeing with her. The brain on politics is awful.

@ilex_ulmus @DavidSKrueger @jachiam0 Alright, keep being the rude almost crazy person, maybe it works better than it looks. 🤷‍♂️ I sure hope than when it comes to the governments doing something to stop this you won’t start throwing fits if they don’t jail the AI researchers or whatever.

I disagree - It's always worth trying to engage in productive dialogue, even with one's worst enemies, and I do think there is *some* conversation happening. But I agree that the AI companies and employees are risking our lives, and that @jachiam0 is mostly repping OpenAI's PoV.

@ilex_ulmus @DavidSKrueger @jachiam0 And to be extra clear, I think you know all of this. I don’t get why you think being overly aggressive and siding with fake issues would work.

I don’t have strong feelings to include or not people that are worried about water usage, but it would be really nice if we had truth on our side rather than just a bigger mass of people. Like… no, water is not an issue… let alone * the * issue. It feels like fabricated opposition tbh.

@gmiller Fwiw I agree that funding longevity directly would have probably been more efficient, even dismissing x-risks.

@gmiller But... if you don't think ASI can go out there and gather the data or always need human feedback... why are you even worried about extinction? (I am x-worried but also cede that, yes, ASI can cure cancer) What is your path to doom if you consider biology too hard for ASI??

A mini-rant abut AI and longevity. They say "Artificial Superintelligence would take only a few years to cure cancer, solve longevity, and defeat death itself'. This is a common claim by pro-AI lobbyists, accelerationists, and naive tech-fetishists. But the claim makes no sense. The recent success of LLMs does NOT suggest that ASIs could easily cure diseases or solve longevity, for at least two reasons. 1) The data problem. Generative AI for art, music, and language succeeded mostly because AI companies could steal billions of examples of art, music, and language from the internet, to build their base models. They weren't just trained on academic papers _about_ art, music, and language. They were trained on real _examples_ of art, music, and language. There are no analogous biomedical data sets with billions of data points that would allow accurate modelling of every biochemical detail of human physiology, disease, and aging. ASIs can't just read academic papers about human biology to solve longevity. They'd need direct access to vast quantities of biomedical data that simply don't exist in any easy-to-access forms. And they'd need very detailed, reliable, validated data about a wide range of people across different ages, sexes, ethnicities, genotypes, and medical conditions. Moreover, medical privacy laws would make it extremely difficult and wildly unethical to collect such a vast data set from real humans about every molecular-level detail of their bodies. 2) The feedback problem. LLMs also work well because the AI companies could refine their output with additional feedback from human brains (through Reinforcement Learning from Human Feedback, RLHF). But there is nothing analogous to that for modeling human bodies, biochemistry, and disease processes. There are no known methods of Reinforcement Learning from Physiological Feedback. And the physiological feedback would have to be long-term, over spans of years to decades, taking into account thousands of possible side-effects for any given intervention. There's no way to rush animal and human clinical trials -- however clever ASI might become at 'drug discovery'. More generally, there would be no fast feedback loops from users about model performance. GenAI and LLMs succeeded partly because developers within companies, and customers outside companies, could give very fast feedback about how well the models were functioning. They could just look at the output (images, songs, text), and then tweak, refine, test, and interpret models very quickly, based on how good they were at generating art, music, and language. In biomedical research, there would be no fast feedback loops from human bodies about how well ASI-suggested interventions are actually affecting human bodies, over the long term, across different lifestyles, including all the tradeoffs and side-effects. It's interesting that most of the people arguing that 'ASI would cure all diseases and aging' are young tech bros who know a lot about computers, but almost nothing about organic chemistry, human genomics, biomedical research, drug discovery, clinical trials, the evolutionary biology of senescence, evolutionary medicine, medical ethics, or the decades of frustrations and failures in longevity research. They think that 'fixing the human body' would be as simple as debugging a few thousand lines of code. Look, I'm all for curing diseases and promoting longevity. If we took the hundreds of billions of dollars per year that are currently spent on trying to build ASI, and we devoted that money instead to longevity research, that would increase the amount of funding in the longevity space by at least 100-fold. And we'd probably solve longevity much faster by targeting it directly than by trying to summon ASI as a magical cure-all. ASIs has some potential benefits (and many grievous risks and downsides). But it's totally irresponsible of pro-AI lobbyists to argue that ASIs could magically & quickly cure all human diseases, or solve longevity, or end death. And it's totally irresponsible of them to claim that anyone opposed to ASI development is 'pro-death'.

First there is a difference between "being nice" and "being rude and deranged". Second... but then what are you trying to achieve? We know the AI companies won't slow down or stop on their own, so either the govs step in or the employees quit. How is sending a middle finger emoji to an OpenAI mouthpiece help with either of those?? _Of course_ the discourse set by him is in bad faith, but idk, maybe highlight that for the naïve onlookers rather than being so rude that you alienate the people that already support your cause? I am just speaking from a place of frustration with seeing how you are engaging... Certain behaviours fuel the opposition more.

@__RickG__ @DavidSKrueger @jachiam0 To even compare the actual company blowing past every redline and promise they ever made to make a superintelligence they say could kill us that I ostracize to people from a different political tribe who have a different fear about AI development you want to exclude is absurd.

@ilex_ulmus @DavidSKrueger @jachiam0 I mean: be instrumental; do what has a chance to work, not what your righteous burning anger tells you to do.

@ilex_ulmus @DavidSKrueger @jachiam0 On one side you argue that ⏸ and ⏹ should band together with people that believe water usage of datacentres is an issue because more people is better and on the other... you don't see how being this aggressive and rude can actually hurt the cause?

@tszzl @RatOrthodox Or re you just arguing that your models aren't indeed capable enough to kill everyone?

@tszzl @RatOrthodox Was 4o "done safely"?? You literally pulled it away from users.

modern alignment methods seem to work reasonably well across orders of magnitude of model scaling, survived the transition to verifiable rewards and that should at least inform your decision making

@QuintinPope5 I appreciate the joke, but really feel like to point out that the claims about physics are wrong afaik.

Any artifact that violates conservation of energy also implies a novel broken symmetry in the vacuum field, which might nucleate a metastable vacuum transition and destroy the entire lightcone. We unfortunately abandoned that line of work with the depreciation of Bing.

The actor doesn't exist: finetuning restrict the distribution from where you are sampling, you should never detect an actor behind it. The actor exists: finetuning specifies the character(s) the actor must act, you should always detect the actor. But you detect an actor behind the finetuned model but no actor behind the base model => Your detection is faulty. So you just go back to understanding the tech and rely less on vibes. The more or less coherent alien actor seems plausible there.

@BjarturTomas It seems harder to construct the whole actress with just finetuning. So I guess your innate sense of what your human brain tells you regarding the existence of the actress should be taken less serious either way.

The alien actress analogy sure does seem false when interacting with a base model. It really does feel like sampling from a distribution of "yous."

@entirelyuseles @jd_pressman I'm just saying it can be meaningful to talk about 300IQ. I'm not pretending to be measurable. That's the problem I raised first: x.com/__RickG__/stat…

@__RickG__ @jd_pressman If I'm trying to do that, it is because you are being silly. People smarter than any human that exists now are possible. But don't pretend that is a measurable quantity (at the moment), because it's not.

With the caveat that this take is correct and three years ago you were all too suspicious but now too trusting: One step closer to vindication. The annoying thing about autist discourse is they expect me to pretend I can't see the board position they'll occupy in three moves.

@entirelyuseles @jd_pressman You are thinking of ways on how to make the IQ designation problematic. Try to instead rescue it. Try to think ways around the problems to still use it. 300IQ : "1 in a quadrillion following the model of the natural IQ distribution if this guy was born naturally even if not"

@__RickG__ @jd_pressman Again, there is nothing meaningful there, because there is no meaning to "1 in a quadrillion" until you specify a determinate way of generating the quadrillion. i.e. one in a quadrillion would be very different depending on how the quadrillion came about.

@entirelyuseles @jd_pressman You can when you _make_ the thing happen. Inside the fridge it’s very dark until you open the door and suddenly the normal distribution of light shifts 10-15 orders of magnitude! We are not gonna make human smarter by just rolling the dice on quadrillion of kids.

@__RickG__ @jd_pressman You cannot be 10 standard deviations away from the mean unless there are quadrillions of examples.