Raymond Arnold

@scoopdiddy1 @robertskmiles I mean ask a lot of nuanced questions. But, I was figuring he had a more streamlined system.

@Raemon777 @robertskmiles What would you do to check that a human understands?

Writing advice: Give your essay to an LLM, see where it misunderstands you, and fix it. Once the LLM understands everything, switch to a smaller model. If Haiku misunderstands something, some humans will too.

@xuenay yeah fair.

@Raemon777 Oh :D didn't think that counted as particularly recent

I used to think the Star Trek world was unrealistic in terms of how it used AI. But some recently argued to me, it's actually very straightforwardly a world where everyone knows they *could* build more advanced AI, but, they know that the alignment problem is unsolved. So, they don't. Instead, they limit themselves to LLM-like AI, which operates on discrete tasks. Data is a one-of-a-kind wonder people don't know how to replicate. Pretty much every other time someone tries to build advanced AI, something goes wrong. (Data's creator made a second android, named Lore, who was erratic and manipulative, and eventually turned against the humans) Most other advanced AI in the show either grow into godlike power outside human control, or get shut down while weaker but would clearly become a problem if unchecked. (V'ger, Moriarty). The more I looked at it, the more it seemed Canon Star Trek just straightforwardly depicts an adult civilization that has chosen to be careful.

@xuenay You were one I was referring to with "some recently argued"

@Raemon777 Yeah I had the same conclusion: lesswrong.com/posts/7N7JGyTm…

@benlandautaylor I'm interested to hear back when you're done.

I finally started reading Moby Dick and I'm blown away, it's absolutely *riveting* and so far he's just ruminating while he wanders towards the port. This is gonna rock once he's actually on the ship.

@EpistemicHope yep (for non-paying users)

@Raemon777 They have ads now?

The problem here is real, but this analysis is of why it occurs is mistaken. The AI companies are NOT incentived to maximize engagement the way that social media companies are, because they have a different business model. 🧵

@EpistemicHope I think this is false now for OpenAI?

The AI companies are different. So far, they don't make money from ads. Currently, their revenue comes from subscriptions. Unlike serving webpages of user-generated content, running inference on their AI models is a cost. They only have so many GPUs.

@TylerAlterman Warhammer 40k: The Musical youtube.com/watch?v=59NORG…

Retrospective: what were the best April Fools jokes this year?

@1a3orn Yeah same.

@Raemon777 And post hoc I have some explanations for why this is important about context rot, which I've ported into my general concept of intelligence not just for LLMs (how much of my memories are just overfitting, would I be better without them) But hard to have ahead of time.

I'm a bit retroactively surprised that, before LLMs, I... don't recall any sci-fi stories where the AIs operated in short bursts of thinking, each mediated by a human. Or, where the AI is in a "memento" situation where it keeps getting reset. The story I recall that at least touches on this is some novel in the Star Wars extended universe, where it's remarked that droids are supposed to get "reset" periodically so they don't get wonky. Luke hasn't reset C3PO, which is part of why 3PO has acquired such a personality, and has maybe become sentient (which apparently isn't normal). Accelerando's early chapters has the main character send little AI agents off to do stuff, which seem like they could at least in principle be something like an OpenClaw instance, but it's not super specified. It is interesting that this feels like a relatively obvious story concept (in retrospect) but it didn't come up.

Yeah I wish I could edit the OP to say "the 'reset' thing is not actually central'", it's related but a different thing than the world where it is just totally normal to not even have ever gotten to the point where there was an object with continuity in the first place who could complain about getting reset.

@Raemon777 The possibility of getting reset does come up in the 2010 "The Lifecycle of Software Objects," where "do I reset my virtual child" is an ethics thing and one that the software objects object to at some points. But played for contrast with continuity rather than recurring.

To clarify: the C3PO "reset" thing isn't a central example of what I meant, it was just the closest thing I remembered offhand. The part that feels interesting is that the fundamental structure of the AI is such that it just goes in short little bursts and then winds down, and each instance has no memory except what you choose to give it. (and, the the part where you can swap memories into/out of different AIs while keeping the overall superstructure the same)

@JeffGraw Yeah I agree LLMs are basically Star Trek.

@Raemon777 If you assume we never fundamentally get past LLMs, but continue to scale, and consider what AI might look like in a hundred years or so, then that might look a lot like the Enterprise computer.

I think even without the "token-specific" model, something like "the AI thinks for a chunk of time, you can basically see a stream of what it's thinking as it goes (however interpretable that turns out to be), and by default completes short tasks and then turns off... idk just seems like kinda how you'd maybe even want it to be. I think this sort of did come up with oracle AI in LW circles. I think the part that feels novel is that the structure that calls the AI is pretty straightforward to swap up out different AIs who are good at different things. The part that feels surprising is more that "ephemeral little blip of AI consciousness" didn't show up as a story premise for AI character development. (People are suggesting various stories that featured some of this. I haven't looked through them all yet. Many of them are about "getting reset" like C3PO was supposed to, but, that was more of an edge case that isn't really the thing.)

@Raemon777 The idea that you can just string tokens together one at a time and these can function as powerful thoughts (maybe even achieving 100%+ of what thought does) is in retrospect deserving of high plausibility, yet never occurred to me as a plausible scenario.

@crystalwizard That is in fact what happens at the beginning of Contagion.

you probably didn't read any pandemic stories where the first thing everyone did was buy all the toilet paper, either - but that's what happened with covid you can't base real reality on what happens in fiction. science fiction is, and always has been, cautionary warnings about the possible future that the writer has extrapolated for whatever reason. not a book of absolute 'this will happen' facts.

@flutterwhat Yeah I actually was pretty into Marathon back in the day.

@Raemon777 in many of bungie's stories for Marathon and Halo, AI becomes 'rampent' in 7 years time. Rapnecy is when an AI can no longer be predicted or controled, to put it lightly. Durandal is an amazing story of a rampent AI causing tragedy that still plays out in the modern marathon

Nod, but, the point of CEV is roughly to operationalize how to deal with that sort of thing. (I really recommend reading the whole thing. I think there are reasonable disagreements one could have with it but they did think through a lot of the obvious things and I think most people are imagining a straw version of it). But, the short answer is "insofar as people have deep conflicts, the AI doesn't intervene on those conflicts, it intervenes in the specific places where people turn out to want the same things." It's not exactly spelled out but I think fairly strongly implied, it might include things like "well, it turns out there are some deep conflict even after doing a good reflection process. Do you have any metapreferences about how to handle that?" And it might turn out that people agree with things like "I would opt into the world where the AI restricts my ability to do violence in return for it also restricting my enemies' ability to do violence" so conflicts can be resolved more peacefully. (Or, something like that).

@robbensinger I should read that page soon but the high-level issue is that it's easy to gloss over "just don't be a dick" without addressing the deep issues with operationalizing that when you have people with profoundly conflicting preferences

I feel like the main reason people are skeptical of CEV is just that (a) they haven't read lesswrong.com/w/coherent-ext…, and (b) CEV has a weighty philosophical name rather than "Six Rules of Thumb for Not Being a Dick (in the context of possessing effectively unlimited power)".

@almostlikethat Sci fi didn't seem to be otherwise shying away from existentially horrifying things tho

@Raemon777 I think it didn't come up much in mainstream scifi because It is kind of existentially horrifying (the way Memento itself was grappling with, once that way for a person-shaped-thing to be was "part of the text") and they were not interested in writing about existential horror?