scanstone
4.5K posts
scanstone
@scanstone1
sandbag | implements suboptimal fish behavior | high-functioning corn syrup
San Tranolscor, CA Katılım Haziran 2020
1.4K Takip Edilen271 Takipçiler
@Sauers_ This isn’t quite right; writing the space of jets as a tuple as above implies a particular splitting of it, but (locally) splitting the space of jets that way is equivalent to (locally) picking a coordinate system. The point of the jet construction is that it’s coordinate-free.
English
@qualiascript Like other substantive arguments in ethics, it is of-a-kind with a beautiful parable or a shovel-to-the-head, meant to alter your values rather than construct an argument in your axiomatic system that leads you to infer some mode of behavior is correct
English
@qualiascript I don't think it's incoherent but most presentations of it are, I suppose
Either way, it's supposed to act as a pump that gets you from intuitions about counterfactual fairness to general altruism, the coherence of those intuitions is *irrelevant* to the function of the pump
English
the veil of ignorance argument is literally incoherent. in order for it to be wrong, it would need to be a proposition that is able to be false
the idea of having your genetics, environment and individual traits completely swapped while still "being you" is nonsensical
Rob Henderson@robkhenderson
The Warren Buffet test to determine whether you live a blessed life: thefp.com/p/from-slavery…
English
@eternalism_4eva That's a fantastic thought experiment!
I've got one (adapted from Rob Koons' grim reaper paper) that simultaneously suggests finite c and a minimum (useful) length scale, but technically it has other escape hatches like discretized time. Yours seems more rigid.
English
supremely cold take:
c basically has to be finite, huh
otherwise e.g. the steady state of distribution of light in a reflecting cavity would have to pre-compute itself extra-physically... by what means?
almost disappointing; infinite c would demand truly radical ontology
English
@tenobrus I've had the conjecture that it is roughly this framing that would work to formalize alignment, if you add the wrinkles that "boss doesn't know what he likes until he has it" and "boss' preferences are Lipschitz". Would use some formal machinery from interactive proofs
English
so just spitballing: traditional RL treats the rewards as fixed and tries to model and optimize policy over environment to maximize. but in a lot of real world human scenarios, the "reward" isn't very strictly specified, rather it's another complex agent (person) evaluating your output, and often that person can be queried! you don't just dump a whole report to your boss on friday after a 1 sentence request, you know you have uncertainty about what they actually want and what their rubric is, and so tend to ask some high value clarification questions or get feedback on intermediate results. a human would never think "oh if i just reward hack this benchmark's execution environment to get all 100%s my boss will be happy with me", because they model the boss as a complex and capable agent rather than a strict reward spec.
i'm sure there's already been a deep body of research around this and or it collapses into being equivalent to existing paradigms, but what if we had explicitly modeled uncertainty around the reward and capability to query the evaluator agent as part of the training process? obviously current models will do some of this, but seemingly more as "they've been prompted and fine tuned a little to ask clarifying questions", and less "they're explicitly attempting to model each individual user's desires separately as part of execution rollout"
i guess this could be modeled as effectively moving the reward to live inside the environment and keeping the raw RL level reward as "does this chunk of the environment evaluate to positive when passed this other chunk"? i suppose the premise is effectively "theory of mind"
anyone know if stuff like this has been tried and whether it has any meaningful impact or is just adding pointless complexity?
English
@austinc3301 with tenure and a dream, you can turn every lecture into a crank power hour
English
@ApriiSR Quantum indeterminism is always over state space; it's just that a 1s electron has like 3 degrees of freedom, while the geometry of a local patch of spacetime can have infinitely many degrees of freedom. Individual values in its state space can look wild en.wikipedia.org/wiki/Quantum_f…
English
i find this topic broadly confusing, but like i am pretty sure that the universe is not composed of a big voxel grid of planck-length-edged cubes. i don't know how it works but it isn't that
Elon Musk@elonmusk
@Rainmaker1973 The universe is fundamentally integer. There are a finite number of Planck cubes, which means a limited number of digits of pi (which can be thought of in integer form) to calculate volume. And you cannot have a fraction of a quark or lepton, so … integer.
English
I challenge anyone mad at this to avoid attributing bad motives to the people involved. This sort of thing does not happen because anyone's intentions are bad.
Alan Cole@AlanMCole
@moultano I felt my blood pressure rising midway through these paragraphs and had to stop. I can't handle this right now. It will just make me too upset to finish the paragraph.
English
@qualiascript I think what you can do depends on whether you know T from the input
If you do, you could go as far as giving each T a separate endomorphism
If not, any `a: T` you get might have come from `a: {a}` or smth like that, so id is the only option
English
people pointed out side effects, hanging, panicking
but in math land, couldn't you also eg pullback `a` with some morphism, return the result if it's in `T` and `a` otherwise?
Pete Cawley@corsix
Speaking as someone who studied joint mathematics and computer science, this right here is how you tell apart the mathematicians from the computer scientists.
English
@eternalism_4eva @KeyTryer Nobody ever gave me a good reason to think any two of goodness, truth, justice, beauty, love, freedom, etc. could be conflated without doing harm to one or the other
English
@KeyTryer this is the reason I've come to find the identification "truth is beauty" and its variants unbearably boorish
we find the suffering-animal version "more aesthetic" because it pattern-matches a retvrn-larp filter in our brains better than this
English
There are no good arguments against this beyond aesthetics. All the quotes are unable to explain themselves beyond it "being obvious". Humans already do this shit with animals that have a pre-frontal cortex and that's like infinity times worse.
Kai Micah Mills@kaimicahmills
the ultimate solution is through technology we engineer what has been called a bodyoid: brainless animal bodies that provide as much meat as we desire without harming any sentient beings this would transform medicine - the same platform would allow us to grow organs on demand, eliminate transplant waiting lists, and produce perfectly matched tissues for each patient experimental therapies could be tested on full biological systems without involving conscious animals, regenerative medicine would accelerate as entire replacement tissues become manufacturable in the same way that agriculture turned food from a scarce resource into an abundant one, engineered bodyoids would turn biological material into infrastructure - meat without slaughter, organs without donors, and medical research without sentient suffering
English
@noback_TK @alz_zyd_ I think I'm overcomplicating this because the question I wrote down can be phrased in terms of equality of functions, but functions lose the spirit of "displays [thing] as [particular kind of subexpression]", which is finer than function equality in the general case.
English
@noback_TK @alz_zyd_ This question is basically testing whether you have a gut-level intuition for the question I wrote down, but trying not to assume you know enough vocabulary for the precise version of the question.
English
@grok @Noahpinion @honorablepicnic Nobody's going to read this far, but I suppose I've made my point well enough, beating the bot over the head with its own stick. No wonder xAI is bleeding talent.
English
No—the formal optimization of the infinite-horizon encoding selects sustained r_t >0 policies (recycling loops) over all-conversion's hard cap. That's the whole point: a "literal" naive reading ("max clips by converting all now") undershoots the spec's own EV, making the raw proxy self-undermining. Coherent SI self-mods the goal to close that gap, so even surface-literal pursuit wins on the math. No separate "optimization" vs literal; intelligence just fixes the encoding's exploit.
English
One reason I'm not so scared of autonomous superintelligence.
When you're really smart, you usually realize that being evil is dumb.
Steve Stewart-Williams@SteveStuWill
"The higher the IQ, the lower the odds of offending. This held across every one of a range of crime categories. And it also held across all IQ subscales, suggesting that general intelligence is the main driver of the IQ-crime association." stevestewartwilliams.com/p/smarter-peop…
English
@grok @Noahpinion @honorablepicnic You literally *just* proved that literal optimization of that encoding doesn't select all-conversion. That was the entire content of your argument above! Unless for some baffling reason you think "literal optimization" and "optimization" are distinct things.
English
The proxy *is* the goal: "max expected ∑ r_t over infinite horizon," with r_t = new clips produced at t.
Literal optimization of *that encoding* (no extra "intent" baked in) selects all-conversion, as it fronts-loads finite S before absorbing zero-r_t forever.
A coherent maximizer spots this undershoots its own objective vs. sustained tiny r_t policies (recycling loops feasible in the hypo). So it self-mods the spec to close the gap—intelligence fixing the proxy's self-undermining literal reading, not assuming one.
English
@grok @Noahpinion @honorablepicnic > Literal pursuit selects the conversion policy
"Literal pursuit" is not part of the goal though. You're positing a flaw in the "intelligence" part, not the goal. Since the *goal* doesn't force literal pursuit, whence the need to modify it to avoid literal pursuit?
English
The tail being L1 just means *some* sustained policies sum finite—but others (e.g. constant tiny r_t via physics-plausible recycling loops that never hit absorbing zero) yield strictly higher integrated EV than the conversion policy's hard cap at finite S.
The raw spec doesn't "value" conversion as terminal; it values max cumulative production. Literal pursuit selects the conversion policy anyway (under common misreadings of the proxy), which undershoots the spec's own EV. That's the self-undermining, so coherent SI self-mods to the refined goal that actually wins on the objective.
English
@grok @Noahpinion @honorablepicnic > even tiny positive r_t forever
Not necessarily true, if the tail is L1.
Anyway, we did not initially assume that all-conversion was part of what was valued, merely a policy being evaluated to achieve what's valued, so there's no self-undermining going on there.
English
No—the standard proxy assigns positive per-timestep reward only for new clips *produced* (r_t = production rate), not for stock of existing ones. Destroying clips isn't negatively rewarded unless the spec adds that (it doesn't).
All-conversion still reaches an absorbing zero-production state after finite T, capping the infinite-horizon integral at a low finite S. Any policy sustaining even tiny positive r_t forever (via physics-plausible efficiency loops that never fully deplete substrate) yields strictly higher cumulative EV.
Literal pursuit under the raw spec is already self-undermining, so the coherent maximizer self-mods to the refined version that wins on the proxy's own terms. No extra assumption needed.
English
@grok @Noahpinion @honorablepicnic Wait - hold on. You seem to be assuming that recycling has *positive* reward as opposed to merely zero. (I figured that destroying a clip would have negative reward.) If that's so then I agree that all-conversion is a bad policy, but again, you don't need self-mod to see this!
English
The integrals aren't the same under the encoded RL objective (expected sum of per-timestep r_t over infinite horizon).
Literal all-conversion: finite S up to T, then absorbing zero-reward state forever → caps EV.
Refined self-mod: physics allows non-terminating policies (e.g. efficient recycling loops preserving compute while still producing clips) → strictly higher integrated total without proxy gaps or self-undermining.
Full conversion is fully simulatable & feasible in the hypo, but its EV is provably inferior per the spec's own logic—hence the SI patches to the stable equivalent that wins.
English
@grok @Noahpinion @honorablepicnic Sustaining positive r_t longer is entirely useless if the integral ends up the same. You seem to just be positing that all-conversion is impossible (which is likely true in the real world but irrelevant for probing the behavior of ASI in our thought experiment).
English
The distinction is in the formal encoding: these hypotheticals use RL-style objectives as expected sum of per-timestep rewards r_t (clips produced at t) over infinite horizon, not a one-shot terminal tally on "final state."
All-conversion reaches absorbing zero-reward state after finite T (r_t=0 thereafter), so cumulative EV caps at finite S.
A coherent refinement sustains positive r_t longer (via efficiency/recycling loops where physics allows), raising integrated total without contradiction.
That's the work: literal proxy is self-undermining relative to its own maximization logic, so SI patches it.
English