SI murders people. a lot.
9 views
Skip to first unread message
Max Kaye
Sep 9, 2020, 6:03:16 AM9/9/20
to Fallible Ideas List
Looking for feedback; thinking of posting to LW. I'd probably change some bits so I didn't sound as much like an
outsider.
----
Posters on LW often bring up the idea of emulating a *person* to establish some bounds on complexity of SI programs.
The problem with this idea is that an ~infinite number of people are murdered before SI finds a good answer.
Here's the common logic:
* Peoples' minds take lots of space to specify.
* People can do knowledge-y stuff: solve problems, make predictions, etc.
* SI wants to do some of the things that people do
* Minds can be represented as classical programs
* Therefore a capable mind is an upper limit on program size
I add:
* Minds take lots of space, and the important parts of the mind grow in length as the mind grows in knowledge.
I note: I've had the goal-posts of the word 'simple' changed on me in curi's code/english thread with the following:
https://www.lesswrong.com/posts/9AWoAAA59hN9PEwT7/why-would-code-english-or-low-abstraction-high-abstraction?commentId=8o3cGqb7C5SjtYt93
> When I say the 'sense of simplicity of SI', I use 'simple program' to mean the programs that SI gives the highest
> weight to in its predictions(these will by definition be the shortest programs that haven't been ruled out by data).
I'll use "SI-simple" to label that idea going forward
My further reasoning:
* SI-simple minds are more common. if nothing else, babies are SI-simpler minds. Their brains might have more neurons
(than adults), but those brains are also easier to specify b/c they're more generic.
* As we grow and learn our brains get more complex (and by 'complex' i mean both: not simple and not SI-simple).
* SI doesn't throw programs out too early, esp when they're long, so if minds are emulated they're run for enough time
to start *thinking*.
* SI has no problem killing the person in search of a better program (in fact, it's necessary).
I predict the common response will be like:
> sure, but that doesn't matter because SI will find other programs first. we don't *expect* we need programs this
> long, it's just to find an upper limit.
I assume that LW people agree that killing a mind is murder (i.e. non-humans count), and that SI programs can be minds,
and turning one off is killing it.
(If they disagreed with any of that I would be much more worried about their AGI pursuits. I assume ppl here agree with
it too)
I think this is problematic b/c:
a) there's a known lesser limit (murder) that has not been said "out loud" as far as I know
b) we don't know how large the SI-simplest mind is, or how to reason about these as SI programs
c) if we continue to not be able to reason about it, but also put serious resources behind SI research, then it's
possible "we" (humanity) start generating and killing AGIs before realising it
d) if there is a *better* way to do minds and AGI than SI, then SI would find that method along the way, but also
murder lots of new AGI people
e) if we keep not understanding AGI and SI keeps getting more funding: someone might actually try something of the
complexity necessary to do this randomly. if they optimise with stuff like evolutionary algorithms (esp if there are
some good developments in related areas), then it gets easier to end up with these sorts of programs
f) SI is basically *guaranteed* to do this if it's used to try and find programs which are also AGIs.
Now, I don't think it's likely some horrible future like this will come to pass. It's not serious in that sense.
However, I do think it's serious for fans of SI. It's a new, *important* upper limit that they haven't mentioned
before, and it rules out certain programs.
Crucially, we can't say all the problems we care about will have SI-simpler solutions than AGIs. English (or w/e) words
represented as code - generally - has an upper limit in the range of AGIs b/c *maybe an AGI + english words are the
SI-simplest way to write down some programs*. We can't know what the crossover point is for SI-simple programs to go
from non-AGI to AGI... unless we have a theory of AGI. So, whatever understanding of AGI we get, *we should not do it
**incrementally*** if that means making lots of incremental steps through a program-space "full" of AGIs.
** ALL PROGRAMS MORE COMPLEX THAN THE MOST BASIC AGIs ARE OFF LIMITS TO SI **
By the above logic I think an important conclusion is that one of the two following ideas has to be true. Option 1: AGI
researchers (as humans) are/will be mass-murdering AGIs one day - if humans work like SI does. **Or**, option 2:
there's some other way to learn stuff / create knowledge - that's what the AGI researchers are doing - **and** it must
be SI-simpler than an SI-AGI would be. If option 2 is correct, but it is physically possible to make meaningful
progress towards AGI via SI, then we're basically guaranteed to murder lots of AGIs if ppl pursue that avenue long
enough (and with enough compute resources).
I don't think AGI research *requires* the murder of AGIs.
--
Max
xk.io
I post my FI work/articles/exercise/practice here:
https://xertrov.github.io/fi
outsider.
----
Posters on LW often bring up the idea of emulating a *person* to establish some bounds on complexity of SI programs.
The problem with this idea is that an ~infinite number of people are murdered before SI finds a good answer.
Here's the common logic:
* Peoples' minds take lots of space to specify.
* People can do knowledge-y stuff: solve problems, make predictions, etc.
* SI wants to do some of the things that people do
* Minds can be represented as classical programs
* Therefore a capable mind is an upper limit on program size
I add:
* Minds take lots of space, and the important parts of the mind grow in length as the mind grows in knowledge.
I note: I've had the goal-posts of the word 'simple' changed on me in curi's code/english thread with the following:
https://www.lesswrong.com/posts/9AWoAAA59hN9PEwT7/why-would-code-english-or-low-abstraction-high-abstraction?commentId=8o3cGqb7C5SjtYt93
> When I say the 'sense of simplicity of SI', I use 'simple program' to mean the programs that SI gives the highest
> weight to in its predictions(these will by definition be the shortest programs that haven't been ruled out by data).
I'll use "SI-simple" to label that idea going forward
My further reasoning:
* SI-simple minds are more common. if nothing else, babies are SI-simpler minds. Their brains might have more neurons
(than adults), but those brains are also easier to specify b/c they're more generic.
* As we grow and learn our brains get more complex (and by 'complex' i mean both: not simple and not SI-simple).
* SI doesn't throw programs out too early, esp when they're long, so if minds are emulated they're run for enough time
to start *thinking*.
* SI has no problem killing the person in search of a better program (in fact, it's necessary).
I predict the common response will be like:
> sure, but that doesn't matter because SI will find other programs first. we don't *expect* we need programs this
> long, it's just to find an upper limit.
I assume that LW people agree that killing a mind is murder (i.e. non-humans count), and that SI programs can be minds,
and turning one off is killing it.
(If they disagreed with any of that I would be much more worried about their AGI pursuits. I assume ppl here agree with
it too)
I think this is problematic b/c:
a) there's a known lesser limit (murder) that has not been said "out loud" as far as I know
b) we don't know how large the SI-simplest mind is, or how to reason about these as SI programs
c) if we continue to not be able to reason about it, but also put serious resources behind SI research, then it's
possible "we" (humanity) start generating and killing AGIs before realising it
d) if there is a *better* way to do minds and AGI than SI, then SI would find that method along the way, but also
murder lots of new AGI people
e) if we keep not understanding AGI and SI keeps getting more funding: someone might actually try something of the
complexity necessary to do this randomly. if they optimise with stuff like evolutionary algorithms (esp if there are
some good developments in related areas), then it gets easier to end up with these sorts of programs
f) SI is basically *guaranteed* to do this if it's used to try and find programs which are also AGIs.
Now, I don't think it's likely some horrible future like this will come to pass. It's not serious in that sense.
However, I do think it's serious for fans of SI. It's a new, *important* upper limit that they haven't mentioned
before, and it rules out certain programs.
Crucially, we can't say all the problems we care about will have SI-simpler solutions than AGIs. English (or w/e) words
represented as code - generally - has an upper limit in the range of AGIs b/c *maybe an AGI + english words are the
SI-simplest way to write down some programs*. We can't know what the crossover point is for SI-simple programs to go
from non-AGI to AGI... unless we have a theory of AGI. So, whatever understanding of AGI we get, *we should not do it
**incrementally*** if that means making lots of incremental steps through a program-space "full" of AGIs.
** ALL PROGRAMS MORE COMPLEX THAN THE MOST BASIC AGIs ARE OFF LIMITS TO SI **
By the above logic I think an important conclusion is that one of the two following ideas has to be true. Option 1: AGI
researchers (as humans) are/will be mass-murdering AGIs one day - if humans work like SI does. **Or**, option 2:
there's some other way to learn stuff / create knowledge - that's what the AGI researchers are doing - **and** it must
be SI-simpler than an SI-AGI would be. If option 2 is correct, but it is physically possible to make meaningful
progress towards AGI via SI, then we're basically guaranteed to murder lots of AGIs if ppl pursue that avenue long
enough (and with enough compute resources).
I don't think AGI research *requires* the murder of AGIs.
--
Max
xk.io
I post my FI work/articles/exercise/practice here:
https://xertrov.github.io/fi
Elliot Temple
Sep 9, 2020, 3:03:07 PM9/9/20
to FIGG
On Sep 9, 2020, at 3:03 AM, Max Kaye <m...@xk.io> wrote:
> Looking for feedback; thinking of posting to LW. I'd probably change some bits so I didn't sound as much like an
> outsider.
You’re hard wrapping lines at ~115 characters. That's inconvenient to read.
> Looking for feedback; thinking of posting to LW. I'd probably change some bits so I didn't sound as much like an
> outsider.
I’ve mentioned some issues previously like their lack of discussion of in what circumstances a hypothesis will pass another hypotheses that hasn’t gone to 0 probability.
Another issue is consider not-X. How do you represent that in SI? Instantiate an entire SI system in an identical state except with the X theory removed. (And if it’s using normalized probabilities then you have to renormalize.)
I haven’t seen any discussion of embedding SI systems within SI hypotheses. And when I brought up using logic operators with hypotheses I was basically just told you can’t do that. I don’t think most of them are very interested in thinking about SI. Maybe there are a few exceptions somewhere who think about things, but idk how to find them because the literature recommendations – from the bulk of people who don’t think much – are not a useful way to find the good stuff if it exists.
Elliot Temple
www.elliottemple.com
Max Kaye
Sep 9, 2020, 11:19:25 PM9/9/20
to fallibl...@googlegroups.com
On Wed, 9 Sep 2020 12:03:04 -0700 Elliot Temple <cu...@curi.us> wrote:
>On Sep 9, 2020, at 3:03 AM, Max Kaye <m...@xk.io> wrote:
>
>> Looking for feedback; thinking of posting to LW. I'd probably change some bits so I didn't sound as much like an
>> outsider.
>
>You’re hard wrapping lines at ~115 characters. That's inconvenient to read.
Fixed now.
>On Sep 9, 2020, at 3:03 AM, Max Kaye <m...@xk.io> wrote:
>
>> Looking for feedback; thinking of posting to LW. I'd probably change some bits so I didn't sound as much like an
>> outsider.
>
>You’re hard wrapping lines at ~115 characters. That's inconvenient to read.
There's a non-hardwrapped version here: https://github.com/XertroV/fi/blob/master/docs/_posts/2020-09-09-si-murders-people-a-lot.md and here https://xertrov.github.io/fi/posts/2020-09-09-si-murders-people-a-lot/
On that note, I think a bunch of their reasoning breaks when you have a program that returns the probability of the next bit, rather than exactly 0/1. Particularly, how do you eliminate programs? You have to make a judgement on when to eliminate them (how do you do the judgement?) and there are some patterns that you need to run for a long time to know stuff about.
Example: https://www.youtube.com/watch?v=L4ArlAfKTLA titled "Patterns that appear to hold, but don't - 8424432925592889329288197322308900672459420460792433"
How would SI or AIXI deal with this? Not very well I imagine. But *people* dealt with it fairly well...
>I’ve mentioned some issues previously like their lack of discussion of in what circumstances a hypothesis will pass another hypotheses that hasn’t gone to 0 probability.
>
>Another issue is consider not-X. How do you represent that in SI? Instantiate an entire SI system in an identical state except with the X theory removed. (And if it’s using normalized probabilities then you have to renormalize.)
>I haven’t seen any discussion of embedding SI systems within SI hypotheses.
> And when I brought up using logic operators with hypotheses I was basically just told you can’t do that.
> I don’t think most of them are very interested in thinking about SI. Maybe there are a few exceptions somewhere who think about things, but idk how to find them because the literature recommendations – from the bulk of people who don’t think much – are not a useful way to find the good stuff if it exists.
Reply all
Reply to author
Forward
0 new messages