Another voice of AI doom -- and why I think there may be a chance for us (where us may not be something you recognize)

[Ned]

Just saw this on HackerNoon: https://goo.gl/ZMVsZa

And I had some thoughts after our conversation last week. For a couple of reasons, I don't think humans are doomed, for a sufficiently broad definition of "human."

So I'll start with: humans in their present meat-computer form are surely doomed. Or at least, most of us are. I think it's equally obvious that no "cage" designed by mere humans is going to hold a super-AGI (if for no other reason that some hubristic idiot will try to take it out "on a leash" to accomplish some purpose, and will be unable to control it).

Some competing scenarios:

0. Some ecological disaster or bio-weapon or nuclear exchange wipes out homo not-so-sapiens before we achieve an AGI. That would really be the worst case. I grew up during the cold war, so I think we really aren't likely to do this, but...

1. Somebody builds a super-AGI. Even AGI means super-AGI very soon thereafter.

a. "The" AGI is sociopathic. We have a model of this today, the modern corporation which has taken "maximize stockholder value" as the only objective function. If corporations are people, they're sociopaths, or psychopaths.

b. The AGI is autistic. It's bright, but can be taught empathy if we can figure out how. But empathy requires identification with the other, which won't last.

2. AGIs appear gradually as we learn to blend the many algorithms that are successful in various domains. "Empathy-driven" algorithms for helping victims of PTSD, abuse, and other societal ills are among the engines we include.

3. AGIs result from interfacing a human brain with algorithms and data. "We merge with the technology." There's a lot of brain research that should make this possible within a couple of decades.

4. AGIs result from transferring ("uploading") human brain wiring diagrams, then adding the interfaces of scenario 3. "We" as patterns merge with the technology. 

4b. An interesting variation is infecting my brain with nano-machines that gradually replace every neuron with a more robust electronic or photonic circuit. Now either crank up the clock rate (there are about 6 orders of magnitude available!) or do the interface, or both. This is the boat paradox -- replace very board in the boat, one by one, is it still the same boat? I already replace all the molecules in my body every couple of years, so I'm already n generations of boat-paradox into the process.

I don't think scenario 1 is realistic. Not to say that the autistic-at-best AI wouldn't be the end of us, I just don't think "THE" super-AGI is likely. There will be lots of flavors and variants. Which leads to my thesis:

This is Darwin's Universe. 

Things self-assemble, replicate with errors and selection, and monocultures are self-eliminating. That last is important: the biggest threat to any organism is its identical twin. To the degree that you and I vary, we can coexist. And descent with variation and selection causes unfilled niches to be filled. We won't have ONE AGI, we'll arrive along many disparate paths. This scenario is more like humans as well: we don't have ONE objective function, each of us has many, and they compete for our focus.

So 2. is the doomsday scenario for humans, period, but there's some chance some of the AIs will enjoy keeping us around and not care about the niches we fill.

3. and 4. spell doom for humans in their present form. You just can't keep up as a meat-only organism. Notice that you already don't: technologies invented in the past couple of decades obviously confer benefits on you as a cyborg that you don't have as a pure animal. But older technologies have the same impact, you just don't see them because they were already there when you were born. Imagine life without trains, planes, and automobiles. Or mass production.Or printing. Or written language. Or shoes. (Even your naked ancestors carried clubs). We have always merged with our technologies, generally with very negative results for some of the early adopters, but big wins for others and their descendants. (With variation.)

So my money is on scenario 2 giving way to 3 giving way to 4. As a side-bet, I expect 4 to be at least partly CRISPR-CAS9 (or it's successors)-moderated organic material where that's the best solution to the engineering problems of Life 3.0. DNA-based life has some advantages: DNA is nanotechnology, self-replication at a very fast nano-scale. It may be the cheapest quantum computer you can build.

But pure-meat humans went extinct a long time ago, and majority-meat humans will be gone within a couple of centuries, at most.