>I am NOT anthropomorphizing.
Why not? Perhaps you should do a little anthropomorphizing because if used intelligently anthropomorphizing can be a very useful tool. If something behaves like me then maybe it feels like me too, that's the only reason I believe my fellow human beings are probably just as conscious as I am.
> I am asking questions about what it is that we value so much in humanity and whether we are necessarily unique in that regard. There have long been simple answers: We are made in the image of God. Might makes right. We are the pinnacle of evolution... All bullshit.
I agree.
> I’m hoping (not asserting) that we can develop some better answers from building and studying AIs.
You're never going to get an axiomatic system of ethics that contains no contradictions and covers every eventuality, thanks to Kurt Godel we know that would be impossible even for something as uncontroversial as arithmetic.
>Years ago, many people thought that the Turning Test was a sufficient condition for moral worth. That is no longer a widely held opinion
That was believed back when most thought a computer would never be able to pass the Turing Test, but now when computers can pass that test with flying colors many have changed their mind, but I have not. However I don't think it's important if humans believe intelligent computers deserve moral consideration or not, of much more practical importance is whether computers think humans deserve moral consideration.
> even if they are true, there’s no reason to believe the biological evolution is so special that the kind of evolution that produces LLMs might not also create those qualities.
There is no way random mutation and natural selection could have produced consciousness unless consciousness was the inevitable byproduct of intelligent behavior. It must be a brute fact that consciousness is the way data feels when it is being processed intelligently.
John K Clark