Total confusing "Permission denied" message

[samwyse]

I need to set up a service account that will be used by some DBAs
after my contract is done. I built and tested everything in my
personal account and now I'm moving everything to the service
account. Unfortunately, commands that were working perfectly are now
failing. I've even tried copying all of my .ssh files to the service
account, and that also fails. I've run both client commands with
three '-v' options and included the 'diff' below. Can anyone thing of
anything I'm missing, or even a good "next step"? Thanks.

example|global$ uname -a
SunOS example 5.10 Generic_127111-06 sun4v sparc SUNW,Sun-Fire-T200

example|global$ ssh -l samwyse -o PreferredAuthentications=publickey
localhost echo hi
hi

example|global$ ssh -l symcli -o PreferredAuthentications=publickey
localhost echo hi
Permission denied (gssapi-keyex,gssapi-with-
mic,publickey,password,keyboard-interactive).

example|global$ ls -al ~symcli/.ssh
total 292
drwxr-xr-x 2 symcli unix 512 Oct 30 20:39 .
drwxrwxr-x 5 symcli root 512 Nov 2 20:36 ..
-rw-r--r-- 1 symcli unix 61060 Aug 21 20:03 authorized_keys
-rw------- 1 symcli unix 668 May 27 18:59 id_dsa
-rw-r--r-- 1 symcli unix 608 May 27 18:59 id_dsa.pub
-rw------- 1 symcli unix 883 May 27 18:59 id_rsa
-rw-r--r-- 1 symcli unix 228 May 27 18:59 id_rsa.pub
-rw-r--r-- 1 symcli unix 71926 Sep 23 20:37 known_hosts

example|global$ sudo sum ~samwyse/.ssh/*
44068 120 /export/home/samwyse/.ssh/authorized_keys
54835 2 /export/home/samwyse/.ssh/id_dsa
50690 2 /export/home/samwyse/.ssh/id_dsa.pub
6518 2 /export/home/samwyse/.ssh/id_rsa
19323 1 /export/home/samwyse/.ssh/id_rsa.pub
63777 141 /export/home/samwyse/.ssh/known_hosts

example|global$ sudo sum ~symcli/.ssh/*
44068 120 /opt/emc/.ssh/authorized_keys
54835 2 /opt/emc/.ssh/id_dsa
50690 2 /opt/emc/.ssh/id_dsa.pub
6518 2 /opt/emc/.ssh/id_rsa
19323 1 /opt/emc/.ssh/id_rsa.pub
45485 141 /opt/emc/.ssh/known_hosts

example|global$ diff samwyse.trace symcli.trace
102,103c102,103
< debug1: dh_gen_key: priv key bits set: 136/256
< debug1: bits set: 1638/3191
---
> debug1: dh_gen_key: priv key bits set: 125/256
> debug1: bits set: 1599/3191
110c110
< debug1: bits set: 1602/3191
---
> debug1: bits set: 1560/3191
135,192c135,143
< debug1: Server accepts key: pkalg ssh-rsa blen 149 lastkey 6cd40
hint 1
[...]
---
> debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
> debug1: Trying public key: /export/home/samwyse/.ssh/id_dsa
> debug3: send_pubkey_test
> debug2: we sent a publickey packet, wait for reply
> debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
> debug2: we did not send a packet, disable method
> debug1: No more authentication methods to try.
> Permission denied (gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive).
> debug1: Calling cleanup 0x3429c(0x0)

[samwyse]

I figured out how to run traces on the server side, whereupon I found
a directory permissions problem. I fixed it by moving to a new home
directory, but things are still not working. Here's the interesting
bit from running sshd with an alternate port and three '-d' options:

< debug2: userauth_pubkey: authenticated 0 pkalg ssh-rsa
< Failed publickey for symcli from 127.0.0.1 port 62697 ssh2
---
> debug2: userauth_pubkey: authenticated 1 pkalg ssh-rsa
> Accepted publickey for samwyse from 127.0.0.1 port 62712 ssh2

[samwyse]

I solved the problem! None of the debug traces were showing it, but
PAM was vetoing the session because the account had gotten locked
somehow.

example|global> grep -n symcli /etc/passwd /etc/shadow
/etc/passwd:50:symcli:x:21044:4100:Solutions Enabler service account:/
opt/emc:/bin/ksh
/etc/shadow:50:symcli:*LK*XF1QAndz2L5Cs:14550::::::10
example|global> passwd symcli
New Password:
Re-enter new Password:
passwd: password successfully changed for symcli
example|global> grep -n symcli /etc/passwd /etc/shadow
/etc/passwd:50:symcli:x:21044:4100:Solutions Enabler service account:/
opt/emc:/bin/ksh
/etc/shadow:50:symcli:aDQu8BFQwWjTM:14551::::::
example|global> cd ~symcli
example|global> ./test-both-ends symcli
Success!

I finally found the solution here:
http://www.semicomplete.com/blog/geekery/solaris-10-sshd-publickey-solution.html#1169550893.03

BTW, here's the script I used to debug everything:

example|global> cat test-both-ends
#!/bin/ksh -u
# Produces maximal debugging output for both ends of an SSH
connection.
PORT=4407 # Use a port no one else is using.
for USER in "${@:?missing user name(s)}"
do
/usr/lib/ssh/sshd -d -d -d -p $PORT 2>sshd_$USER.txt &
/bin/ssh -p $PORT -v -v -v -i ~$USER/.ssh/id_rsa -l $USER -o
PreferredAuthentications=publickey localhost echo "Success!" 2>ssh_
$USER.txt
((PORT+=1)) # Increment to avoid timeout issues.
done

On Nov 2, 4:59 pm, samwyse <samw...@gmail.com> wrote:
> I figured out how to run traces on the server side, whereupon I found
> a directory permissions problem.  I fixed it by moving to a new home
> directory, but things are still not working.
>