ssh daemon asks for a password on CentOS 7.1

[dilber...@gmail.com]

ssh daemon asks for a password with OpenSSH_6.9p1

I have the following after a brand new gitolite install

git clone g...@10.24.142.183:gitolite-admin
Cloning into 'gitolite-admin'...
g...@10.24.142.183's password:

or

ssh g...@10.24.142.183 info
g...@10.24.142.183's password:

A bit of history!

I already have 2 successful install of gitolite over the last 5 years
one install with 120 repos on a redhat Red Hat Enterprise Linux Server release 4 (Tikanga) Linux 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
and another one with 20 repos on a ubuntu server 12.04 LTS

NO problem with those servers

However when I tried to migrate the redhat, as the server will be decommissioned soon, I'm running in the above problem.
I failed the migration for that reason using same gitolite version (gitolite3 v3.6-2-g2d5c690 on git 1.8.2.1) and same pubkey.

I then tried to install a brand new gitolite version from latest gitolite github as of 2015-12-11
with old same pubkey and another install with a brand new pubkey made on the new server itself.
I followed http://gitolite.com/gitolite/gitolite.html#qi and http://www.bigfastblog.com/gitolite-installation-step-by-step
No success so far.

After reading carefully and following the easy installation procedure, I'm now convinced the problem is not with gitolite but ssh(d) config only.

the new server details is
cat /etc/centos-release
CentOS Linux release 7.1.1503 (Core)
uname -a
Linux 3.10.0-229.20.1.el7.x86_64 #1 SMP Tue Nov 3 19:10:07 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
git --version
git version 1.8.3.1


I'm not a Secure Shell guru and I fail to spot the problem with the configuration.
I must add I check line by line the /etc/ssh/sshd_config on the redhat and centos and they have the same config

I join the centos sshd_config and the bash console log from 'ssh -vvv git@server info' if it can help anyone to give some pointers on what is wrong.


When I compare the 2 ssh logs with the correct pubkey between the successful gitolite and the faulty gitolite
I see the following
good
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Remote: Forced command: /serial/releng/gitolite/src/gitolite-shell -s <username>
debug1: Remote: Port forwarding disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Agent forwarding disabled.
debug1: Server accepts key: pkalg ssh-rsa blen 279
faulty
and line 131 in g...@10.24.142.183info.log
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

I checked that in server git .ssh/authorized_keys starting with command="/serial/gitolite/bin/gitolite-shell johndoe",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa ...
and the client pubkey is the same

I read and reread http://gitolite.com/gitolite/gitolite.html#stsapp1 and with a good server working, I'm running out of ideas to find out what's wrong!

Any help is appreciated!

[Sitaram Chamarty]

Have you looked at /var/log/secure (or maybe even /var/log/messages).
When pubkey access is ignored, it will usually tell you.

[Konstantin Gribov]

If you have selinux enabled (which is default for centos/rhel), you can have issues with incorrect selinux context on .ssh directory.

First, check if selinux is enabled with `sestatus`. If it is, do `restorecon -Rv ~/.ssh` for appropriate user. Also, check that .ssh has 0700 perms.

пт, 11 дек. 2015 г. в 14:47, Sitaram Chamarty <sita...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "gitolite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gitolite+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards,

Konstantin Gribov

[Tom Harrison]

Try connecting via using ssh, e.g. "ssh g...@git.example.com" -- you won't get logged in, but you should be able to see your repos -- if this prompts for pw, it's probably just an ssh configuration thing -- ssh is (properly) picky about permissions of the .ssh directory and other sensitive files.  You can pass -v (or -vv or -vvv) to get increasingly verbose, albeit obtuse information about what's happening.

In this case despite using the "git@<hostname>" you're really using your own ssh credentials

A working connection with a single -v should have output something like this (everything after # are my comments)

debug1: Reading configuration data /etc/ssh_config

debug1: /etc/ssh_config line 20: Applying options for *

debug1: Connecting to git.example.com [192.168.x.x] port 22.

debug1: Connection established.      # good!

debug1: identity file /Users/tom/.ssh/id_rsa type 1

debug1: identity file /Users/tom/.ssh/id_rsa-cert type -1

debug1: identity file /Users/tom/.ssh/id_dsa type -1

debug1: identity file /Users/tom/.ssh/id_dsa-cert type -1

debug1: Enabling compatibility mode for protocol 2.0  # ssh worked out which method and type of ssh I want to use

debug1: Local version string SSH-2.0-OpenSSH_6.2

debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2

debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH*

debug1: SSH2_MSG_KEXINIT sent

debug1: SSH2_MSG_KEXINIT received

debug1: kex: server->client aes128-ctr hmac-m...@openssh.com none

debug1: kex: client->server aes128-ctr hmac-m...@openssh.com none

debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP

debug1: SSH2_MSG_KEX_DH_GEX_INIT sent

debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY

debug1: Server host key: RSA 61:8f:4e:de:f3:f8:b5:22:fb:e2:31:39:f7:d2:7d:a3  # server has responded with a signtaure

debug1: Host 'git.example.com' is known and matches the RSA host key.  

debug1: Found key in /Users/tom/.ssh/known_hosts:1  # server is matched to known_hosts locally

debug1: ssh_rsa_verify: signature correct

debug1: SSH2_MSG_NEWKEYS sent

debug1: expecting SSH2_MSG_NEWKEYS

debug1: SSH2_MSG_NEWKEYS received

debug1: Roaming not allowed by server

debug1: SSH2_MSG_SERVICE_REQUEST sent

debug1: SSH2_MSG_SERVICE_ACCEPT received

debug1: Authentications that can continue: publickey,password  

debug1: Next authentication method: publickey  # well, actually the first authentication method

debug1: Offering RSA public key: /Users/tom/.ssh/id_rsa  # does the public key on gitolite server fit with my local private key?

debug1: Server accepts key: pkalg ssh-rsa blen 279

debug1: read PEM private key done: type RSA

debug1: Authentication succeeded (publickey).  # yes!  If you then get "Next authentication method: password" it means public key failed

Authenticated to git.example.com ([192.168.x.x]:22).

[dilber...@gmail.com]

Long time no posting on google groups, and for some reason could't post attachment from office at initial post, and got no hint the post was actually posted.
Omissions fixed, here joined files I allude in the post
I did a meld between the 2
good connection
g...@localhostinfo.log
failing connection
g...@10.24.142.183info.log

same config except gitolite versions, and I cannot spot what's wrong

@Sitaram
I did have a look to /var/log/secure and from memory I got one line log which does not say anything what's wrong. On Monday I will copy the log if it can be any help.
I could try to increase sshd verbosity it its possible.
I have to add also, I did the setup on two physical box with Centos 7.1. thinking the first box was dirty with too many tests and the knowledge building. I send now the logs of the 2nd box which behave similarly after following the idiot setup.

@Konstantin
If I remember properly our sysadmin didn't set selinux on this box, but on Monday I will check to be sure.
What I can add is the gitolite home dir is not in /home or /var/lib but in a different direcory which has a daily backup. I wonder if that has any influences on the openssh inner working.
I will try on Monday a setup in /home , for a new testing avenue

@Tom
I did check many times .ssh dir rights user access (comparing with 2 box good one and faulty one with `ls -lha`).
I join the logs made with -vvv switch
I did compare a working log and the faulty one but failed to spot something.
I don't claim guru status on ssh topic.

[Sitaram Chamarty]

On 12/12/15 10:11, dilber...@gmail.com wrote:
> Long time no posting on google groups, and for some reason could't post attachment from office at initial post, and got no hint the post was actually posted.
> Omissions fixed, here joined files I allude in the post
> I did a meld between the 2
> good connection
> g...@localhostinfo.log
> failing connection
> g...@10.24.142.183info.log
>
> same config except gitolite versions, and I cannot spot what's wrong
>
> @Sitaram
> I did have a look to /var/log/secure and from memory I got one line log which does not say anything what's wrong. On Monday I will copy the log if it can be any help.
> I could try to increase sshd verbosity it its possible.
> I have to add also, I did the setup on two physical box with Centos 7.1. thinking the first box was dirty with too many tests and the knowledge building. I send now the logs of the 2nd box which behave similarly after following the idiot setup.
>
> @Konstantin
> If I remember properly our sysadmin didn't set selinux on this box, but on Monday I will check to be sure.

If selinux is on, use the "yum install gitolite3" method -- I am not an
expert but I imagine that will work (i.e., I am pretty sure that would
have all the policy bits set up correctly).

> What I can add is the gitolite home dir is not in /home or /var/lib but in a different direcory which has a daily backup. I wonder if that has any influences on the openssh inner working.

A daily backup script, if it has the power to make all files and
directories "readable" for example, can certainly piss off sshd. Sshd
is very picky about permissions; this is mentioned in the sts link (if I
recall).

[DBS]

On 12/12/2015 11:50 AM, Sitaram Chamarty wrote:

On 12/12/15 10:11, dilber...@gmail.com wrote:

Long time no posting on google groups, and for some reason could't post attachment from office at initial post, and got no hint the post was actually posted.
Omissions fixed, here joined files I allude in the post
I did a meld between the 2 
good connection
g...@localhostinfo.log
failing connection
g...@10.24.142.183info.log

same config except gitolite versions, and I cannot spot what's wrong

@Sitaram
I did have a look to /var/log/secure and from memory I got one line log which does not say anything what's wrong. On Monday I will copy the log if it can be any help.
I could try to increase sshd verbosity it its possible.
I have to add also, I did the setup on two physical box with Centos 7.1. thinking the first box was dirty with too many tests and the knowledge building. I send now the logs of the 2nd box which behave similarly after following the idiot setup.

@Konstantin
If I remember properly our sysadmin didn't set selinux on this box, but on Monday I will check to be sure.

If selinux is on, use the "yum install gitolite3" method -- I am not an
expert but I imagine that will work (i.e., I am pretty sure that would
have all the policy bits set up correctly).

Hmm interesting, I was not aware of this OS package (I checked on my home Fedora22, and it is there), I always go the manual way with gitolite. I will try on Monday on one of the 2 Centos box the yum package. thanks for this pointer.

What I can add is the gitolite home dir is not in /home or /var/lib but in a different direcory which has a daily backup. I wonder if that has any influences on the openssh inner working.

A daily backup script, if it has the power to make all files and
directories "readable" for example, can certainly piss off sshd.  Sshd
is very picky about permissions; this is mentioned in the sts link (if I
recall).

Very aware of that pickiness, but during install backup doesn't run, only at night. and the redhat box (working gitolite one) has also gitolite set in a exotic backup dir for the last 2 years, no problems reported so far with jenkins pulling repo 24 hours a day.
I will still carry on  a new fresh setup on /home anyway.

[dilber...@gmail.com]

Follow-up and End.

On one centos box I follow Sitaram advise

I installed gitolite3 maintained package and it did installed following dependencies: perl-Digest perl-Digest-MD5 perl-Env perl-IO-Socket-Timeout perl-PerlIO-via-Timeout perl-Redis perl-Try-Tiny

I installed gitolite again as /var/lib/gitolite3 was still empty

It fails, same git prompt

I then switch to the second Centos box (the next to be, failover box)
I decided to go the manual route again but applying Konstantin advise.

sudo su -
yum install perl-Digest perl-Digest-MD5 perl-Env perl-IO-Socket-Timeout perl-PerlIO-via-Timeout perl-Redis perl-Try-Tiny
mv /__/gitolite /var/lib/gitolite
vim /etc/passwd
replace /__/gitolite by /var/lib/gitolite
chmod 700 /var/lib/gitolite/.ssh
su - git
restorecon -Rv ~/.ssh
#gitolite was already setup, I stick to Sitaram saying running setup never hurts
gitolite setup
#At this point on the client, it complains it cannot find gitolite-shell
vim .ssh/authorized_keys
replace /__/releng/gitolite/src/gitolite-shell by /var/lib/gitolite/bin/gitolite-shell
as I did previously a `gitolite/install -to /___/gitolite/bin`

It works!

Back to the first main box
yum remove gitolite3
userdel gitolite3
rm -rf /var/lib/gitolite3
useradd -m -r --shell /bin/bash -c 'gitolite3 SCMS' -l -d /var/lib/gitolite git
su - git
#following is the standard install
#but after I ran again
restorecon -Rv ~/.ssh

try again on the client
It works!
running gitolite3 v3.6.4-7-ge430ba6 on git 1.8.3.1
R W gitolite-admin
R W testing

@Konstantin
When I ran `restorecon -Rv ~/.ssh`
I got the following log on both boxes
restorecon reset /var/lib/gitolite/.ssh context unconfined_u:object_r:gitosis_var_lib_t:s0->unconfined_u:object_r:ssh_home_t:s0
restorecon reset /var/lib/gitolite/.ssh/authorized_keys context unconfined_u:object_r:gitosis_var_lib_t:s0->unconfined_u:object_r:ssh_home_t:s0
I cannot understand what that means!

My 2 cents go as such:
Following multiple failed trials, I got the whole setup screwed up on both boxes.
But I think not realizing that selinux was running by default on Centos 7.1, the conjunction of having the gitolite home dir in an exotic location (in *nix understanding) plus the gitolite program itself in also another exotic directory prevent sshd or whatever to operate normally.
By moving to /var/lib and doing a ../gitolite/install -to ~/bin (do not copy as-is, full path required ;-) ), I got gitolite back on track.
What bothers me is the maintainer package failed to install, but will not report as my gitolite3 package install came after multiple botched installs.

@Sitaram
Should we emphasis the need to check if SELinux is running (or not using non conventional directory), as from now in a post-Snowden world, screws are tighten hard, and distro maker do their best not to jeopardize their user base security (cf the unfortunate scenario as described above), assuming than my analysis gathers some support.
Knowing, there are many other ways to backup gitolite without interfering with sshd and SELinux security mechanisms, sticking to conventions won't hurt me.

Now I will go to the original plan of the gitolite migration and server relocation with an accrued knowledge!
Thanks all for your kind help.

[Konstantin Gribov]

Your issue is not related to gitoline itself, it's related to ssh (in selinux-enabled environment).

When sshd is running under targeted selinux policy it's constrained by selinux (as known service), so sshd have access only to some directories.

Selinux access checks are based on so-called context labels (like unconfined_u:object_r:ssh_home_t:s0). 

So, when sshd process tries to read authorized_keys kernel checks normal unix permissions, then selinux context label on file. Only if both kernel checks pass, sshd can read file.

If user .ssh/authorized_keys are populated with selinux-aware ssh-copy-id (quite usual case) into standard location, then .ssh has appropriate context, and so has .ssh/authorized_keys. But if you create this directory manually or in non-standard it will have default context (uncofined_t or user_home_t, var_t or gitosis_var_lib_t, depending on parent directory context), which prevents sshd from even listing .ssh. Just check with ls -Z.

Default selinux policies have predefined contexts for well-known .ssh home directories. 

To list them use semanage fcontext -l | grep '\.ssh' (semanage is util from policycoreutils-python). 

These paths include at least /home/[^/]+/.ssh, /root/.ssh, /var/lib/[^/]+/.ssh, /var/lib/gitolite/.ssh and /var/lib/gitolite3/.ssh. So restorecon works fine on /var/lib/gitolite/.ssh.

To place gitolite to some non-standard location (like /srv/git) you can exec semanage fcontext -a -t ssh_home_t '/srv/git/.ssh(/.*)?'. Then you can call restorecon -Rv /srv/git.

[1]: https://wiki.centos.org/HowTos/SELinux

пн, 14 дек. 2015 г. в 6:47, <dilber...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "gitolite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gitolite+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dilber...@gmail.com]

You are right, I did check on the decommissioned Redhat server, no SELinux is running, that's why I was allowed a free ride for setting non conventional system directories.

Thank you very much for this enlightening explanation and the time to write it down.