Gerrit 2.15.12 - Kerberos + GSSAPI + Active Directory - possible bug in sending SPN

[Frank Burns]

Running on RHEL 7.5 with Java 8. Kerberos 5 release 1.15.1.

We are seeing a strange behaviour with this set-up that has been seen in all versions since 2.11.10.

Note, I can't post direct logs or config as it my company blocks this.

Steps to reproduce

1) Configure gerrit to use kerberos

gerrit.config

[container]

javaHome = <path to JRE>

javaOptions = -Djava.security.auth.login.config=<path to jaas.conf>

[auth]

type = LDAP

[ldap]

authentication = GSSAPI

server = ldap://<AD Realm>

<.. other AD related stuff..>

jaas.conf

KerberosLogin {
    com.sun.security.auth.module.Krb5LoginModule
            required
            useTicketCache=true
            doNotPrompt=true
            renewTGT=true;
};

which is direct from the documentation.

2) kinit the keytab to create a ticket in the cache.

3) Try to login. It fails with "Server not found in Kerberos database (7)".

It will also fail if you change the jaas.conf to try and use the keytab directly.

You can access LDAP directly using the username/password but due to Company restrictions we can't have an unencrypted password at rest on a device so this is not a viable long, term solution.

We have taken packet captures of the traffic to the AD Realm and we see the same behaviour whether we use the keytab or the cache.

1) For the kinit we see one request to AD with the SPN field set to the SPN from the keytab. This, of course, works fine.

2) For any request from Gerrit we see TWO requests to AD, the first has the correct SPN from the cache/keytab the second tries to send an SPN of "ldap/<AD Realm>" no matter what value of SPN is set. This second request is what is causing the error as that SPN is not recognised b AD.. Note, we have tried keytabs with various SPN's (HTTP/device, host/device, HTTP/device@<REALM> etc etc). The same thing happens every time.

This may well be something very simple is wrong in our config but we have been banging our heads on this for weeks now.

[Gert van Dijk]

On Friday, 29 March 2019 09:02:30 UTC+1, Frank Burns wrote:

Running on RHEL 7.5 with Java 8. Kerberos 5 release 1.15.1.

• What is the AD server type that you're using? E.g. actual MS WIndows Active Directory version ... or Samba AD, or RedHat's Directory Server, or FreeIPA, or ...
• How 'big' is your AD set up? E.g. a forest with several servers, using regular DNS records or using SRV records to indicate which replica should be contacted, etc. (krb5 docs: "Hostnames for KDCs")
  For complex setups and finding the right LDAP server to talk to based on the realm, fixes are present in 2.16.x with StartTLS support via Change 106141.
  
• Also, what is the JRE that you're using? I've seen several errors being fixed on the JRE 1.8 updates related to Kerberos over the last few years.
  

 

[container]

javaHome = <path to JRE>

javaOptions = -Djava.security.auth.login.config=<path to jaas.conf>

How are you starting Gerrit? Note that this section is only taken into account when starting with the bin/gerrit.sh script, but ignored if you start Gerrit directly with Java (as I always prefer to do in containerized environments).

 

[auth]

type = LDAP

[ldap]

authentication = GSSAPI

server = ldap://<AD Realm>

<.. other AD related stuff..>

jaas.conf

KerberosLogin {
    com.sun.security.auth.module.Krb5LoginModule
            required
            useTicketCache=true
            doNotPrompt=true
            renewTGT=true;
};

which is direct from the documentation.

2) kinit the keytab to create a ticket in the cache.

3) Try to login. It fails with "Server not found in Kerberos database (7)".

Kerberos library errors are very very misleading sometimes.

Did you also, as per documentation (Gerrit + JRE docs) follow the following points?

• "system property java.security.krb5.conf must point to the appropriate krb5.ini"
• On Windows servers the registry key HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters must have the DWORD value allowtgtsessionkey set to 1
• the account must not have local administrator privileges.
  

 

It will also fail if you change the jaas.conf to try and use the keytab directly.

Hmm. Could you double check the key types and principals with running "klist -e"? I believe it is easy to still obtain encryption types which are not accepted any longer on modern AD servers.

Also, for some reason I had to use explicitly enable AES keytype on Windows AD and Samba AD or else you'll get ancient encryption types. After that I could properly restrict "/etc/krb5.conf" settings on permitted_enctypes, default_tgs_enctypes, default_tkt_enctypes.

You can access LDAP directly using the username/password but due to Company restrictions we can't have an unencrypted password at rest on a device so this is not a viable long, term solution.

I'm not here to judge your company policy regarding that. I think key based authentication is cool and better than a simple password, but in practice here this is very much the same? It's not like a scoped API key or something, the ticket cache with TGT gives full access to the account, just like a password does.

 

We have taken packet captures of the traffic to the AD Realm and we see the same behaviour whether we use the keytab or the cache.

1) For the kinit we see one request to AD with the SPN field set to the SPN from the keytab. This, of course, works fine.

2) For any request from Gerrit we see TWO requests to AD, the first has the correct SPN from the cache/keytab the second tries to send an SPN of "ldap/<AD Realm>" no matter what value of SPN is set. This second request is what is causing the error as that SPN is not recognised b AD.. Note, we have tried keytabs with various SPN's (HTTP/device, host/device, HTTP/device@<REALM> etc etc). The same thing happens every time.

Have you tried logging in with Kerberos an another JRE-based application? I really like Apache Directory Studio, but it seems to have bugs in Kerberos authentication to my Samba AD for LDAP, which really sounds very similar to what you're seeing. So it might be something worth investigating if this is a common issue for JRE based applications.

Or, going back one more step, can you verify everything is okay on the Kerberos+LDAP level by using the plain LDAP utilities? E.g. command line with "ldapsearch -Y GSSAPI [...]". If that's not working already, then you have a problem unrelated to Gerrit. If it is, you may want to try Apache Directory Studio on the same OS/JRE and see if it's JRE common, or Gerrit specific.

 

This may well be something very simple is wrong in our config but we have been banging our heads on this for weeks now.

Yeah, Kerberos can be very hard to set up, unfortunately, with mostly vague errors as the cause for it. But when it works, I really like the integration (mostly for users, the SSO part which is truly multi-protocol, e.g. Gerrit SSH without having to set up SSH keys!).

I've had loads of issues with DNS too, especially some application requiring fully symmetrical DNS, but I saw I could save valuable time going down that route by setting "rdns = false" & "realm_try_domains = false" for example... I can imagine that the JRE won't pick up all the krb5.conf settings with its own implementation, so that it may behave differently from native applications.

HTH

[Frank Burns]

1) Huge corporate AD with thousands of users. Fully Windows.

2) It's JRE 1.8

3) Gerrit is always started with ../bin/gerrit.sh

4) Gerrit can use LDAP fine just using the user name and a password in secure.config.

5) I added "ldap/<REALM>" as an SPN on the account in AD and I now get LDAP error 49 AcceptSecurityContextError 52e which implies the key exchange is working its now having problems negotiating AD with the same set up that works if I log into AD directly.

6) klist -e shows only AES being used (As I expected).

7) ldapsearch isn't installed and to get it installed would take 4 weeks and a 4 figure sum to the worst outsource supplier around. I don't think plain ldap is the issue as Gerrit can use plain LDAP fine (see 4).

Point 5 above seems to strongly suggest how Gerritt is addressing GSSAPI is the issue.