Reverse name DNS lookup for Kerberos SPN building problem

[Michael -O]

Hi devs,

I have a case where serf builds an incorrect SPN with GSS-API (Heimdal) for proxy authentication. The usecase is like this:

Proxy hostname: proxyfarm.company.net
serf_get says: Get principal HTTP/<hostname after reverse and forward dns resolve>

This fails completely because of the DNS lookup the server is not known to the KDC (AD). That specific SPN is not bound to a machine account but to a service account with is used by the proxy server software on various machines to unify all requests.

I'd like to narrow down wether serf is performing the DNS lookup or some else is interfering. I have conveyed the very same test with curl on Windows and FreeBSD.

FreeBSD (Heimdal): Same result as with serf
Windows (SSPI): SPN is built as-is and works as expected.

Any thoughs?

Michael

[Lieven Govaerts]

Hi,

On Fri, Feb 7, 2014 at 3:17 PM, Michael -O <1983-...@gmx.net> wrote:
> Hi devs,
>
> I have a case where serf builds an incorrect SPN with GSS-API (Heimdal) for
> proxy authentication. The usecase is like this:
>
> Proxy hostname: proxyfarm.company.net
> serf_get says: Get principal HTTP/<hostname after reverse and forward dns
> resolve>
>
> This fails completely because of the DNS lookup the server is not known to
> the KDC (AD). That specific SPN is not bound to a machine account but to a
> service account with is used by the proxy server software on various
> machines to unify all requests.
>
> I'd like to narrow down wether serf is performing the DNS lookup or some
> else is interfering. I have conveyed the very same test with curl on Windows
> and FreeBSD.
>

this looks like issue #125:
https://code.google.com/p/serf/issues/detail?id=125

The reverse lookup is done in auth_spnego.c with the call to apr_getnameinfo

Lieven

> FreeBSD (Heimdal): Same result as with serf
> Windows (SSPI): SPN is built as-is and works as expected.
>
> Any thoughs?
>
> Michael
>

> --
> You received this message because you are subscribed to the Google Groups
> "Serf Development List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to serf-dev+u...@googlegroups.com.
> To post to this group, send email to serf...@googlegroups.com.
> Visit this group at http://groups.google.com/group/serf-dev.
> For more options, visit https://groups.google.com/groups/opt_out.

[Michael-O]

> Hi,
>
> On Fri, Feb 7, 2014 at 3:17 PM, Michael -O <1983-...@gmx.net> wrote:
> > Hi devs,
> >
> > I have a case where serf builds an incorrect SPN with GSS-API (Heimdal) for
> > proxy authentication. The usecase is like this:
> >
> > Proxy hostname: proxyfarm.company.net
> > serf_get says: Get principal HTTP/<hostname after reverse and forward dns
> > resolve>
> >
> > This fails completely because of the DNS lookup the server is not known to
> > the KDC (AD). That specific SPN is not bound to a machine account but to a
> > service account with is used by the proxy server software on various
> > machines to unify all requests.
> >
> > I'd like to narrow down wether serf is performing the DNS lookup or some
> > else is interfering. I have conveyed the very same test with curl on Windows
> > and FreeBSD.
> >
>
> this looks like issue #125:
> https://code.google.com/p/serf/issues/detail?id=125
>
> The reverse lookup is done in auth_spnego.c with the call to apr_getnameinfo

Any how I can help to resolve the issue?

Michael

[Ivan Zhakov]

On 7 February 2014 18:33, Lieven Govaerts <l...@apache.org> wrote:
> Hi,
>
> On Fri, Feb 7, 2014 at 3:17 PM, Michael -O <1983-...@gmx.net> wrote:
>> Hi devs,
>>
>> I have a case where serf builds an incorrect SPN with GSS-API (Heimdal) for
>> proxy authentication. The usecase is like this:
>>
>> Proxy hostname: proxyfarm.company.net
>> serf_get says: Get principal HTTP/<hostname after reverse and forward dns
>> resolve>
>>
>> This fails completely because of the DNS lookup the server is not known to
>> the KDC (AD). That specific SPN is not bound to a machine account but to a
>> service account with is used by the proxy server software on various
>> machines to unify all requests.
>>
>> I'd like to narrow down wether serf is performing the DNS lookup or some
>> else is interfering. I have conveyed the very same test with curl on Windows
>> and FreeBSD.
>>
>
> this looks like issue #125:
> https://code.google.com/p/serf/issues/detail?id=125
>
> The reverse lookup is done in auth_spnego.c with the call to apr_getnameinfo
>

Serf SSPI based SPNEGO implementation on Windows doesn't use reverse
lookup to build SPN on Windows. It uses canonical name of target for
SPN.


--
Ivan Zhakov
CTO | VisualSVN | http://www.visualsvn.com

[Michael-O]

How is that different? Is uses getaddrinfo in contrast ot the apr function.

Michael

[Lieven Govaerts]

Hi Michael,


Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.

What it does is use the hostname as provided by the application,
instead of doing a reverse lookup from the address structure the
application provided.

Can you test it?

Lieven

On Fri, Feb 7, 2014 at 3:17 PM, Michael -O <1983-...@gmx.net> wrote:

[Michael-O]

Am 2014-02-08 01:07, schrieb Lieven Govaerts:
> Hi Michael,
>
>
> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>
> What it does is use the hostname as provided by the application,
> instead of doing a reverse lookup from the address structure the
> application provided.
>
> Can you test it?

Hi Lieven,

thanks for the patch. I will apply and test on Monday and tell you the
results on Tuesday.

Michael

[Ivan Zhakov]

Hi Lieven,

On 8 February 2014 04:07, Lieven Govaerts <l...@apache.org> wrote:
> Hi Michael,
>
>
> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>
> What it does is use the hostname as provided by the application,
> instead of doing a reverse lookup from the address structure the
> application provided.

I believe this workaround should be moved to platform specific code to
keep current Windows behavior unaffected. Because SSPI based
implementation already does this lookup internally.

--
Ivan Zhakov

[Michael-O]

By platform specific you are referring to GSS-API based?

Are you able to provide me a 64 bit binary of serf_get? I'd like to test
SSPI code against various services at work.

Thanks

[Michael-O]

Lieven,

Am 2014-02-08 01:07, schrieb Lieven Govaerts:

> Hi Michael,
>
>
> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>
> What it does is use the hostname as provided by the application,
> instead of doing a reverse lookup from the address structure the
> application provided.
>
> Can you test it?

I have tried your patch and it works great with Heimdal. Reverse DNS
lookup should be left over to GSS-API itself and not serf or apr. In
order to make that work, I have added rnds = false into my krb5.conf. A
correct ticket has been obtained, just as with SSPI and curl. Regarding
the rdns value, there was already a discussion with MIT Kerberos [1].

Please apply this patch to serf trunk and branch 1.3.x.

I'd like to repeat my testing with MIT Kerberos on RHEL 6 but scons
requires Python 2.7 which I do not have in the Red Hat repo. So manual
building might take a while.

During testing I have found another bug in serf, I guess. I am
requesting a URL from a host which advertises Negotiate and I can see
that Heimdal tries multiple times to issue a TGS-REQ but the response is
always the same: PRINCIPAL_UNKNOWN. (Yes there is not SPN registered for
that machine). The weird stuff is that serf still has a ticket which is
sent over to the host. This cannot be correct. Are you interested to see
Wireshark output and logs?

Michael

[1]
https://groups.google.com/forum/#!topic/comp.protocols.kerberos/JgUUDTxkr5U

[Ivan Zhakov]

On 10 February 2014 23:07, Michael-O <1983-...@gmx.net> wrote:
> Am 2014-02-10 10:43, schrieb Ivan Zhakov:
>
>> Hi Lieven,
>>
>> On 8 February 2014 04:07, Lieven Govaerts <l...@apache.org> wrote:
>>>
>>> Hi Michael,
>>>
>>>
>>> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>>>
>>> What it does is use the hostname as provided by the application,
>>> instead of doing a reverse lookup from the address structure the
>>> application provided.
>>
>> I believe this workaround should be moved to platform specific code to
>> keep current Windows behavior unaffected. Because SSPI based
>> implementation already does this lookup internally.
>
>
> By platform specific you are referring to GSS-API based?
>

Serf SPNego authentication has platform independent code
(auth_spnego.c) and platform dependent code: auth_spnego_sspi.c for
Windows (SSPI) and auth_spnego_gss.c for GSS-API based code. So I
suggest to implement workaround in auth_spneg_gss.c since Windows code
is working fine already.

> Are you able to provide me a 64 bit binary of serf_get? I'd like to test
> SSPI code against various services at work.
>

Sure, I can send you 64-bit binary of serf_get. Do you need patched
version or just serf 1.3.4?

--
Ivan Zhakov

[Lieven Govaerts]

Hey Ivan.

On Mon, Feb 10, 2014 at 10:43 AM, Ivan Zhakov <iv...@visualsvn.com> wrote:
> Hi Lieven,
>
> On 8 February 2014 04:07, Lieven Govaerts <l...@apache.org> wrote:
>> Hi Michael,
>>
>>
>> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>>
>> What it does is use the hostname as provided by the application,
>> instead of doing a reverse lookup from the address structure the
>> application provided.

> I believe this workaround ...

This is not a workaround, but removal of an unneeded workaround.

> ... should be moved to platform specific code to

> keep current Windows behavior unaffected. Because SSPI based
> implementation already does this lookup internally.
>

It won'tl impact the SSPI implementation negatively, on the contrary,
it'll reduce one lookup when working with a proxy:

What we were doing now before this patch for Negotiate authn over a proxy:
1. application: hostname -> ip address
2. auth_spnego: ip address -> canonical hostname
3. auth_spnego_sspi: canonical hostname -> canonical hostname
4. auth_spnego_sspi: canonical hostname passed to InitializeSecurityContextA

The patch eliminates the second step, which means that the flow for
SPNego with SSPI now becomes:
1. application: hostname -> ip address
2. auth_spnego_sspi: hostname -> canonical hostname
3. auth_spnego_sspi: canonical hostname passed to InitializeSecurityContextA

And for the MIT Kerberos/Heimdal implementation:
1. application: hostname -> ip address
2. auth_spnego_gss: hostname -> gss_init_sec_context

Letting the GSSAPI mechanism do the canonicalisation gives me the
option to bypass the getaddrinfo call in MIT Kerberos via the "rdns =
false" option in the krb5.conf file.

Makes me wonder why you added get_canonical_hostname in the first
place, doesn't SSPI/Negotiation or SSPI/NTLM canonicalise the hostname
already?

regards,

Lieven

[Michael-O]

Am 2014-02-10 20:44, schrieb Ivan Zhakov:
> On 10 February 2014 23:07, Michael-O <1983-...@gmx.net> wrote:
>> Am 2014-02-10 10:43, schrieb Ivan Zhakov:
>>
>>> Hi Lieven,
>>>
>>> On 8 February 2014 04:07, Lieven Govaerts <l...@apache.org> wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>>
>>>> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>>>>
>>>> What it does is use the hostname as provided by the application,
>>>> instead of doing a reverse lookup from the address structure the
>>>> application provided.
>>>
>>> I believe this workaround should be moved to platform specific code to
>>> keep current Windows behavior unaffected. Because SSPI based
>>> implementation already does this lookup internally.
>>
>>
>> By platform specific you are referring to GSS-API based?
>>
> Serf SPNego authentication has platform independent code
> (auth_spnego.c) and platform dependent code: auth_spnego_sspi.c for
> Windows (SSPI) and auth_spnego_gss.c for GSS-API based code. So I
> suggest to implement workaround in auth_spneg_gss.c since Windows code
> is working fine already.

Well,

why is apr_getnameinfo performed on the proxy host at all and not for
both hosts? Shouldn't the patch apply for both and let SSPI and GSS-API
peform canonicalization at their will?

>> Are you able to provide me a 64 bit binary of serf_get? I'd like to test
>> SSPI code against various services at work.
>>
> Sure, I can send you 64-bit binary of serf_get. Do you need patched
> version or just serf 1.3.4?

I would like to try both.

Thanks,

Michael

[Ivan Zhakov]

Oops, sorry. I missed that patch actually removes DNS lookup. Sorry.

> Letting the GSSAPI mechanism do the canonicalisation gives me the
> option to bypass the getaddrinfo call in MIT Kerberos via the "rdns =
> false" option in the krb5.conf file.
>
> Makes me wonder why you added get_canonical_hostname in the first
> place, doesn't SSPI/Negotiation or SSPI/NTLM canonicalise the hostname
> already?
>

SSPI doesn't resolve provided SPN and uses it as is. That's why we
need to resolve it on our side.


--
Ivan Zhakov

[Michael-O]

Am 2014-02-11 13:13, schrieb Ivan Zhakov:
> [..]

>> Makes me wonder why you added get_canonical_hostname in the first
>> place, doesn't SSPI/Negotiation or SSPI/NTLM canonicalise the hostname
>> already?
>>
> SSPI doesn't resolve provided SPN and uses it as is. That's why we
> need to resolve it on our side.

And why is that exactly? If the instructure uses functional accounts
with appropriate SPNs that wouldn't work if you do canonicalization.

Still waiting for the binaries to verify that.

Michael

[Michael-O]

Am 2014-02-08 01:07, schrieb Lieven Govaerts:

> Hi Michael,
>
>
> Attached patch fixes the problem for me, on Mac OS X using MIT kerberos.
>
> What it does is use the hostname as provided by the application,
> instead of doing a reverse lookup from the address structure the
> application provided.
>
> Can you test it?

Just made a quick test on RHEL6. Serf_get crashes with a illegal pointer
reference.

I will perform several tests and would like to provide your all
necessary data privately.

Did you had a chance to read the possible bug description in my prior mail?

Michael

[Lieven Govaerts]

yes, you can send them privately.

Thx,

Lieven

 

Michael

[1] https://groups.google.com/forum/#!topic/comp.protocols.kerberos/JgUUDTxkr5U

[Michael-O]

Am 2014-02-11 13:13, schrieb Ivan Zhakov:

> [...]

> SSPI doesn't resolve provided SPN and uses it as is. That's why we
> need to resolve it on our side.

Having checked both versions against our special configuration, there
was no difference with SSPI.

Is there any Microsoft reference that you have to resolve the canonical
host yourself before passing to SSPI?

Michael