krb5 with anonymous kinit, "Cannot allocate memory"

James Croall

unread,

Oct 11, 2013, 5:49:18 PM10/11/13

to kerb...@mit.edu

Hi All,

Thanks again for the help getting anonymous kinit running! We have been running in production for over a month and things are going… well. Until today.

This week a new error occurred on the KDC side:

Oct 11 21:25:57 sso krb5kdc[10394](info): AS_REQ (4 etypes {18 17 16 23}) 10.0.1.13: NEEDED_PREAUTH: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Additional pre-authentication required
Oct 11 21:25:58 sso krb5kdc[10394](info): AS_REQ (4 etypes {18 17 16 23}) 10.0.1.13: KDC_RETURN_PADATA: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Cannot allocate memory

It is the second line that is problematic. The kinit side reports:

kinit: Generic error (see e-text) while getting initial credentials

The system is not out of memory. No system configuration changes have been made. I am at a loss. Googling around I see strange reports of this error coming and then going and I don't know what to make of it.

Any ideas?

- James

James Croall | Senior Product Manager
Coverity | 185 Berry Street | Suite 6500, Lobby 3 | San Francisco, CA 94107
Office: 415.694.5354 | Mobile: 202.246.6613 | jcr...@coverity.com<mailto:jcr...@coverity.com>
The Leader in Development Testing

James Croall

unread,

Oct 11, 2013, 5:57:33 PM10/11/13

to James Croall, kerb...@mit.edu

I should add, this error occurs when running kinit -n.

I can still kinit as a user on an already setup host and get a TGT.

>running in production for over a month and things are goingŠ well. Until

>today.
>
>This week a new error occurred on the KDC side:
>
>Oct 11 21:25:57 sso krb5kdc[10394](info): AS_REQ (4 etypes {18 17 16 23})
>10.0.1.13: NEEDED_PREAUTH: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for
>krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Additional
>pre-authentication required
>Oct 11 21:25:58 sso krb5kdc[10394](info): AS_REQ (4 etypes {18 17 16 23})
>10.0.1.13: KDC_RETURN_PADATA: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for
>krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Cannot allocate memory
>
>It is the second line that is problematic. The kinit side reports:
>
>kinit: Generic error (see e-text) while getting initial credentials
>
>The system is not out of memory. No system configuration changes have
>been made. I am at a loss. Googling around I see strange reports of this
>error coming and then going and I don't know what to make of it.
>
>Any ideas?
>
>- James
>
>James Croall | Senior Product Manager
>Coverity | 185 Berry Street | Suite 6500, Lobby 3 | San Francisco, CA
>94107
>Office: 415.694.5354 | Mobile: 202.246.6613 |
>jcr...@coverity.com<mailto:jcr...@coverity.com>
>The Leader in Development Testing

>________________________________________________
>Kerberos mailing list Kerb...@mit.edu
>https://mailman.mit.edu/mailman/listinfo/kerberos
>

James Croall

unread,

Oct 11, 2013, 9:38:16 PM10/11/13

to kerb...@mit.edu

Poking around with strace, and running krb5kdc with debug enabled, I see
no smoking gun that there is a lack memory problem.

Searching the kerberos mailing list and other forums I see similar
reports, but no explanation of cause or possible solutions. A bit lost
here. It was working great for a month.

Here's what happens when I run kinit -n:

Oct 12 01:35:39 sso krb5kdc[1786](debug): checking padata
Oct 12 01:35:39 sso krb5kdc[1786](debug): .. pa_type 0x95
Oct 12 01:35:39 sso krb5kdc[1786](debug): client needs preauth, no hw
preauth; request has no preauth, no hw preauth
Oct 12 01:35:39 sso krb5kdc[1786](info): AS_REQ (4 etypes {18 17 16 23})
10.0.0.252: NEEDED_PREAUTH: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for
krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Additional
pre-authentication required
Oct 12 01:35:39 sso krb5kdc[1786](debug): checking padata
Oct 12 01:35:39 sso krb5kdc[1786](debug): .. pa_type 0x85
Oct 12 01:35:39 sso krb5kdc[1786](debug): .. pa_type 0x10
Oct 12 01:35:39 sso krb5kdc[1786](debug): .. pa_type pkinit
Oct 12 01:35:39 sso krb5kdc[1786](debug): .. .. ok
Oct 12 01:35:39 sso krb5kdc[1786](debug): client needs preauth, no hw
preauth; request has preauth, no hw preauth
Oct 12 01:35:39 sso krb5kdc[1786](debug): original preauth mechanism list:
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... etype-info(11)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... etype-info2(19)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pw-salt(3)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... encrypted_challenge(138)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(16)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(14)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(15)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(147)
Oct 12 01:35:39 sso krb5kdc[1786](debug): sorted preauth mechanism list:
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(16)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(14)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(15)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... encrypted_challenge(138)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... etype-info(11)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... etype-info2(19)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pw-salt(3)
Oct 12 01:35:39 sso krb5kdc[1786](debug): ... pkinit(147)
Oct 12 01:35:39 sso krb5kdc[1786](info): AS_REQ (4 etypes {18 17 16 23})
10.0.0.252: KDC_RETURN_PADATA: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for
krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Cannot allocate memory

Any suggestions appreciated.

Thanks,

Benjamin Kaduk

unread,

Oct 11, 2013, 9:45:54 PM10/11/13

to James Croall, kerb...@mit.edu

There are certainly some places in the pkinit code where the return value
is initialized to ENOMEM which can get returned for failures other than
memory allocation. It's hard to venture a guess as to which one(s) you
are running into, though.

Do you have a sense for how reproducible the problem is? (E.g., on a
single client/machine level, all requests, somewhere in between.) If it
is reproducible, a captured packet could in principle be replayed against
a debugging KDC and the execution stepped through to find where the error
is returned.

One coarse-grained factor is whether you are using the openssl or NSS
backend for pkinit.

-Ben Kaduk

James Croall

unread,

Oct 11, 2013, 9:54:43 PM10/11/13

to Benjamin Kaduk, kerb...@mit.edu

Since discovering the symptoms it is reproducible every time - from
systems that are able to kinit normally, it happens when I kinit -n. From
the new systems that are trying to bootstrap, it happens when I kinit -n.

Nothing has (to my knowledge) changed on these hosts. Indeed the KDC and
normal Kerberos clients have been up for 80 days now with no
patches/updates!

I will try and capture the transaction/packets.

James Croall

unread,

Oct 11, 2013, 11:54:20 PM10/11/13

to James Croall, Benjamin Kaduk, kerb...@mit.edu

Some sleuthing and adding DEBUG to pkinit.so reveals:

pkinit_find_realm_context: returning context at 0x20108c0 for realm
'TRIAL.COVERITY.COM'
pkinit_return_padata: entered!
KDC picked etype = 18
received DH key delivery AS REQ
building certificate chain
cert = /C=US/ST=CA/L=San Francisco/O=Coverity Free
Trial/CN=sso.trial.coverity.com
callback function: 10 (certificate has expired)
failed to create a certificate chain: certificate has expired <===
failed to create pkcs7 signed data
pkinit_fini_kdc_req_context: freeing reqctx at 0x2030c30
pkinit_fini_req_crypto: freeing ctx at 0x2030950
Oct 12 03:51:02 sso krb5kdc[2507](info): AS_REQ (4 etypes {18 17 16 23})

10.0.0.252: KDC_RETURN_PADATA: WELLKNOWN/ANON...@TRIAL.COVERITY.COM for
krbtgt/TRIAL.COV...@TRIAL.COVERITY.COM, Cannot allocate memory

AHA! I must have accidentally set the certificate to expire in a month
rather than a year. Approximate times line up. Reasonable user error. Very
poor error reporting though!

Greg Hudson

unread,

Oct 12, 2013, 12:43:21 AM10/12/13

to James Croall, kerb...@mit.edu

On 10/11/2013 11:54 PM, James Croall wrote:
> AHA! I must have accidentally set the certificate to expire in a month
> rather than a year. Approximate times line up. Reasonable user error. Very
> poor error reporting though!

I believe I improved the error reporting for this case in 1.11:

https://github.com/krb5/krb5/commit/6d19259c7eb9277c12a7f2eec9aa80563b4c5acc

Can you confirm that you are running 1.10 or earlier?

James Croall

unread,

Oct 12, 2013, 11:12:37 AM10/12/13

to Greg Hudson, kerb...@mit.edu

Hi Greg,

Yup - Running "1.10+dfsg~beta1" - the default on my Ubuntu systems.

In retrospect I should have not just followed the pkinit setup
instructions blindly, running openssl commands without giving them some
thought. Without specifying days it will default to 30 days, and combined
with the lack of good error reporting... Whew, panic time!

Appear to be all good now.

Cheers,