Debugging why KRB5_KTNAME isn't working

[Brian J. Murrell]

I am trying to debug why having KRB5_KTNAME set in the environment of a
process is not actually making that process use that keytab file but
instead is using the default /etc/krb5.keytab.

The process is Postfix's SMTP deamon (smtpd).

I have confirmed that the method of setting the environment variable is
working:

# ps -ef | grep smtpd
postfix 3726845 3515138 0 11:56 ? 00:00:00 smtpd -n smtp -t inet -u -o stress= -s 2
# tr '\0' '\n' < /proc/3726845/environ
KRB5_KTNAME=/etc/postfix/smtp.keytab
KRB5_CLIENT_KTNAME=/etc/postfix/smtp.keytab
KRB5_TRACE=/tmp/smtpd_krb5_trace
GENERATION=55

However when looking at what the process is actually trying to open
with strace, it's clear that it's not using /etc/postfix/smtp.keytab:

# grep keytab /tmp/smtpd.strace6
3726845 openat(AT_FDCWD, "/etc/krb5.keytab", O_RDONLY) = -1 EACCES (Permission denied)
3726845 openat(AT_FDCWD, "/etc/krb5.keytab", O_RDONLY) = -1 EACCES (Permission denied)

As you can see, I did attempt to try to trace the kerberos library with

KRB5_TRACE=/tmp/smtpd_krb5_trace

but that file does not actually get anything written to it:

-rwxrwxrwx. 1 root root 0 Jan 27 11:27 smtpd_krb5_trace

and the above strace doesn't show any sign of trying to open or even
stat the file:

# grep smtpd_krb5_trace /tmp/smtpd.strace6
[nothing]

Kerberos version appears to be MIT 1.18.2.

Any thoughts/ideas?

Cheers,
b.

[Ken Hornstein]

>I am trying to debug why having KRB5_KTNAME set in the environment of a
>process is not actually making that process use that keytab file but
>instead is using the default /etc/krb5.keytab.
>
>The process is Postfix's SMTP deamon (smtpd).

>[...]
>Any thoughts/ideas?

Is it possible Postfix is clearing out the environment at startup? I'm
a little unclear if the "environ" file in proc shows the _current_
environment or just the environment at process start (a very brief
test suggests to me that "environ" contains the environment at process
start and modification of the current environment isn't reflected there,
so if Postfix was resetting the environment you wouldn't know it).

--Ken

[Brian J. Murrell]

On Thu, 2022-01-27 at 13:03 -0500, Ken Hornstein wrote:
>
>
> Is it possible Postfix is clearing out the environment at startup?

As anything, I suppose it is possible. It would be doing so in
violation of exactly the purpose of the mechanism that is being used to
set the environment though. Meaning, the environment is not just being
set by the caller and Postfix is clearing it as a matter of good
housekeeping. The variable is actually being specified in a for-
purpose Postfix configuration mechanism. This mechanism is
specifically defined to set environment variables in Postfix processes.

I will ask the Postfix mailing list in any case.

> a very brief
> test suggests to me that "environ" contains the environment at
> process
> start and modification of the current environment isn't reflected
> there,

That's the result of my testing too. I must admit to being a little
surprised at that though. Seems to significantly diminish the
purpose/usefulness of that particular /proc entry.

> so if Postfix was resetting the environment you wouldn't know it

Indeed.

I wonder if you can suggest a simple test that would verify/demonstrate
the functionality of the KRB5_KTNAME env. var. that I can use in my
problem report to the Postfix devs.

Cheers,
b.

[Greg Hudson]

On 1/27/22 12:01 PM, Brian J. Murrell wrote:
> I am trying to debug why having KRB5_KTNAME set in the environment of a
> process is not actually making that process use that keytab file but
> instead is using the default /etc/krb5.keytab.

There are three possible reasons why environment variables might be
ignored. First, Postfix might be asking for a secure krb5 context
(krb5_init_secure_context()). Second (and I think the most likely), the
process may be running with elevated privilege as determined by
secure_getenv(). A setuid or setgid bit on the executable could be
enough to trigger this. Third, as Ken suggested, the program might
clean up its own environment.

If any of these are true, then you have limited options to affect the
program behavior from outside of the process. You can change the
default keytab location in /etc/krb5.conf, but that would be global (and
of course you can't point the program at a different config file via
environment variable because those are ignored).

Of course, the program itself can provide configuration for the keytab
file. I couldn't find any gss_ or krb5_ calls in the Postfix source
code (looking at Viktor Dukhovni's git mirror), so I don't have any
immediate insight as to whether that's currently possible or what would
need to change.

[Ken Hornstein]

>> Is it possible Postfix is clearing out the environment at startup?
>
>As anything, I suppose it is possible. It would be doing so in
>violation of exactly the purpose of the mechanism that is being used to
>set the environment though.

Hm. From postconf(5):

import_environment (default: see postconf -d output)
The list of environment parameters that a privileged Postfix process
will import from a non-Postfix parent process, or name=value environ-
ment overrides. Unprivileged utilities will enforce the name=value
overrides, but otherwise will not change their process environment.

Is that what you're using? It looks to me that if the variable isn't
listed in the import_environment configuration entry, it doesn't make
it very far and is removed by the function clean_env().

(If you want to demonstrate to others how KRB5_KTNAME is supposed to
work, just include the output of "env KRB5_KTNAME=/dev/stdout kinit" or
some other Kerberos program).

--Ken

[Brian J. Murrell]

On Thu, 2022-01-27 at 13:45 -0500, Ken Hornstein wrote:
> >
>
> import_environment (default: see postconf -d output)

> Is that what you're using?

Yes. That is the "for-purpose" mechanism that I alluded to earlier
which is why I posited that if smtpd was clearing the environment it
was doing so in violation of the specific mechanism that was supposed
to make this all work.

> It looks to me that if the variable isn't
> listed in the import_environment configuration entry, it doesn't make
> it very far and is removed by the function clean_env().

In my case, I am using the "name=value" variant so that KRB5_KTNAME is
supposed to be getting an explict value even, rather than relying on
the environment already having the variable set.

> (If you want to demonstrate to others how KRB5_KTNAME is supposed to
> work, just include the output of "env KRB5_KTNAME=/dev/stdout kinit"
> or
> some other Kerberos program).

Indeed. I used as my example:

# KRB5_KTNAME=/etc/postfix/smtp.keytab klist -k
Keytab name: FILE:/etc/postfix/smtp.keytab
KVNO Principal
---- --------------------------------------------------------------------------
1 smtp/server.ex...@EXAMPLE.COM

Cheers,
b.

[Jochen Kellner]

Greg Hudson <ghu...@mit.edu> writes:

> Of course, the program itself can provide configuration for the keytab
> file. I couldn't find any gss_ or krb5_ calls in the Postfix source
> code (looking at Viktor Dukhovni's git mirror), so I don't have any
> immediate insight as to whether that's currently possible or what would
> need to change.

I once configured postfix to uses sasl:

main.cf:83:smtpd_sasl_auth_enable = yes

And in /etc/postfix/sasl/smtpd.conf:

pwcheck_method: auxprop saslauthd
#pwcheck_method: saslauthd
mech_list: plain login gssapi
log_level: 0
keytab: /etc/smtp.keytab

That at least worked some time ago...

Jochen

--
This space is intentionally left blank.

[Ken Hornstein]

>Yes. That is the "for-purpose" mechanism that I alluded to earlier
>which is why I posited that if smtpd was clearing the environment it
>was doing so in violation of the specific mechanism that was supposed
>to make this all work.

Oh, hm. I might be reading the code wrong, but it looks like the
SASL library (which is what eventually calls the Kerberos library) is
initialized _before_ the environment is reset. So if you're not also
setting those variables in the environment in the "traditional" way then
it might not see them.

(And Jochen's suggestion is also good, I forgot about the SASL config
file).

--Ken

[Brian J. Murrell]

On Thu, 2022-01-27 at 20:31 +0100, Jochen Kellner wrote:
>
> I once configured postfix to uses sasl:
>
> main.cf:83:smtpd_sasl_auth_enable = yes

I do have that already.

> And in  /etc/postfix/sasl/smtpd.conf:

Hrm. I don't have this file. But I never did and this all worked
prior to a few days ago when the machine was upgraded from EL7 to EL8,
which unsurprisingly upgrades a lot of things in big jumps. So maybe
this is now necessary.

Ahh. Looking at smtpd's strace output, it seems it's looking in
/etc/sasl2/smtpd.conf on my machine and I do have that file with:

pwcheck_method: saslauthd
mech_list: gssapi plain login

> keytab: /etc/smtp.keytab

And indeed, winner winner, chicken dinner! Adding a "keytab:
/etc/postfix/smtp.keytab" to that file is making smtpd use the correct
keytab file now.

So this must all be new behavior in some upgraded versions.

Cheers,
b.

[Simo Sorce]

The keytab option for the cyrus-sasl gssapi plugin is somewhat new
(considering that RHEL-8 is almost 3 years old now) and is probably
causing the change in behavior wrt environment variables that you are
experiencing.

--
Simo Sorce
RHEL Crypto Team
Red Hat, Inc