Bug#846377: [systemd] /lib/systemd/systemd --user starts dbus-daemon without AFS token

[Dirk Heinrichs]

Package: systemd
Version: 232-6
Severity: important

--- Please enter the report below this line. ---
I'm running systems with user home directories located in an OpenAFS
network filesystem. This used to work fine for years. However, since
some time now, some desktop environments/applications (KDE, Evolution,
etc.) have trouble writing their config files, while writing to the
same file from within a shell worked fine.

I did some investigation and found out that dbus-daemon is not started
be the pam-authenticated user session anymore, but
via /lib/systemd/systemd --user.

This in itself wouldn't be a problem, but /lib/systemd/systemd --user
has been started by PID 1 and thus doesn't run with an AFS token, which
means that all processes spawned from it don't have one either:

testuser 2013 1 0 18:54 ? 00:00:00 /lib/systemd/systemd
--user
testuser 2015 2013 0 18:54 ? 00:00:00 (sd-pam)
testuser 7783 2013 0 19:29 ? 00:00:01 /usr/bin/dbus-daemon
--session --address=systemd: --nofork --nopidfile --systemd-activation

This means that any application that wants to access files through dbus
fails to do so, for example:

(evolution:9447): dconf-WARNING **: failed to commit changes to dconf:
GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code2:
Cannot open dconf database: Failed to open file
'/afs/altum.de/home/testuser/.config/dconf/user': Permission denied

To verify, I added an AFS ACL entry to each sub-directory of testuser's
home, which allowed write access for system:anyuser. Afterwards, the
errors were gone.

Of course, it's not a solution to grant unauthenticated
users write access to every user's home directory.

So, in it's current form, this setup makes most desktop environments
simply unusable.

--- System information. ---
Architecture: Kernel: Linux 4.8.0-1-amd64

Debian Release: stretch/sid
990 testing www.deb-multimedia.org 990 testing
ftp.de.debian.org 500 syncthing apt.syncthing.net 500 stable
update.devolo.com 500 stable repo.saltstack.com
--- Package information. ---
Depends (Version) | Installed
===========================================-+-=========================
libacl1 (>= 2.2.51-8) | 2.2.52-3
libapparmor1 (>= 2.9.0-3+exp2) | 2.10.95-6
libaudit1 (>= 1:2.2.1) | 1:2.6.7-1
libblkid1 (>= 2.19.1) | libc6
(>= 2.17) | libcap2 (>= 1:2.10) |
libcryptsetup4 (>= 2:1.4.3) | libgcrypt20
(>= 1.7.0) | libgpg-error0 (>= 1.14) |
libidn11 (>= 1.13) | libip4tc0
| libkmod2 (>= 5~) |
liblz4-1 (>= 0.0~r127) | liblzma5 (>=
5.1.1alpha+20120614) | libmount1 (>= 2.26.2) |
libpam0g (>= 0.99.7.1) | libseccomp2
(>= 2.3.1) | libselinux1 (>= 2.1.9) |
libsystemd0 (= 232-6) | util-linux
(>= 2.27.1) | mount (>= 2.26) |
adduser |

Package Status (Version) | Installed
==============================-+-===========
udev | 232-6
dracut | initramfs-tools | 0.125


Recommends (Version) | Installed
=============================-+-===========
libpam-systemd | 232-6
dbus | 1.10.12-1


Suggests (Version) | Installed
================================-+-===========
systemd-ui | systemd-container | 232-6
policykit-1 | 0.105-17



--- Output from package bug script ---




--
Dirk Heinrichs <dirk.he...@altum.de>
GPG Public Key CB614542 | Jabber: dirk.he...@altum.de
Tox: he...@toxme.se
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de

[Michael Biebl]

Am 30.11.2016 um 20:01 schrieb Dirk Heinrichs:
> Package: systemd
> Version: 232-6
> Severity: important
>
> --- Please enter the report below this line. ---
> I'm running systems with user home directories located in an OpenAFS
> network filesystem. This used to work fine for years. However, since
> some time now, some desktop environments/applications (KDE, Evolution,
> etc.) have trouble writing their config files, while writing to the
> same file from within a shell worked fine.
>
> I did some investigation and found out that dbus-daemon is not started
> be the pam-authenticated user session anymore, but
> via /lib/systemd/systemd --user.

How is this AFS token set? Is it an environment variable?


--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Michael Biebl]

Am 30.11.2016 um 20:01 schrieb Dirk Heinrichs:

> Package: systemd
> Version: 232-6
> Severity: important
>
> --- Please enter the report below this line. ---
> I'm running systems with user home directories located in an OpenAFS
> network filesystem. This used to work fine for years. However, since
> some time now, some desktop environments/applications (KDE, Evolution,
> etc.) have trouble writing their config files, while writing to the
> same file from within a shell worked fine.
>
> I did some investigation and found out that dbus-daemon is not started
> be the pam-authenticated user session anymore, but
> via /lib/systemd/systemd --user.
>
> This in itself wouldn't be a problem, but /lib/systemd/systemd --user
> has been started by PID 1 and thus doesn't run with an AFS token, which
> means that all processes spawned from it don't have one either:
>
> testuser 2013 1 0 18:54 ? 00:00:00 /lib/systemd/systemd
> --user
> testuser 2015 2013 0 18:54 ? 00:00:00 (sd-pam)
> testuser 7783 2013 0 19:29 ? 00:00:01 /usr/bin/dbus-daemon
> --session --address=systemd: --nofork --nopidfile --systemd-activation
>
> This means that any application that wants to access files through dbus
> fails to do so, for example:

Afaics, this will affect any service which was started as a systemd
--user service. dbus is just one of them.

This was mentiond on IRC:

> <grawity> afaik, AFS tokens are stored as special keys in the
> keyring, nowadays... so it might work if afs was patched to look in
> the 'user' keyring, or if regular logins somehow joined systemd's
> session keyring instead of creating a new one
> <grawity> (CIFS has the same problem)

So this looks like something the openafs maintainers have to look into,
I've CCed their maintainers for their input.

Should we assign this to openafs? Is there something which needs to be
done on the systemd side, and if so, further information and help would
be welcome.

Regards,
Michael

[Russ Allbery]

Michael Biebl <bi...@debian.org> writes:

> Should we assign this to openafs? Is there something which needs to be
> done on the systemd side, and if so, further information and help would
> be welcome.

I don't suppose there's any way to get systemd --user to open a PAM
session on behalf of the user before starting to run programs? That would
probably solve the problem (although there may still be some complications
in making sure it has correct visibility to the Kerberos ticket cache).

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

[Benjamin Kaduk]

On Wed, Nov 30, 2016 at 09:11:58PM +0100, Michael Biebl wrote:
> Am 30.11.2016 um 20:01 schrieb Dirk Heinrichs:
>
> Afaics, this will affect any service which was started as a systemd
> --user service. dbus is just one of them.

I have not absorbed the full report yet, but wanted to note that Dave Botsch (IIRC)
put together some notes on using AFS with systemd --user at:
https://docs.google.com/document/d/1P27fP1uj-C8QdxDKMKtI-Qh00c5_9zJa4YHjnpB6ODM/pub

-Ben

[Felipe Sateler]

On 30 November 2016 at 17:30, Russ Allbery <r...@debian.org> wrote:
>
> Michael Biebl <bi...@debian.org> writes:
>
> > Should we assign this to openafs? Is there something which needs to be
> > done on the systemd side, and if so, further information and help would
> > be welcome.
>
> I don't suppose there's any way to get systemd --user to open a PAM
> session on behalf of the user before starting to run programs? That would
> probably solve the problem (although there may still be some complications
> in making sure it has correct visibility to the Kerberos ticket cache).

systemd --user does open a pam session (with the systemd-user name).
Maybe the problem is that libpam-afs-session (is this the right
module?) registers itself in the common-session include file but
systemd-user loads only common-session-noninteractive ?


--

Saludos,
Felipe Sateler

[Michael Biebl]

Just a quick side-note while reading that document:

> Unfortunately, to get the service to start when you log in, you need to set up systemd --user directories and symlinks.
>
> Create a $HOME/.config/systemd/user/default.target.wants directory
> Create a symlink in that directory to /etc/systemd/user/aklog.service

You can just add a

[Install]
WantedBy=default.target

to the service file and then run systemctl enable --user aklog.service
This will make sure to create all necessary directories and symlinks.

[Russ Allbery]

That's the right module. Hm, yeah, it's probably a combination of that
plus the fact that it doesn't know how to find the Kerberos ticket cache
and therefore can't get a token.

It's a little weird to me that systemd --user loads
common-session-interactive and then apparently starts xterms in this
particular situation. Those are kind of interactive. But presumably it's
assuming xterm will open its own interactive session?

Anyway, it certainly could be registered in -noninteractive (there was
some reason why I didn't do that), but I think the Kerberos ticket cache
problem will still be an issue. Is there some mechanism to convey the
value of KRB5CCNAME from the user's login environment to systemd --user?

[Michael Biebl]

Am 30.11.2016 um 23:12 schrieb Russ Allbery:
> It's a little weird to me that systemd --user loads
> common-session-interactive and then apparently starts xterms in this
> particular situation. Those are kind of interactive. But presumably it's
> assuming xterm will open its own interactive session?

We originally used common-session.
See #739676 for why this was changed

> Anyway, it certainly could be registered in -noninteractive (there was
> some reason why I didn't do that), but I think the Kerberos ticket cache
> problem will still be an issue. Is there some mechanism to convey the
> value of KRB5CCNAME from the user's login environment to systemd --user?

systemctl --user set-environment
might be what you are looking for.

[Michael Biebl]

Am 30.11.2016 um 23:20 schrieb Michael Biebl:
> Am 30.11.2016 um 23:12 schrieb Russ Allbery:
>> It's a little weird to me that systemd --user loads
>> common-session-interactive and then apparently starts xterms in this
>> particular situation. Those are kind of interactive. But presumably it's
>> assuming xterm will open its own interactive session?
>
> We originally used common-session.
> See #739676 for why this was changed

Fwiw, the processes started by systemd --user are not really
interactive. See dconf for example.
xterm is not started as a user service. I assume you meant the
gnome-terminal-server.service user service, which is the background
process and for which gnome-terminal is a frontend.

[Felipe Sateler]

On 30 November 2016 at 19:20, Michael Biebl <bi...@debian.org> wrote:
> Am 30.11.2016 um 23:12 schrieb Russ Allbery:
>> Anyway, it certainly could be registered in -noninteractive (there was
>> some reason why I didn't do that), but I think the Kerberos ticket cache
>> problem will still be an issue. Is there some mechanism to convey the
>> value of KRB5CCNAME from the user's login environment to systemd --user?
>
> systemctl --user set-environment
> might be what you are looking for.

`systemctl --user import-environment KRB5CCNAME` might be more
appropriate if this variable should be copied from an already existing
environment.


--

Saludos,
Felipe Sateler

[Russ Allbery]

Apologies for my lack of knowledge of systemd in user mode -- it's really
neat but I haven't had a chance to play with it yet. Who would run this
command? Is it something that libpam-afs-session would run from a
postinst maintainer script, or is it something more complicated than that?

[Felipe Sateler]

Well, this command imports an environment variable from the current
environment into the systemd --user one. Therefore, it would need to
be run after each time that environment variable is set...

--

Saludos,
Felipe Sateler

[Benjamin Kaduk]

On Wed, Nov 30, 2016 at 07:31:24PM -0300, Felipe Sateler wrote:
>
> `systemctl --user import-environment KRB5CCNAME` might be more
> appropriate if this variable should be copied from an already existing
> environment.

But when would this run, and what package would be responsible for causing
it to be run? (I would prefer to not require that the user is responsible
for causing it to be run.)

Michael Biebl <bi...@debian.org> writes:

> This was mentiond on IRC:
>
> > <grawity> afaik, AFS tokens are stored as special keys in the
> > keyring, nowadays... so it might work if afs was patched to look in
> > the 'user' keyring, or if regular logins somehow joined systemd's
> > session keyring instead of creating a new one
> > <grawity> (CIFS has the same problem)

The AFS tokens are scoped to a specific PAG (Process Authentication Group),
which can provide cross-process isolation. Processes can request to be
put in a new PAG explicitly if they desire separation, and PAGS are
identified by the afs_pag key type in the session keyring. We generally
don't want to use the user keyring since that could lead to neutering of
the cross-process isolation that the PAGs are expected to provide.

-Ben

P.S. Looking more closely at the linked google doc, it was more likely
to be Jonathan Billings than Dave Botsch who wrote it.

[Benjamin Kaduk]

So, libpam_krb5?

-Ben

[Russ Allbery]

Urk. I definitely want to avoid forking external programs in PAM modules
if possible.

Isn't there a way to configure a set of environment variables that are
automatically populated into the systemd user environment from the user's
session? I mean, this stuff is all set up at login, which I assume is
before systemd --user spawns.

[Michael Biebl]

Am 01.12.2016 um 04:15 schrieb Russ Allbery:
> Benjamin Kaduk <ka...@mit.edu> writes:
>> On Wed, Nov 30, 2016 at 08:55:02PM -0300, Felipe Sateler wrote:
>
>>> Well, this command imports an environment variable from the current
>>> environment into the systemd --user one. Therefore, it would need to be
>>> run after each time that environment variable is set...
>
>> So, libpam_krb5?
>
> Urk. I definitely want to avoid forking external programs in PAM modules
> if possible.

There is a D-Bus API, which you could use.

> Isn't there a way to configure a set of environment variables that are
> automatically populated into the systemd user environment from the user's
> session? I mean, this stuff is all set up at login, which I assume is
> before systemd --user spawns.

The start of the systemd --user instance is triggered via the
pam_systemd module from /etc/pam.d/common-session.



That said, could we try and see if importing the KRB5CCNAME env variable
actually helps.

Dirk, could run
systemctl --user import-environment KRB5CCNAME
systemctl --user restart dbus.service dbus.socket
then kill the running dconf-service process and see if it restarts with
the correct context

[Dirk Heinrichs]

Am 30.11.2016 um 21:42 schrieb Benjamin Kaduk:

> I have not absorbed the full report yet, but wanted to note that Dave Botsch (IIRC)
> put together some notes on using AFS with systemd --user at:
> https://docs.google.com/document/d/1P27fP1uj-C8QdxDKMKtI-Qh00c5_9zJa4YHjnpB6ODM/pub

Will take a look, thanks.

Bye...

Dirk

[Dirk Heinrichs]

Am 01.12.2016 um 12:35 schrieb Michael Biebl:

> Dirk, could run
> systemctl --user import-environment KRB5CCNAME
> systemctl --user restart dbus.service dbus.socket
> then kill the running dconf-service process and see if it restarts with
> the correct context

Sure. Doesn't seem to help. Started evolution from the same shell
afterwards and got the same error as before.

Bye...

Dirk

[Benjamin Kaduk]

On Thu, Dec 01, 2016 at 05:50:33PM +0100, Dirk Heinrichs wrote:
> Am 01.12.2016 um 12:35 schrieb Michael Biebl:
>
> > Dirk, could run
> > systemctl --user import-environment KRB5CCNAME
> > systemctl --user restart dbus.service dbus.socket
> > then kill the running dconf-service process and see if it restarts with
> > the correct context
>
> Sure. Doesn't seem to help. Started evolution from the same shell
> afterwards and got the same error as before.

I think that the KRB5CCNAME thing is only expected to help when combined
with a change to run libpam-afs-session from common-session-noninteractive
instead of common-session only.

-Ben

[Dirk Heinrichs]

Am 01.12.2016 um 18:12 schrieb Benjamin Kaduk:
> I think that the KRB5CCNAME thing is only expected to help when combined
> with a change to run libpam-afs-session from common-session-noninteractive
> instead of common-session only.

On my system, configured with pam-auth-update (so no manual changes),
it's in both.

[Russ Allbery]

Dirk Heinrichs <dirk.he...@altum.de> writes:
> Am 01.12.2016 um 18:12 schrieb Benjamin Kaduk:

>> I think that the KRB5CCNAME thing is only expected to help when
>> combined with a change to run libpam-afs-session from
>> common-session-noninteractive instead of common-session only.

> On my system, configured with pam-auth-update (so no manual changes),
> it's in both.

It may be helpful to manually add the "debug" flag to the pam_afs_session
line and then try this again and see if the syslog debugging log provides
any hints for why it's failing to create a token.

[Martin Pitt]

Hello Dirk,

sorry, I haven't had time to dive into this, but I did notice that
we don't put pam_env.so into /etc/pam.d/systemd-user. Does that help
by any chance?

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Michael Biebl]

Am 02.12.2016 um 22:18 schrieb Russ Allbery:
> Dirk Heinrichs <dirk.he...@altum.de> writes:
>> Am 01.12.2016 um 18:12 schrieb Benjamin Kaduk:
>
>>> I think that the KRB5CCNAME thing is only expected to help when
>>> combined with a change to run libpam-afs-session from
>>> common-session-noninteractive instead of common-session only.
>
>> On my system, configured with pam-auth-update (so no manual changes),
>> it's in both.
>
> It may be helpful to manually add the "debug" flag to the pam_afs_session
> line and then try this again and see if the syslog debugging log provides
> any hints for why it's failing to create a token.

So I guess we are kind of stuck here and I don't know enough of AFS to
move this forward (and I don't have a setup to test anything).
Are we (Debian the distro) really the first to run into this issue?
Maybe there is some work we can steal from other distros, like Fedora.

Michael

[Russ Allbery]

Michael Biebl <bi...@debian.org> writes:

> So I guess we are kind of stuck here and I don't know enough of AFS to
> move this forward (and I don't have a setup to test anything). Are we
> (Debian the distro) really the first to run into this issue? Maybe
> there is some work we can steal from other distros, like Fedora.

I don't personally use AFS any more -- I maintain pam-afs-session for lack
of anyone else having picked it up, and because I wrote it and it uses all
my standard machinery, but unfortunately I'm not in a great position to
work out tricky integrations with AFS home directories. :(

[Michael Biebl]

Control: tags -1 + help

Hi Russ, hi Benjamin

Any idea who we could ask who could help with that?

[Trent Lloyd]

I believe I know the cause for this issue.  I ran into the same issue
when trying to use fscrypt to setup ext4 encryption for /home on Ubuntu
Bionic

/etc/pam.d/systemd-user does not currently call pam_keyinit.so. This
means that the keyring does not link to the user keyring as it should,
and will cause issues with programs needing a key from the keyring.

Something non-obvious about this, is that many desktop session processes
are started under 'systemd-user' instead of the 'session' - this
includes gnome-terminal-server which means any gnome-terminal shell runs
under this context. If you start xterm instead of gnome-terminal, you
get a different keyring and this can cause confusion when debugging the
issue as some processes are in one state and the others are in another
including your primary debug tool gnome-terminal. You can verify this by
running 'systemctl status $(pidof gnome-terminal)' and 'systemctl status
$(pidof xterm)' and note the different hierachy.

The change to add pam_keyinit.so was made upstream in December 2016:
https://github.com/systemd/systemd/commit/ab79099d1684457d040ee7c28b2012e8c1ea9a4f

In my test I add the usual pam_keyinit.so line after "pam_limits.so" and
before "common-session-noninteractive". I am not sure if this is the
most ideal location (but it appears to work).

You can test the behavior by running "keyctl show @s" in both contexts

Working contexts:
- xterm
- SSH login

Broken contexts:
- gnome-terminal
- systemd-run --user keyctl show @s (then check output with journalctl
--user --follow)

When the configuration is broken you will see this output:
lathiat@ubuntu:~/src/systemd$ keyctl show @s
Keyring
  59654779 --alswrv 1000 1000 keyring: _ses
   6806191 ----s-rv 0 0 \_ user: invocation_id

When the configuration is working, you will see a link to the user
session instead:
lathiat@ubuntu:~/src/systemd$ keyctl show @s
Keyring
  59654779 --alswrv 1000 1000 keyring: _ses
   6806191 ----s-rv 0 0 \_ keyring: _uid.1000

As background, what is broken on my test setup with libpam-fscrypt?
gnome-terminal for example is unable to write any file in my encrypted
/home which means that it cannot save preferences, so if you go into
preferences and try to tick a checkbox it will immediately revert and an
error is logged to the journal. You can use the guide at
https://github.com/google/fscrypt to setup such a system if you wish to
fully test my case. But you can simply verify the behavior as above.

This sounds very similar to the behavior described by the AFS users.   A
quick google suggests as I would expect that AFS is in fact using the
kernel keyring.

I detailed most of this same info in my bug for Ubuntu here, including
for reference:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1754270

Regards,
Trent

[Michael Biebl]

Thanks Trent for further investigating this.

Dirk, can you confirm that adding pam_keyinit.so to
/etc/pam.d/systemd-user solves the problem for you as well?

> _______________________________________________
> Pkg-systemd-maintainers mailing list
> Pkg-systemd...@lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-systemd-maintainers

[Dirk Heinrichs]

Am 08.03.2018 um 12:11 schrieb Michael Biebl:

Dirk, can you confirm that adding pam_keyinit.so to /etc/pam.d/systemd-user solves the problem for you as well?

No, it doesn't. After adding it and logging out and back in I still get this:

% keyctl show @s
Keyring
 918482795 ---lswrv      0     0  keyring: _ses.20321
  92578899 ----s--v      0     0   \_ afs_pag: _pag

and, for example:

% systemctl --user enable syncthing
Failed to enable unit: Access denied

However, I got the hint in the related systemd issue, that it might be possible to solve this in AFS, by using the user keyring instead of the session keyring. Will start a discussion on this on openafs-info soon...

Bye...

    Dirk

-- 
Dirk Heinrichs <dirk.he...@altum.de>
GPG Public Key: D01B367761B0F7CE6E6D81AAD5A2E54246986015
Sichere Internetkommunikation: http://www.retroshare.org
Privacy Handbuch: https://www.privacy-handbuch.de