Bug#622394: systemd: nfs-common and rpcbind unit files to fix systemd NFS issues properly

[Matthew Grant]

Package: nfs-common
Version: 1:1.2.8-6
Followup-For: Bug #622394

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

* What led up to the situation?

When running under systemd:

o NFS mounts from /etc/fstab do not work.

o NFS exports also fail due to rpcbind not starting before nfs-common and nfs-
kernel-server

systemd is the new default system init for linux. The above should just work.

* What exactly did you do (or not do) that was effective (or
ineffective)?

Created my own /etc/tmpfiles.d/rpcbind.conf:

--------
#Type Path Mode UID GID Age Argument
d /run/rpcbind 0755 root root - -
f /run/rpcbind/rpcbind.xdr 0600 root root - -
f /run/rpcbind/portmap.xdr 0600 root root - -
--------

and /lib/systemd/system file (I did this one in /etc/systemd/system):

-------
[Unit]
Description=RPC bind portmap service
After=systemd-tmpfiles-setup.service
Wants=remote-fs-pre.target
Before=remote-fs-pre.target
DefaultDependencies=no

[Service]
ExecStart=/sbin/rpcbind -f -w
KillMode=process
Restart=on-failure

[Install]
WantedBy=sysinit.target
Alias=portmap
--------

and enabled above unit:

# systemctl enable rpcbind.service

Did for nfs-common to make NFS rpc support to start at correct time:

Created /etc/systemd/system/nfs-common.service (can be put in
/lib/systemd/system
--------
[Unit]
Description=NFS Common daemons
Wants=remote-fs-pre.target
DefaultDependencies=no

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/init.d/nfs-common start
ExecStop=/etc/init.d/nfs-common stop

[Install]
WantedBy=sysinit.target

---------
# systemctl enable nfs-common


* What was the outcome of this action?

Rpc Bind starting correctly, with registration state saving over restart, NFS
service working normally

# systemctl status rpcbind
rpcbind.service - RPC bind portmap service
Loaded: loaded (/etc/systemd/system/rpcbind.service; enabled)
Drop-In: /run/systemd/generator/rpcbind.service.d
└─50-rpcbind-$portmap.conf
Active: active (running) since Wed 2014-05-14 10:38:13 NZST; 13min ago
Main PID: 5066 (rpcbind)
CGroup: name=systemd:/system/rpcbind.service
└─5066 /sbin/rpcbind -f -w

May 14 10:38:13 moriah systemd[1]: Started RPC bind portmap service.

# systemctl status nfs-common
nfs-common.service - NFS Common daemons
Loaded: loaded (/etc/systemd/system/nfs-common.service; enabled)
Active: active (exited) since Wed 2014-05-14 10:35:01 NZST; 19min ago
Main PID: 259 (code=exited, status=0/SUCCESS)
CGroup: name=systemd:/system/nfs-common.service

Warning: Journal has been rotated since unit was started. Log output is
incomplete or unavailable.

All the NFS RPC daemons have port activation in latest nfs-utils upstream, and
service files. Please consider using these as the socket activation saves
haing to manually configure which NFS RPC daemons are needed.




-- Package-specific info:
-- rpcinfo --
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 2 tcp 2049
100227 3 tcp 2049
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 2 udp 2049
100227 3 udp 2049
100021 1 udp 38783 nlockmgr
100021 3 udp 38783 nlockmgr
100021 4 udp 38783 nlockmgr
100021 1 tcp 49538 nlockmgr
100021 3 tcp 49538 nlockmgr
100021 4 tcp 49538 nlockmgr
100005 1 udp 58915 mountd
100005 1 tcp 40052 mountd
100005 2 udp 40524 mountd
100005 2 tcp 60384 mountd
100005 3 udp 55957 mountd
100005 3 tcp 49758 mountd
-- /etc/default/nfs-common --
NEED_STATD=
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=yes
RPCGSSDOPTS=""
-- /etc/idmapd.conf --
[General]
Verbosity = 0
Pipefs-Directory = /run/rpc_pipefs
Domain = internal.anathoth.net
Local-Realms = ANATHOTH.NET
[Translation]
Method = nsswitch
[Mapping]
Nobody-User = nobody
Nobody-Group = nogroup
-- /etc/fstab --
#shalom:/src /media/src nfs noauto,defaults,user,exec 0 0
#shalom:/home /media/home nfs noauto,defaults,user,exec 0 0
#en-gedi:/home /srv/home nfs noauto,async,_netdev,soft,intr,defaults,exec 0 0
-- /proc/mounts --
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0

-- System Information:
Debian Release: jessie/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.14-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_NZ.utf8, LC_CTYPE=en_NZ.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages nfs-common depends on:
ii adduser 3.113+nmu3
ii initscripts 2.88dsf-53
ii libc6 2.18-5
ii libcap2 1:2.22-1.2
ii libcomerr2 1.42.9-3
ii libdevmapper1.02.1 2:1.02.83-2
ii libevent-2.0-5 2.0.21-stable-1
ii libgssglue1 0.4-2
ii libk5crypto3 1.12.1+dfsg-1
ii libkeyutils1 1.5.6-1
ii libkrb5-3 1.12.1+dfsg-1
ii libmount1 2.20.1-5.7
ii libnfsidmap2 0.25-5
ii libtirpc1 0.2.2-7
ii libwrap0 7.6.q-25
ii lsb-base 4.1+Debian12
ii rpcbind 0.2.1-3
ii ucf 3.0028

Versions of packages nfs-common recommends:
ii python 2.7.5-5

Versions of packages nfs-common suggests:
pn open-iscsi <none>
pn watchdog <none>

Versions of packages nfs-kernel-server depends on:
ii libblkid1 2.20.1-5.7
ii libc6 2.18-5
ii libcap2 1:2.22-1.2
ii libgssglue1 0.4-2
ii libsqlite3-0 3.8.4.3-3
ii libtirpc1 0.2.2-7
ii libwrap0 7.6.q-25
ii lsb-base 4.1+Debian12
ii ucf 3.0028

-- Configuration Files:
/etc/default/nfs-common changed [not included]

-- no debconf information

[Simon McVittie]

Control: tags 622394 + unreproducible

On Wed, 14 May 2014 at 11:15:19 +1200, Matthew Grant wrote:
> When running under systemd:
>
> o NFS mounts from /etc/fstab do not work.
>
> o NFS exports also fail due to rpcbind not starting before nfs-common and nfs-
> kernel-server

I think both of these might have been fixed by some combination
of fixing #510528, and changes in systemd v214 improving support for rcS?

Test case:

* two jessie VMs, "server" and "client", in 192.168.122.x subnet
* install nfs-kernel-server on server
* /etc/exports on server: /srv 192.168.122.0/255.255.255.0(rw,sync)
* mkdir /srv/hello on server
* reboot server, it works, no sequencing issues seen in "journalctl -b"
* /etc/fstab on client: 192.168.122.215:/srv /srv nfs _netdev,defaults 0 0
* reboot client, it works, no sequencing issues seen in "journalctl -b"
* client has /srv/hello as desired

Having native systemd units would of course be nice, and native systemd
units for all rcS services seem like a good release goal for jessie+1,
but it seems #622394 (and #748074) might not need to be RC for jessie.

I have some natively systemd-ized versions of Matthew's systemd units
(with enhancements for simpler tmpfiles handling than Riku's debdiff,
more correct dependencies, and /etc/default handling) but so far
they are untested. I'll try to test them on these VMs later.

Regards,
S


--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

[Simon McVittie]

On 07/11/14 19:42, Simon McVittie wrote:
> On 07/11/14 14:57, Chris Butler wrote:
>> Not sure that's the case. The systems where I was seeing the problem
>> are running:
>>
>> ii nfs-common 1:1.2.8-9
>> ii systemd 215-5+b1
>
> Your fstab says:

Sorry, I'm mixing up the various people reporting related but distinct
symptoms. That was Matthew's fstab.

For an attempt at more clarity, here are the distinct sets of symptoms
that have been reported. Please correct me if I'm wrong on any of these.

Bug A, reported by Alban Browaeys, present in 1:1.2.3-2: "systemd[1]:
Found ordering cycle on basic.target/start" and an unbootable system. I
believe newer systemd and/or nfs-common have probably fixed this, so
#622394 should probably have been closed, and the other symptoms should
probably have had distinct bug numbers.

Bug M1, reported by Matthew Grant, present in 1:1.2.8-6: "NFS mounts
from /etc/fstab do not work". (What symptoms? What error messages?) This
one might be related to NFS entries in /etc/fstab that don't have _netdev.

Bug M2, reported by Matthew Grant, present in 1:1.2.8-6: "NFS exports

also fail due to rpcbind not starting before nfs-common and nfs-

kernel-server".

Bug C1, sort-of-reported by Chris Butler, present in 1:1.2.8-9: "the
boot order problem on NFS clients that mount remote filesystems on
boot". This might be the same thing as M1. (Do you have _netdev in the
options of the relevant mounts?)

Bug C2, reported by Chris Butler, present in 1:1.2.8-9 with Matthew's
systemd units added: 'tries to start "statd" before "rpcbind"'. I
believe this is an omission in one of Matthew's systemd units, and I am
about to test a version of nfs-common that should fix that bit.

[Simon McVittie]

Package: nfs-common
Version: 1:1.2.8-6

Severity: wishlist
Control: submitter -1 ma...@mattgrant.net.nz
Control: block -1 by 756900

On Wed, 14 May 2014 at 11:15:19 +1200, Matthew Grant wrote:

> All the NFS RPC daemons have port activation in latest nfs-utils upstream, and
> service files. Please consider using these as the socket activation saves
> haing to manually configure which NFS RPC daemons are needed.

Opening a new bug for this feature request, since it isn't going to happen
for Debian 8.

It will be necessary to be a bit careful about this, since upstream seems
to use nfs-utils.service for the "start all the things" service, whereas
Debian uses (/etc/init.d/nfs-common, corresponding to) nfs-common.service.
nfs-common.service could probably be an Alias for nfs-utils.service
or something.

[Simon McVittie]

Package: rpcbind
Version: 0.2.1-6
Severity: important
X-Debbugs-Cc: nfs-c...@packages.debian.org
X-Debbugs-Cc: Matthew Grant <ma...@mattgrant.net.nz>

On 08/11/14 01:16, Ben Hutchings wrote:
> On Sat, 2014-11-08 at 00:37 +0000, Simon McVittie wrote:

>> On 07/11/14 20:23, Simon McVittie wrote:
>>> Bug M2, reported by Matthew Grant, present in 1:1.2.8-6: "NFS
>>> exports also fail due to rpcbind not starting before nfs-common
>>> and nfs- kernel-server".
>>

>> I have not been able to reproduce this, with or without native
>> systemd units.
>>
>> One possible reason why this could happen sometimes is that
>> /etc/init.d/rpcbind has "Provides: rpcbind", which seems ...
>> redundant. I would expect it to have "Provides: $portmap" in
>> order to be ordered before nfs-common, which has "Required-Start:
>> $portmap".
>
> So would I. But this should result in breakage under sysvinit
> too.

Opening this as a separate bug in rpcbind. It should maybe be
considered RC... but I couldn't reproduce the failure, and as Ben
points out, we'd expect this to have caused breakage under sysvinit
already. So perhaps there's something I haven't spotted that mitigates
this.

Regards,

[Simon McVittie]

On Sat, 08 Nov 2014 at 01:16:27 +0000, Ben Hutchings wrote:
> On Sat, 2014-11-08 at 00:37 +0000, Simon McVittie wrote:

> > The symptom that *I* get on some boots is these messages in the NFS
> > client's Journal/syslog (my test-case for NFS is mounting one machine's
> > /srv onto the other machine's /srv):
> >
> > mount[309]: mount.nfs: Network is unreachable
> > systemd[1]: srv.mount mount process exited, code=exited status=32
> > systemd[1]: Failed to mount /srv.
> > systemd[1]: Dependency failed for Remote File Systems.
>
> This line seems to indicate that /srv was a dependency for 'Remote File
> Systems', i.e. it has correctly been recognised as remote.

That's a good point.

> But network-online.target doesn't seem to have any integration with
> ifupdown, i.e. there is no ifupdown-wait-online.service.

As is that. Michael Biebl has proposed an ifupdown patch in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766943#129
which makes /etc/init.d/networking wait for "udevadm settle",
then bring up allow-hotplug interfaces in addition to auto interfaces.

As far as I can see, that should make networking.service wait for
network interfaces (or at least, the network interfaces that would
have been brought up at this point by Debian 7); if network.target
gained After=networking.service, then that ought to be enough to
fix what I described as bug M1?

(Elsewhere on that bug he also provided a proof-of-concept
ifupdown-wait-online.service which would remove the need for network.target
to have an explicit After=networking.service, AIUI.)

> > > Bug M2, reported by Matthew Grant, present in 1:1.2.8-6: "NFS exports
> > > also fail due to rpcbind not starting before nfs-common and nfs-
> > > kernel-server".
> >

> > I have not been able to reproduce this, with or without native systemd
> > units.
> >
> > One possible reason why this could happen sometimes is that
> > /etc/init.d/rpcbind has "Provides: rpcbind", which seems ... redundant.
> > I would expect it to have "Provides: $portmap" in order to be ordered
> > before nfs-common, which has "Required-Start: $portmap".
>
> So would I. But this should result in breakage under sysvinit too.

I opened https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=768548.

Regards,

[Simon McVittie]

Control: clone 622394 -2 -3
Control: retitle -2 NFS mounts from /etc/fstab do not work
Control: submitter -2 ma...@mattgrant.net.nz
Control: retitle -3 NFS exports fail due to rpcbind not starting before nfs-common and nfs-kernel-server
Control: submitter -3 ma...@mattgrant.net.nz
Control: tags -3 moreinfo unreproducible
Control: user release.d...@packages.debian.org
Control: usertag -2 - jessie-is-blocker
Control: usertag -3 - jessie-is-blocker
Control: tags 622394 unreproducible

On 16/01/15 21:27, Niels Thykier wrote:
> Can anyone provide me with an up to date status on this bug? It seems
> to be a mismatch of no less than 3 different bugs if I understand
> Simon's comment in message #41 correctly (see [1] for the full text).

That is my understanding, but I am not an expert on this package (I
don't use NFS myself).

I was hoping for a response from a maintainer and/or bug reporters,
but apparently I'm not going to get one, so... cloning it is. I hope
I got the usertags and stuff right - please check.

> In particular, this bug is correctly marked as a Jessie blocker due to
> the current bug title, which suggests that nfs-common has a circular
> dependency causing NFS to break under systemd. However, reading the bug
> I am understanding that this loop is now assumed to be fixed?

That's my "bug A" and yes I suspect it's been fixed by better
sysvinit compatibility in systemd... but I could be wrong, and I
don't know specifically what the fix was. Let's use #622394 for this.

I haven't closed it immediately, but I've tagged it unreproducible.
I certainly didn't see it myself.

> If the other bugs (see below) are still present, let us clone this
> bug/file separate bugs for those!

I believe M1 and M2 may be separate bugs, and C1 might be the same
thing as M1 or a fourth bug (it's unclear).

The clone "NFS mounts from /etc/fstab do not work" is M1 and probably
also C1; I'm leaving them entangled for now. Matthew, Chris, if
you have any additional info / newer test results / etc. on this
specific issue, please send it to that clone (check
https://bugs.debian.org/nfs-common for its bug number).

The clone "NFS exports fail due to rpcbind not starting before
nfs-common and nfs-kernel-server" is M2. Matthew, if you have
additional info on that, please send it there (again, check
https://bugs.debian.org/nfs-common for its bug number).

> On the topic of "_netdev": Did you conclude that systemd correctly
> auto-discovered NFS as a _netdev *without* explicitly stating it in
> /etc/fstab?

I believe so. systemd does have code to do this, and Ben's logic
concluding that the test results indicate it is working
correctly does seem valid.

[Dennis Schafroth]

Control: tags 775542 -moreinfo -unreproducible


I am seeing this too. I am running wheezy with backports using systemd as PID 1.

After a reboot:
Jan 29 14:47:58 debian kernel: [ 24.331523] Installing knfsd (copyright (C) 1996 ok...@monad.swb.de).
Jan 29 14:48:02 debian nfs-kernel-server[1384]: Exporting directories for NFS kernel daemon....
Jan 29 14:48:02 debian nfs-kernel-server[1384]: Starting NFS kernel daemon: nfsd
Jan 29 14:48:02 debian nfs-kernel-server[1384]: Not starting: portmapper is not running ... (warning).

debian:~# /etc/init.d/nfs-kernel-server status
nfs-kernel-server.service - LSB: Kernel NFS server support
Loaded: loaded (/etc/init.d/nfs-kernel-server)
Active: active (exited) since Thu 2015-01-29 14:48:00 CET; 38s ago
Process: 1384 ExecStart=/etc/init.d/nfs-kernel-server start (code=exited, status=0/SUCCESS)

debian:~# /etc/init.d/rpcbind status
rpcbind.service - LSB: RPC portmapper replacement
Loaded: loaded (/etc/init.d/rpcbind)
Drop-In: /run/systemd/generator/rpcbind.service.d
└─50-rpcbind-$portmap.conf
Active: inactive (dead)

Now stopping and starting nfs-kernel-server makes it works:

Jan 29 14:50:01 debian nfs-kernel-server[2729]: Stopping NFS kernel daemon: mountd nfsd.
Jan 29 14:50:01 debian nfs-kernel-server[2729]: Unexporting directories for NFS kernel daemon....
Jan 29 14:50:06 debian systemd[1]: Starting LSB: RPC portmapper replacement...
Jan 29 14:50:06 debian rpcbind[2781]: Starting rpcbind daemon....
Jan 29 14:50:06 debian systemd[1]: Started LSB: RPC portmapper replacement.
Jan 29 14:50:06 debian nfs-kernel-server[2794]: Exporting directories for NFS kernel daemon....
Jan 29 14:50:07 debian kernel: [ 153.371831] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
Jan 29 14:50:07 debian nfs-kernel-server[2794]: Starting NFS kernel daemon: nfsd mountd.
Jan 29 14:50:07 debian rpc.mountd[2825]: Version 1.2.6 starting

What should I provide of logging?

cheers,
:-Dennis

On Sun, 25 Jan 2015 11:00:51 +0100 Niels Thykier <ni...@thykier.net> wrote:
> Control: severity 775542 important
>
> Hi,
>
> I am downgrading this bug as it is tagged moreinfo and unreproducible.
> If you still experience this issue, please follow up with more
> information and remove the "moreinfo" and "unreproducible" tags[1].
>
> For the sake of clarity - this is about "NFS exports fail due to rpcbind
> not starting before nfs-common and nfs-kernel-server". You may be
> receiving this mail if you were involved in #622394, which got split
> into 3 different bugs:
>
> * #622394 - nfs-common: breaks systemd - dependency cycle in
> require-start leads to removal of critical jobs
> - Just got closed
> * #775541 - NFS mounts from /etc/fstab do not work
> * #775542 - NFS exports fail due to rpcbind not starting before
> nfs-common and nfs-kernel-server
> - "This bug"
>
> Thanks,
> ~Niels
>
> [1] This can be done by replying to this mail and then add the following
> as the *very first line* of the mail:
>
> Control: tags 775542 -moreinfo -unreproducible

[Martin Pitt]

Hey all,

Simon McVittie [2014-11-08 0:37 +0000]:
> I've now tried a version of nfs-common and rpcbind with systemd units
> similar to those that Matthew Grant attached to bugs #622394 and
> #748074. See attached for the patches I used.

For the record, I just uploaded the rpcbind one to Ubuntu. I'm
currently systemd'ifying nfs-utils (see https://launchpad.net/bugs/1312976)
and ran into that ordering cycle, due t rpcbind's init.d script
requiring $network. The unit avoids that nicely and works fine.

Thanks!

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Martin Pitt]

Hello all,

Simon McVittie [2014-11-08 0:37 +0000]:

> --- rpcbind-0.2.1/debian/rpcbind.service 1970-01-01 01:00:00.000000000 +0100
> +++ rpcbind-0.2.1/debian/rpcbind.service 2014-11-07 10:32:08.000000000 +0000
> @@ -0,0 +1,19 @@
> +[Unit]
> +Description=RPC bind portmap service
> +After=systemd-tmpfiles-setup.service
> +Wants=remote-fs-pre.target
> +Before=remote-fs-pre.target
> +DefaultDependencies=no
> +
> +[Service]
> +Environment="OPTIONS=-w"
> +ExecStart=/sbin/rpcbind $OPTIONS
> +EnvironmentFile=-/etc/rpcbind.conf
> +EnvironmentFile=-/etc/default/rpcbind
> +Type=forking
> +KillMode=process
> +Restart=on-failure
> +
> +[Install]
> +WantedBy=sysinit.target
> +Alias=portmap

For the record, this should be "portmap.service", not just "portmap".
This is present in all proposed patches in this bug report so far.
Thanks to Michael Biebl for spotting this!

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)