[gentoo-user] nfsv4 issues

[Adam Carter]

I'm trying to troubleshoot a newly setup nfs server, which, sometimes has a 30 second pause (tcpdump shows its server waiting).

# time touch /usr/portage/distfiles/testfile

real    0m30.088s
user    0m0.000s
sys     0m0.001s

I cant see anything in the nfs server debugging so i want to strace nfsd. First i tell it to use single thread so i know i;m stracing the correct thread, but;

# grep OPTS_RPC_NFSD nfs
#OPTS_RPC_NFSD="8 -N2 -V 3 -V 4 -V 4.1"
OPTS_RPC_NFSD="1 -N2 -V 3 -V 4 -V 4.1"
#  systemctl restart nfs-server
# pgrep -lf nfsd
23546 nfsd4_callbacks
23548 nfsd
23549 nfsd
23550 nfsd
23551 nfsd
23552 nfsd
23553 nfsd
23554 nfsd
23555 nfsd

So its not respecting the nproc setting. Any ideas? I also tried changing EXPORTFS_TIMEOUT= since its currently set at 30. It didnt help, but perhaps that's because its being ignored too.

[Tom H]

Are the nfsd versions that you're setting being respected? You can
check with "rpcinfo -s" or "cat /proc/fs/nfsd/versions".

You can change the number of threads on the fly with "echo 1 >
/proc/fs/nfsd/threads".

I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
systemd units that I think that Gentoo's using, you have to re-run
nfs-config.service - or run the script that it calls - in order to
update the "/run/sysconfig/nfs-utils" environment file that's sourced
by the nfs-server.service unit.

Does "/var/lib/nfs/v4recovery/" exist?

Does adding the client to "/etc/hosts" - or to your reverse dns zone -
eliminate the delay?

[Adam Carter]

Are the nfsd versions that you're setting being respected? You can

check with "rpcinfo -s" or "cat /proc/fs/nfsd/versions".

Yep;
 # cat /proc/fs/nfsd/versions
-2 +3 +4 +4.1 +4.2

 

You can change the number of threads on the fly with "echo 1 >
/proc/fs/nfsd/threads".

That works too, but then;
# ps -ef | grep nfsd
root      1454  1426  0 12:47 pts/0    00:00:00 grep --colour=auto nfsd
root     23546     2  0 Jul19 ?        00:00:00 [nfsd4_callbacks]
root     23548     2  0 Jul19 ?        00:00:00 [nfsd]
# strace -p 23548
strace: attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted

I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
systemd units that I think that Gentoo's using, you have to re-run
nfs-config.service - or run the script that it calls - in order to
update the "/run/sysconfig/nfs-utils" environment file that's sourced
by the nfs-server.service unit.

 

In /usr/lib/systemd/system/nfs-server.service
[Service]
EnvironmentFile=/etc/conf.d/nfs

Does "/var/lib/nfs/v4recovery/" exist?

No
 # ls /var/lib/nfs/
etab  export-lock  rmtab  rpc_pipefs  sm  sm.bak  state  xtab

Does adding the client to "/etc/hosts" - or to your reverse dns zone -
eliminate the delay?

DNS is setup and both client and server can forward and reverse lookup each other.

[Tom H]

On Wed, Jul 20, 2016 at 10:51 PM, Adam Carter <adamc...@gmail.com> wrote:


>> I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
>> systemd units that I think that Gentoo's using, you have to re-run
>> nfs-config.service - or run the script that it calls - in order to
>> update the "/run/sysconfig/nfs-utils" environment file that's sourced
>> by the nfs-server.service unit.
>
> In /usr/lib/systemd/system/nfs-server.service
> [Service]
> EnvironmentFile=/etc/conf.d/nfs

Sorry. Looking at the ebuild, there's:

<begin>
rm "${D}$(systemd_get_unitdir)"/nfs-config.service || die
sed -i -r \
-e "/^EnvironmentFile=/s:=.*:=${EPREFIX}/etc/conf.d/nfs:" \
-e '/^(After|Wants)=nfs-config.service$/d' \
-e 's:/usr/sbin/rpc.statd:/sbin/rpc.statd:' \
"${D}$(systemd_get_unitdir)"/* || die
</end>

so the upstream "nfs-config.service" waltz is avoided.

But that means that the variables in "/etc/conf.d/nfs" aren't renamed.
So the openrc nfs script uses "${OPTS_RPC_NFSD}", which is defined,
and the systemd service uses "$RPCNFSDARGS", which isn't.

>> Does "/var/lib/nfs/v4recovery/" exist?
>
> No
> # ls /var/lib/nfs/
> etab export-lock rmtab rpc_pipefs sm sm.bak state xtab

IIRC, it's needed to avoid this delay. I thought that I'd saved a url
about this but I can't find it.

Do you have a syslog message about "stable storage"? "man nfsdcltrack".

The openrc script has

<begin>
mkdir_nfsdirs() {
local d
for d in v4recovery v4root ; do
d="/var/lib/nfs/${d}"
[ ! -d "${d}" ] && mkdir -p "${d}"
done
}
</end>

but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
"/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
the Gentoo ebuild should do the same or should ship a
"/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
doesn't exist.

[Adam Carter]

>> I don't use systemd on Gentoo but for the nfs-utils upstream-shipped
>> systemd units that I think that Gentoo's using, you have to re-run
>> nfs-config.service - or run the script that it calls - in order to
>> update the "/run/sysconfig/nfs-utils" environment file that's sourced
>> by the nfs-server.service unit.
>
> In /usr/lib/systemd/system/nfs-server.service
> [Service]
> EnvironmentFile=/etc/conf.d/nfs

Sorry. Looking at the ebuild, there's:

<begin>
rm "${D}$(systemd_get_unitdir)"/nfs-config.service || die
sed -i -r \
-e "/^EnvironmentFile=/s:=.*:=${EPREFIX}/etc/conf.d/nfs:" \
-e '/^(After|Wants)=nfs-config.service$/d' \
-e 's:/usr/sbin/rpc.statd:/sbin/rpc.statd:' \
"${D}$(systemd_get_unitdir)"/* || die
</end>

so the upstream "nfs-config.service" waltz is avoided.

But that means that the variables in "/etc/conf.d/nfs" aren't renamed.
So the openrc nfs script uses "${OPTS_RPC_NFSD}", which is defined,
and the systemd service uses "$RPCNFSDARGS", which isn't.
 

I've added $RPCNFSDARGS to /etc/conf.d/nfs, restarted, and the nproc setting works.

>> Does "/var/lib/nfs/v4recovery/" exist?
>
> No
> # ls /var/lib/nfs/
> etab export-lock rmtab rpc_pipefs sm sm.bak state xtab

IIRC, it's needed to avoid this delay. I thought that I'd saved a url
about this but I can't find it.

Do you have a syslog message about "stable storage"? "man nfsdcltrack".

There's no message about stable storage, but there's this;
kernel: [578030.628415] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.

# which nfsdcltrack
which: no nfsdcltrack in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/5.4.0:/usr/lib64/subversion/bin:/opt/vmware/bin)
# qlist nfs | grep nfsdcltrack
#
 

The openrc script has

<begin>
mkdir_nfsdirs() {
local d
for d in v4recovery v4root ; do
d="/var/lib/nfs/${d}"
[ ! -d "${d}" ] && mkdir -p "${d}"
done
}
</end>

but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
"/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
the Gentoo ebuild should do the same or should ship a
"/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
doesn't exist.

I've added the directory, and after restarting syslog now has new entries;
kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory

I will test shortly and report back - thanks!

[Adam Carter]

I've added the directory, and after restarting syslog now has new entries;

kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory

I will test shortly and report back - thanks!

Confirmed - this fixes the 30 second delay. Should i log a bug for these issues?

[Tom H]

On Fri, Jul 22, 2016 at 8:22 PM, Adam Carter <adamc...@gmail.com> wrote:


>>>> Does "/var/lib/nfs/v4recovery/" exist?
>>>
>>> No
>>> # ls /var/lib/nfs/
>>> etab export-lock rmtab rpc_pipefs sm sm.bak state xtab
>>
>> IIRC, it's needed to avoid this delay. I thought that I'd saved a url
>> about this but I can't find it.
>>
>> Do you have a syslog message about "stable storage"? "man nfsdcltrack".
>
> There's no message about stable storage, but there's this;
> kernel: [578030.628415] NFSD: the nfsdcld client tracking upcall will be
> removed in 3.10. Please transition to using nfsdcltrack.

It's from

https://patchwork.kernel.org/patch/1730241/

> # which nfsdcltrack
> which: no nfsdcltrack in
> (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/5.4.0:/usr/lib64/subversion/bin:/opt/vmware/bin)
> # qlist nfs | grep nfsdcltrack
> #

It depends on the nfs-utils USE settings:

# qlist -U nfs-utils
net-fs/nfs-utils (libmount nfsdcld nfsidmap nfsv4 nfsv41)

# qfile $(which nfsdcltrack)
net-fs/nfs-utils (/sbin/nfsdcltrack)

>> The openrc script has
>>
>> <begin>
>> mkdir_nfsdirs() {
>> local d
>> for d in v4recovery v4root ; do
>> d="/var/lib/nfs/${d}"
>> [ ! -d "${d}" ] && mkdir -p "${d}"
>> done
>> }
>> </end>
>>
>> but systemd doesn't have anything equivalent. On RHEL and Ubuntu,
>> "/var/lib/nfs/v4recovery/" is created at installation time. Perhaps
>> the Gentoo ebuild should do the same or should ship a
>> "/usr/lib/tmpfiles.d/var-lib-nfs.conf" to create it at boot if it
>> doesn't exist.
>
> I've added the directory, and after restarting syslog now has new entries;
> kernel: [912267.948883] NFSD: Using /var/lib/nfs/v4recovery as the NFSv4
> state recovery directory
> kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery
> directory
>
> I will test shortly and report back - thanks!

Good luck. You're welcome.

[Tom H]

Good.

> Should i log a bug for these issues?

If I were you, I'd definitely file a bug reports against nfs-utils for:

1) the creation of "/var/lib/nfs/v4recovery/" when systemd is pid 1;

2) the systemd unit compatible envvars.