Bug#875821: ltsp-client-core: /rofs is mounted on NFS - we get Operation not supported when using sudo -iu <username> as root

[Rainer Stumbaum]

Package: ltsp-client-core
Version: 5.5.9-2
Severity: important
Tags: upstream

Dear Maintainer,

we are upgrading our LTSP image from jessie to stretch.  We are running
LTSP against HA NFS servers since 2010 (previously SUN storage systems
(ZFS based), since 2013 against NetApp filers). We encountered
the following problem:
- LTSP stretch client starts up fine
- SSH daemon on the LTSP client is enabled and starts up fine
- Connecting via SSH does not work
- Login in as root on the LTSP client console works fine
- sudo -iu <username> gives the following error
"sudo: unable to execute /bin/bash: Operation not supported"
- Using strace we found that seresuid and then accessing a file gives a
  problem  with the filesystem (overlay on top of NFS)
- We wrote a small C program  to replicate the error and found that
  overlay on top of NFS gives the above "Operation not supported"
  whereas NFS directly works fine (/bin/bash (overlay) vs.
  /rofs/bin/bash (NFS)).

After touching /bin/bash and therefore copying to upper fs /bin/bash
works fine for some UIDs but not for all user IDs.

Best regards
Rainer

-- System Information:
Debian Release: 9.1
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages ltsp-client-core depends on:
ii  console-setup    1.164
ii  init             1.48
ii  initramfs-tools  0.130
ii  iproute          1:4.9.0-1
ii  iproute2         4.9.0-1
ii  kbd              2.0.3-2+b1
ii  libc6            2.24-11+deb9u1
ii  libpopt0         1.16-10+b2
ii  lsb-base         9.20161125
ii  lsb-release      9.20161125
ii  nbd-client       1:3.15.2-3
ii  pxelinux         3:6.03+dfsg-14.1
ii  python           2.7.13-2
ii  syslinux-common  3:6.03+dfsg-14.1
ii  tftp-hpa         5.2+20150808-1+b1

Versions of packages ltsp-client-core recommends:
ii  acpi-support-base  0.142-8
ii  dbus               1.10.18-1
ii  dnsutils           1:9.10.3.dfsg.P4-12.3+deb9u2
ii  pciutils           1:3.5.2-1
ii  procps             2:3.3.12-3

Versions of packages ltsp-client-core suggests:
pn  alsa-utils                        <none>
ii  cron                              3.0pl1-128+b1
pn  cryptsetup                        <none>
pn  cups-bsd                          <none>
pn  freerdp-x11 | rdesktop            <none>
pn  inputattach | joystick            <none>
pn  ldm                               <none>
pn  libasound2-plugins                <none>
pn  ltspfsd                           <none>
ii  netcat-openbsd [netcat]           1.130-3
ii  netcat-traditional [netcat]       1.10-41+b1
ii  ntpdate                           1:4.2.8p10+dfsg-3
pn  numlockx                          <none>
pn  pulseaudio                        <none>
pn  python-serial                     <none>
ii  rsyslog [system-log-daemon]       8.24.0-1
ii  sane-utils                        1.0.25-4.1
pn  sdm-terminal | x-display-manager  <none>
pn  sshfs                             <none>
ii  x11-xserver-utils                 7.7+7+b1

-- Configuration Files:
/etc/ltsp/update-kernels.conf changed:
BOOT_METHODS="NFS"
CMDLINE_LINUX_DEFAULT="init=/sbin/init-ltsp quiet"
CMDLINE_NFS="root=/dev/nfs ip=dhcp boot=nfs NFSOPTS=\"-o timeo=100,retrans=360,hard,nointr,nocto,noacl,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600\" vmwgfx.enable_fbdev=1 vga=0x317 cgroup_enable=memory swapaccount=1 net.ifnames=0"
CMDLINE_NBD="root=/dev/nbd0"
CMDLINE_AOE="root=/dev/etherd/e0.0"
KERNEL_NAMES='s/\(vmlinu[xz]-\)\([^-]*-[^-]*-\)\(.*\)/& \1 \2 \3/p'
INITRD_NAME='s/vmlinu[xz]/initrd.img/p'
KERNEL_PREFIX="vmlinu[z|x]-*"
KERNEL_SUFFIX=""
LIST_KERNELS_64="amd64"
LIST_KERNELS_PAE="686-pae 686-bigmem 686"
LIST_KERNELS_32="586 486 686"
IPAPPEND=2


-- debconf information:
  ltsp-client/abort-installation:

-- debsums errors found:
debsums: changed file /sbin/init-ltsp (from ltsp-client-core package)
debsums: changed file /usr/share/initramfs-tools/scripts/init-premount/udhcp (from ltsp-client-core package)
debsums: changed file /usr/share/ltsp/init-ltsp.d/05-getltsconffile (from ltsp-client-core package)
debsums: changed file /usr/share/ltsp/init-ltsp.d/09-hostname (from ltsp-client-core package)
debsums: changed file /usr/share/ltsp/init-ltsp.d/50-disable-cronjobs (from ltsp-client-core package)

[Vagrant Cascadian]

On 2017-09-18, Matthew Wyneken wrote:
> I have now set up a new LTSP server running Debian 9 with a Debian 9
> client. The LTSP version is 5.5.9. I've been surprised at how easy the
> process has been, but now I've run into a problem that might be more
> difficult to solve.
...
> That no longer seems to be possible with 5.5.9. I've been getting NFS
> stale file handle messages basically every time I update the chroot,
> and the only way to fix this is to reboot the client. This is not an
> option for my installation.

It sounds like you've got a better working NFS setup than I managed to
get working...

In /usr/share/doc/ltsp-server/NEWS.Debian:

LTSP defaults to using NBD in both ltsp-build-client and in
ltsp-client-core, due to incompatibilities using overlay fs from linux
4.x with NFS as a backend.


Another person reported similar issues:

https://sourceforge.net/p/ltsp/mailman/message/36038589/
https://bugs.debian.org/875821

It may be possible to switch to aufs by using the aufs-dkms package, as
aufs is what was used before stretch.

Another option would be fixing the way LTSP mounts using "overlay" fs,
as NFS + overlay fs is used by FAI apparently without issue. FAI uses
dracut instead of initramfs-tools, so that could be a path of further
investigation.


live well,
vagrant