NFS no_root_squash not working (permission denied)

[Greg Wooledge]

I am trying to backup files from one server, using another server
which has a tape drive attached. I've done this many times before.
The problem is, *this* time, root on the NFS client can't read the files
on the NFS server. It appears that no_root_squash is being ignored.

I have two Debian systems:

1) NFS server: svr4 (jessie)
2) NFS client: arc1 (squeeze) (has tape drive)

On the server (svr4) I have this line in /etc/exports:

/home -no_subtree_check arc1(ro,no_root_squash,sync)

On the client I use autofs, but for purposes of demonstrating the problem
I will use manual mount commands. (Rest assured it breaks just the same
with autofs mounts.)

arc1:~# mount -v -t nfs svr4:/home /mnt
mount.nfs: timeout set for Mon Aug 8 10:57:37 2016
mount.nfs: trying text-based options 'addr=10.76.142.85'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.76.142.85 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 10.76.142.85 prog 100005 vers 3 prot UDP port 58163
svr4:/home on /mnt type nfs (rw)
arc1:~# ls /mnt/wooledg/Maildir
ls: cannot open directory /mnt/wooledg/Maildir: Permission denied
arc1:~# su wooledg -c 'ls /mnt/wooledg/Maildir'
courierimapkeywords courierimapuiddb cur new tmp
arc1:~# umount /mnt

arc1:~# showmount -e svr4 | grep /home
/home arc1.eeg.ccf.org

So, the file system mounts correctly, and the non-root user can read
the files, but root can't read them.

I have rebooted both the client and server machines. I have tried
restarting NFS services on them, too. I have done "exportfs -u -a" and
"exportfs -a". I've done everything I can think of.

This is very similar to what I see in bug #492970 and this past
discussion:

https://lists.debian.org/debian-user/2008/08/msg01943.html
http://bugs.debian.org/492970

However, both of my systems are newer than the systems described in
that bug report, and I don't think "downgrade to nfs-common 1.1.2"
is a viable solution for me.

Is there any package I might be missing on the new jessie server, either
for general NFS operations, or specifically for compatibility with older
Linux NFS clients?

root@svr4:/# uname -a
Linux svr4 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux
root@svr4:/# dpkg -l | grep -E 'nfs|portmap|rpcbind'
ii libnfsidmap2:amd64 0.25-5 amd64 NFS idmapping library
ii nfs-common 1:1.2.8-9 amd64 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.8-9 amd64 support for NFS kernel server
rc portmap 6.0.0-2 amd64 RPC port mapper
ii rpcbind 0.2.1-6+deb8u1 amd64 converts RPC program numbers into universal addresses

arc1:~# uname -a
Linux arc1 2.6.28-1-amd64 #1 SMP Wed Feb 18 17:16:12 UTC 2009 x86_64 GNU/Linux
arc1:~# dpkg -l | grep -E 'nfs|portmap|rpcbind'
ii libnfsidmap2 0.23-2 An nfs idmapping library
ii nfs-common 1:1.2.2-4squeeze3 NFS support files common to client and server
ii nfs-kernel-server 1:1.2.2-4squeeze3 support for NFS kernel server
ii portmap 6.0.0-2 RPC port mapper

root@svr4:/# tail /var/log/daemon.log
...
Aug 8 10:55:37 svr4 rpc.mountd[1312]: authenticated mount request from arc1.eeg.ccf.org:823 for /home (/home)
Aug 8 10:56:05 svr4 rpc.mountd[1312]: authenticated unmount request from arc1.eeg.ccf.org:809 for /home (/home)
...

One final note, though I don't know how it may relate: svr4 was just
upgraded to Jessie on Friday, to replace a machine that died. I reused
the same IP address. The machine that died was running Jessie but with
a Wheezy kernel; it had been upgraded many times, originally installed
from Lenny IIRC. This all worked fine on the old server. It is not
working on the new server. It had crossed my mind that something was
caching the old server's information due to the IP address reuse, but
I have rebooted both systems since then, so that shouldn't be the case.

[Greg Wooledge]

On Mon, Aug 08, 2016 at 11:14:36AM -0400, Greg Wooledge wrote:
> It appears that no_root_squash is being ignored.
>

> 1) NFS server: svr4 (jessie)
>

> /home -no_subtree_check arc1(ro,no_root_squash,sync)

Nobody? :(

Additional information:

* It's not limited to /home. Other exported directories that should
be no_root_squash'ed are in fact squashed.

* It's not limited to the squeeze client arc1. Mounting from another
client (jessie) still shows the problem.

* The previous machine where it worked was i386. The current machine
where it's *not* working is amd64.

* I have another jessie i386 server where no_root_squash works fine.

* If I install nfs-kernel-server on my jessie amd64 desktop PC and
configure /home for export (exactly the same config as above) and
do "sudo systemctl restart nfs-kernel-server.service" to get rpc.mountd
to run, it fails just like svr4.

My conclusion at this point is no_root_squash is completely broken on
jessie amd64, at least in the default configuration. It works on
jessie i386.

Is there ANYONE using NFS with no_root_squash on jessie amd64 successfully?
If so, please tell me how you did it!

[Greg Wooledge]

On Tue, Aug 09, 2016 at 09:14:56AM -0400, Greg Wooledge wrote:
> On Mon, Aug 08, 2016 at 11:14:36AM -0400, Greg Wooledge wrote:
> > It appears that no_root_squash is being ignored.

Opened bug #833925. :(

[Ulf Volmer]

> Is there ANYONE using NFS with no_root_squash on jessie amd64 successfully?
> If so, please tell me how you did it!

NFS run here with jessie amd64 and no_root_squash fine w/o any problems.

Server is jessie, client is centos/fedora. There must something wrong on
your setup.

best regards
Ulf

[Erwan David]

I've seen some problems with a mix of NFS v3 and v4. Especially if v4 is
enabled but not fully configured.

[Greg Wooledge]

On Wed, Aug 10, 2016 at 11:01:41PM +0200, Erwan David wrote:
> Le 10/08/2016 à 22:38, Ulf Volmer a écrit :
> >> Is there ANYONE using NFS with no_root_squash on jessie amd64 successfully?
> >> If so, please tell me how you did it!
> > NFS run here with jessie amd64 and no_root_squash fine w/o any problems.
> >
> > Server is jessie, client is centos/fedora. There must something wrong on
> > your setup.

Any suggestions on what I can try to change? Could you post your jessie
server's /etc/exports and /etc/default/nfs-kernel-server configs, or at
least the parts relevant to the working no_root_squash mount? And any
other configs that I don't know about....

Which kernel version? Just in case this is a kernel issue.

> I've seen some problems with a mix of NFS v3 and v4. Especially if v4 is
> enabled but not fully configured.

I googled "debian disable nfsv4". It came up with
http://unix.stackexchange.com/questions/205403/disable-nfsv4-server-on-debian-allow-nfsv3

I edited /etc/default/nfs-kernel-server as indicated in the answers,
adding "--no-nfs-version 4" to *TWO* places. Restarted nfs-kernel-server.
Mounted from the client. Still permission denied. Re-ran "exportfs -a"
on the server. Re-mounted from the client. Still permission denied.

(All of this on my desktop "server" of course, not the real server.)

[Greg Wooledge]

On Wed, Aug 10, 2016 at 05:17:32PM -0400, Greg Wooledge wrote:
> Any suggestions on what I can try to change? Could you post your jessie
> server's /etc/exports and /etc/default/nfs-kernel-server configs, or at
> least the parts relevant to the working no_root_squash mount? And any
> other configs that I don't know about....
>
> Which kernel version? Just in case this is a kernel issue.

Additional information: installing and rebooting into the jessie-backports
kernel (linux-image-4.6.0-0.bpo.1-amd64) did not change anything.
Still permission denied.

I am still open to any and all suggestions.

(To be more precise: on my desktop machine, I returned
/etc/default/nfs-kernel-server to its default state; tried again;
rebooted; tried again; installed the backport kernel, rebooted; tried
again; disabled NFSv4 in /etc/default/n-k-s; restarted nfs-common and
nfs-kernel-server services; tried again. Everything I have tried so
far has failed with Permission denied.)

[Ulf Volmer]

On 08/10/2016 11:17 PM, Greg Wooledge wrote:

> Any suggestions on what I can try to change? Could you post your jessie
> server's /etc/exports and /etc/default/nfs-kernel-server configs, or at
> least the parts relevant to the working no_root_squash mount? And any
> other configs that I don't know about....
>
> Which kernel version? Just in case this is a kernel issue.

latest jessie kernel (3.16.0-4-amd64 #1 SMP Debian
3.16.7-ckt25-2+deb8u3), no changes in /etc/default/nfs-kernel-server
configs.

/etc/exports is

/export/backup-n40l bob.clients(rw,no_subtree_check)
virt(rw,no_root_squash,no_subtree_check)

best regards
Ulf

[Greg Wooledge]

On Thu, Aug 11, 2016 at 05:43:12PM +0200, Ulf Volmer wrote:
> /etc/exports is
>
> /export/backup-n40l bob.clients(rw,no_subtree_check)
> virt(rw,no_root_squash,no_subtree_check)

I changed my /etc/exports from what I had to:

/home arc1(ro,no_root_squash,sync,no_subtree_check)

Ran exportfs -a, and IT WORKED.

Thank you, Ulf. None of the rest of this is directed at you.

Holy crap. That means I have to put no_subtree_check in EVERY DAMNED
FIELD of EVERY DAMNED LINE.... :-(

Whyyyyy...?!

Why isn't there any way to tell /etc/exports "I want to add these two
options to EVERY SINGLE FIELD of EVERY SINGLE LINE without having to
type it out a couple hundred times"?

The one option we had that allowed us to write it once per LINE instead
of once per FIELD is now BROKEN?!

Well, I guess I have additional information to add in the bug report.