NFS4 secure port
128 views
Skip to first unread message
Jean-Baptiste Denis
Sep 19, 2023, 8:11:57 AM9/19/23
to isilon-u...@googlegroups.com
Hello,
imagine you are in an environment where you have absolute control about who can become root on the system (maybe an HPC
or a k8s cluster).
You are using NFSv3 to mount shares on your clients, using the classical AUTH_SYS security flavor (host based security).
NFSv3 exports on OneFS are secure, meaning mounts requests will only be honored is the origin source port is privileged:
inferior to 1024 so that only root (or executable with the adequate CAP_*linux capability) can perform the initial
mount. It is not kerberos, but works perfectly fine in an environment where you control who has root privileges.
Now imagine that you want to taste some NFSv4 because of reasons (pseudofs, delegation, firewall friendly with only port
2049 to handle...). You enable NFSv4 cluster wide on OneFS and you change vers=3 for vers=4 in your fstab. You keep the
same security flavor, no kerberos involved. You see some nice improvements (or not), or you're happy with the pseudofs
feature with a single mountpoint on the client.
The problem with this setup is that the "secure" export aspect (only root or privileged user can interact with the nfs
server) is not honored. This means that anyone can use a nfs userland client implementation like libnfs (or use an ssh
tunnel) to impersonate any user and access any files from any exported shares (think homes directories on an HPC submit
nodes for example, and projects directories).
So far, the answer has been: use kerberos. I understand why they are saying that, but in that case, they should prohibit
by default the use of NFSv4 with AUTH_SYS security flavor. I think answering using kerberos in this use case is the
wrong answer to the problem. It can take sysadmins by surprise. For us, it means disabling NFSv4 system-wide or
implement stupid firewall rules on each clients.
What do you think ? Are you in this use case ? If you already contacted the support about it, what was the answer ?
========
Here is an exemple on how to reproduce the behavior:
## install some nfs userspace utility (libnfs-utils on RHEL/Debian for example)
$ sudo yum install libnfs-utils
OR
$ git clone https://github.com/sahlberg/libnfs
$ cd libnfs
$ cmake -DENABLE_UTILS=yes .
$ make
Resulting utils binaries are in utils directory.
Try to access a file created by 'bob' (uid is 1234) using 'nfs-cat' under another account by leveraging uid
impersonification. The filesystem can be NOT mounted via fstab, everything is happening in userspace.
$ nfs-cat "nfs://ifs/homes/bob/.ssh/id_rsa?uid=1234&version=4"
42
You can also imagine use the mount command from a system where you've got mount privileges using an ssh tunnel to the
node that have exported shares to it :
remote $ ssh -LN 2049:your_nfs_server:2049 remote_server
From another shell :
remote $ sudo mount -t nfs vers=4 localhost:/ifs/homes /mnt
Create a local user with the 'bob' uid and you can access its home directory, directly from your local machine.
========
Jean-Baptiste
imagine you are in an environment where you have absolute control about who can become root on the system (maybe an HPC
or a k8s cluster).
You are using NFSv3 to mount shares on your clients, using the classical AUTH_SYS security flavor (host based security).
NFSv3 exports on OneFS are secure, meaning mounts requests will only be honored is the origin source port is privileged:
inferior to 1024 so that only root (or executable with the adequate CAP_*linux capability) can perform the initial
mount. It is not kerberos, but works perfectly fine in an environment where you control who has root privileges.
Now imagine that you want to taste some NFSv4 because of reasons (pseudofs, delegation, firewall friendly with only port
2049 to handle...). You enable NFSv4 cluster wide on OneFS and you change vers=3 for vers=4 in your fstab. You keep the
same security flavor, no kerberos involved. You see some nice improvements (or not), or you're happy with the pseudofs
feature with a single mountpoint on the client.
The problem with this setup is that the "secure" export aspect (only root or privileged user can interact with the nfs
server) is not honored. This means that anyone can use a nfs userland client implementation like libnfs (or use an ssh
tunnel) to impersonate any user and access any files from any exported shares (think homes directories on an HPC submit
nodes for example, and projects directories).
So far, the answer has been: use kerberos. I understand why they are saying that, but in that case, they should prohibit
by default the use of NFSv4 with AUTH_SYS security flavor. I think answering using kerberos in this use case is the
wrong answer to the problem. It can take sysadmins by surprise. For us, it means disabling NFSv4 system-wide or
implement stupid firewall rules on each clients.
What do you think ? Are you in this use case ? If you already contacted the support about it, what was the answer ?
========
Here is an exemple on how to reproduce the behavior:
## install some nfs userspace utility (libnfs-utils on RHEL/Debian for example)
$ sudo yum install libnfs-utils
OR
$ git clone https://github.com/sahlberg/libnfs
$ cd libnfs
$ cmake -DENABLE_UTILS=yes .
$ make
Resulting utils binaries are in utils directory.
Try to access a file created by 'bob' (uid is 1234) using 'nfs-cat' under another account by leveraging uid
impersonification. The filesystem can be NOT mounted via fstab, everything is happening in userspace.
$ nfs-cat "nfs://ifs/homes/bob/.ssh/id_rsa?uid=1234&version=4"
42
You can also imagine use the mount command from a system where you've got mount privileges using an ssh tunnel to the
node that have exported shares to it :
remote $ ssh -LN 2049:your_nfs_server:2049 remote_server
From another shell :
remote $ sudo mount -t nfs vers=4 localhost:/ifs/homes /mnt
Create a local user with the 'bob' uid and you can access its home directory, directly from your local machine.
========
Jean-Baptiste
Ryan Parker-Hill
Sep 19, 2023, 8:26:07 PM9/19/23
to isilon-u...@googlegroups.com
Seems like others have had the same question as far back as OneFS 9.2.1, I wouldn't hold your breath for any resolution.
--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/b5677deb-b6a2-8d9e-ce76-a14491c5372c%40pasteur.fr.
Jean-Baptiste DENIS
Sep 21, 2023, 3:13:51 AM9/21/23
to isilon-u...@googlegroups.com
Yes, I really don’t understand why this is not considered as a problem.
Right now, any user having an ssh access to a server with nfs4/auth_sys mountpoints can access everything. Sounds crazy to me, since it is not the case with nfs3 secure mounts.
The problem should be acknowledged and the nfs4/auth_sys setup not possible.
Jean-Baptiste
De : isilon-u...@googlegroups.com <isilon-u...@googlegroups.com> de la part de Ryan Parker-Hill <rya...@aspersion.net>
Envoyé : Wednesday, September 20, 2023 2:25:52 AM
À : isilon-u...@googlegroups.com <isilon-u...@googlegroups.com>
Objet : Re: Isilon-Users NFS4 secure port
Envoyé : Wednesday, September 20, 2023 2:25:52 AM
À : isilon-u...@googlegroups.com <isilon-u...@googlegroups.com>
Objet : Re: Isilon-Users NFS4 secure port
To view this discussion on the web visit
https://groups.google.com/d/msgid/isilon-user-group/CAKsEkJY%2BNidQhU86SevUF%2Ba6Aa8MbmZu%3D73U5wPbV-Gw0juiCA%40mail.gmail.com.
Sebastian Gödecke
Sep 22, 2023, 4:00:50 AM9/22/23
to isilon-u...@googlegroups.com
Okay, i think that is a problem, but in 9.4? or 9.5 there comes a firewall with an update.
Maybe this helps?
To view this discussion on the web visit https://groups.google.com/d/msgid/isilon-user-group/AS8PR09MB657909214F73ED60939F2049A4F8A%40AS8PR09MB6579.eurprd09.prod.outlook.com.
Mit freundlichen Grüßen
Sebastian Gödecke
Sebastian Gödecke
Jean-Baptiste Denis
Sep 22, 2023, 4:58:35 AM9/22/23
to isilon-u...@googlegroups.com
Hello, thank you for your answer.
I don't see a solution based on a firewall on OneFS side though.
I didn't mention this in my initial message because its not security related, but forcing kerberos usage in NFSv4 will
hugely impact performance when clients are using nconnect:
https://access.redhat.com/solutions/6998955
======
The issue was tracked with bugzilla bug 2167197: Bug 2167197 - nfs v4.1+ with kerberos is significantly slower with
nconnect than without (RHEL). As of Mon, March 06 2023, the status of the bugzilla bug 2167197 is CLOSED. This bug has
been closed because the problem described is an issue that will not be fixed. An explanation of why this resolution is
set to WONTFIX should be in the bugzilla and if you cannot access the bug or you want further information contact Red
Hat support.
======
Jean-Baptiste
On 9/22/23 10:00, 'Sebastian Gödecke' via Isilon Technical User Group wrote:
> Okay, i think that is a problem, but in 9.4? or 9.5 there comes a firewall with an update.
> Maybe this helps?
>
> *De :* isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>
> <isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>> de la part de Ryan Parker-Hill
> <rya...@aspersion.net <mailto:rya...@aspersion.net>>
> *Envoyé :* Wednesday, September 20, 2023 2:25:52 AM
> *À :* isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>
> <isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>>
> *Objet :* Re: Isilon-Users NFS4 secure port
> <https://urldefense.com/v3/__https://github.com/sahlberg/libnfs__;!!JFdNOqOXpB6UZW0!t0a4FEBF_0rBI-HeVSoHRxGFHYfl9L1xVGI_v924WdRKhyjIV5fwBTxIunc0CT2dfoLwyO2VDsXJlQpufA$>
--
Jean-Baptiste DENIS
HPC Core Facility
I don't see a solution based on a firewall on OneFS side though.
I didn't mention this in my initial message because its not security related, but forcing kerberos usage in NFSv4 will
hugely impact performance when clients are using nconnect:
https://access.redhat.com/solutions/6998955
======
The issue was tracked with bugzilla bug 2167197: Bug 2167197 - nfs v4.1+ with kerberos is significantly slower with
nconnect than without (RHEL). As of Mon, March 06 2023, the status of the bugzilla bug 2167197 is CLOSED. This bug has
been closed because the problem described is an issue that will not be fixed. An explanation of why this resolution is
set to WONTFIX should be in the bugzilla and if you cannot access the bug or you want further information contact Red
Hat support.
======
Jean-Baptiste
On 9/22/23 10:00, 'Sebastian Gödecke' via Isilon Technical User Group wrote:
> Okay, i think that is a problem, but in 9.4? or 9.5 there comes a firewall with an update.
> Maybe this helps?
>
> Am Do., 21. Sept. 2023 um 09:13 Uhr schrieb Jean-Baptiste DENIS <jbd...@pasteur.fr <mailto:jbd...@pasteur.fr>>:
>
> Yes, I really don’t understand why this is not considered as a problem.
>
> Right now, any user having an ssh access to a server with nfs4/auth_sys mountpoints can access everything. Sounds
> crazy to me, since it is not the case with nfs3 secure mounts.
>
> The problem should be acknowledged and the nfs4/auth_sys setup not possible.
>
> Jean-Baptiste
> ------------------------------------------------------------------------------------------------------------------------
>
> Yes, I really don’t understand why this is not considered as a problem.
>
> Right now, any user having an ssh access to a server with nfs4/auth_sys mountpoints can access everything. Sounds
> crazy to me, since it is not the case with nfs3 secure mounts.
>
> The problem should be acknowledged and the nfs4/auth_sys setup not possible.
>
> Jean-Baptiste
> *De :* isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>
> <isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>> de la part de Ryan Parker-Hill
> <rya...@aspersion.net <mailto:rya...@aspersion.net>>
> *Envoyé :* Wednesday, September 20, 2023 2:25:52 AM
> *À :* isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>
> <isilon-u...@googlegroups.com <mailto:isilon-u...@googlegroups.com>>
> *Objet :* Re: Isilon-Users NFS4 secure port
> Seems like others have had the same question as far back as OneFS 9.2.1, I wouldn't hold your breath for any resolution.
>
> https://www.dell.com/community/en/conversations/isilon/prevent-nfs-mount-over-unprivileged-ports/647f9b4ef4ccf8a8def2cbb3 <https://urldefense.com/v3/__https://www.dell.com/community/en/conversations/isilon/prevent-nfs-mount-over-unprivileged-ports/647f9b4ef4ccf8a8def2cbb3__;!!JFdNOqOXpB6UZW0!t0a4FEBF_0rBI-HeVSoHRxGFHYfl9L1xVGI_v924WdRKhyjIV5fwBTxIunc0CT2dfoLwyO2VDsXXWqUQWA$>
>
> $ cd libnfs
> $ cmake -DENABLE_UTILS=yes .
> $ make
>
> Resulting utils binaries are in utils directory.
>
> Try to access a file created by 'bob' (uid is 1234) using 'nfs-cat' under another account by leveraging uid
> impersonification. The filesystem can be NOT mounted via fstab, everything is happening in userspace.
>
> $ nfs-cat "nfs://ifs/homes/bob/.ssh/id_rsa?uid=1234&version=4"
> 42
>
> You can also imagine use the mount command from a system where you've got mount privileges using an ssh tunnel
> to the
> node that have exported shares to it :
>
> remote $ ssh -LN 2049:your_nfs_server:2049 remote_server
>
> From another shell :
>
> remote $ sudo mount -t nfs vers=4 localhost:/ifs/homes /mnt
>
> Create a local user with the 'bob' uid and you can access its home directory, directly from your local machine.
>
> ========
>
> Jean-Baptiste
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isilon-user-gr...@googlegroups.com <mailto:isilon-user-group%2Bunsu...@googlegroups.com>.
> $ cmake -DENABLE_UTILS=yes .
> $ make
>
> Resulting utils binaries are in utils directory.
>
> Try to access a file created by 'bob' (uid is 1234) using 'nfs-cat' under another account by leveraging uid
> impersonification. The filesystem can be NOT mounted via fstab, everything is happening in userspace.
>
> $ nfs-cat "nfs://ifs/homes/bob/.ssh/id_rsa?uid=1234&version=4"
> 42
>
> You can also imagine use the mount command from a system where you've got mount privileges using an ssh tunnel
> to the
> node that have exported shares to it :
>
> remote $ ssh -LN 2049:your_nfs_server:2049 remote_server
>
> From another shell :
>
> remote $ sudo mount -t nfs vers=4 localhost:/ifs/homes /mnt
>
> Create a local user with the 'bob' uid and you can access its home directory, directly from your local machine.
>
> ========
>
> Jean-Baptiste
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isilon-user-group/b5677deb-b6a2-8d9e-ce76-a14491c5372c%40pasteur.fr
> <https://urldefense.com/v3/__https://groups.google.com/d/msgid/isilon-user-group/b5677deb-b6a2-8d9e-ce76-a14491c5372c*40pasteur.fr__;JQ!!JFdNOqOXpB6UZW0!t0a4FEBF_0rBI-HeVSoHRxGFHYfl9L1xVGI_v924WdRKhyjIV5fwBTxIunc0CT2dfoLwyO2VDsUTDr4dOA$>.
> https://groups.google.com/d/msgid/isilon-user-group/b5677deb-b6a2-8d9e-ce76-a14491c5372c%40pasteur.fr
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isilon-user-gr...@googlegroups.com <mailto:isilon-user-gr...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isilon-user-group/CAKsEkJY%2BNidQhU86SevUF%2Ba6Aa8MbmZu%3D73U5wPbV-Gw0juiCA%40mail.gmail.com <https://urldefense.com/v3/__https://groups.google.com/d/msgid/isilon-user-group/CAKsEkJY*2BNidQhU86SevUF*2Ba6Aa8MbmZu*3D73U5wPbV-Gw0juiCA*40mail.gmail.com?utm_medium=email&utm_source=footer__;JSUlJQ!!JFdNOqOXpB6UZW0!t0a4FEBF_0rBI-HeVSoHRxGFHYfl9L1xVGI_v924WdRKhyjIV5fwBTxIunc0CT2dfoLwyO2VDsU2dWZTuQ$>.
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isilon-user-gr...@googlegroups.com <mailto:isilon-user-gr...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isilon-user-group/AS8PR09MB657909214F73ED60939F2049A4F8A%40AS8PR09MB6579.eurprd09.prod.outlook.com <https://urldefense.com/v3/__https://groups.google.com/d/msgid/isilon-user-group/AS8PR09MB657909214F73ED60939F2049A4F8A*40AS8PR09MB6579.eurprd09.prod.outlook.com?utm_medium=email&utm_source=footer__;JQ!!JFdNOqOXpB6UZW0!o7yMvPgXDNGrzEnhcdr4TzCDRgpeTSZIXoVcDvB1dVKXHYp981WV44dLtVlA1rSpBefdPwDMtRmOXje0zy1u-ntScI9YFdOjLQ$>.
>
>
>
> --
> Mit freundlichen Grüßen
> Sebastian Gödecke
>
>
>
> --
> Mit freundlichen Grüßen
> Sebastian Gödecke
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isilon-user-gr...@googlegroups.com <mailto:isilon-user-gr...@googlegroups.com>.
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isilon-user-group/CAJRNCbb0nacDf5fnkfcp5s%3D-Wu_mbu735Ac3B1Y5yL9ZDz-c6Q%40mail.gmail.com <https://urldefense.com/v3/__https://groups.google.com/d/msgid/isilon-user-group/CAJRNCbb0nacDf5fnkfcp5s*3D-Wu_mbu735Ac3B1Y5yL9ZDz-c6Q*40mail.gmail.com?utm_medium=email&utm_source=footer__;JSU!!JFdNOqOXpB6UZW0!o7yMvPgXDNGrzEnhcdr4TzCDRgpeTSZIXoVcDvB1dVKXHYp981WV44dLtVlA1rSpBefdPwDMtRmOXje0zy1u-ntScI-DD9V29w$>.
--
Jean-Baptiste DENIS
HPC Core Facility
Reply all
Reply to author
Forward
0 new messages