nfsd kernel module not loading

Vince Forgetta

unread,

Sep 23, 2015, 11:51:49 AM9/23/15

to Warewulf

Hi all,

This issues initially arose in the thread on WW3.7 on CentOS 7:

https://groups.google.com/a/lbl.gov/forum/?utm_medium=email&utm_source=footer#!topic/warewulf/_9C67A_7HMQ

I have narrows it down further to a specific issue with the nfsd kernel module on the provisioned node, which for some reason does not load.

1. /proc/fs/nfsd fails to mount:

[root@D1P-HYDRAEX01 ~]# /bin/mount nfsd /proc/fs/nfsd -t nfsd

mount: unknown filesystem type 'nfsd'

2, kernel module is not loaded:

Module Size Used by

isci 137622 0

libsas 83653 1 isci

scsi_transport_sas 41034 2 isci,libsas

megaraid_sas 117760 0

ixgbe 290931 0

mdio 13807 1 ixgbe

igb 192078 0

i2c_algo_bit 13413 1 igb

dca 15130 2 igb,ixgbe

ptp 18933 2 igb,ixgbe

pps_core 19106 1 ptp

i2c_core 40325 2 igb,i2c_algo_bit

ahci 29870 0

libahci 32009 1 ahci

libata 218854 3 ahci,libahci,libsas

sd_mod 45499 0

crc_t10dif 12714 1 sd_mod

crct10dif_common 12595 1 crc_t10dif

3. Kernel module exists:

[root@D1P-HYDRAEX01 ~]# find /lib/modules -name nfsd*

/lib/modules/kernel/fs/nfsd

/lib/modules/kernel/fs/nfsd/nfsd.ko

4. modinfo fails to find module:

[root@D1P-HYDRAEX01 ~]# modinfo nfsd

modinfo: ERROR: Module alias nfsd not found.

All above commands work on the master node.

wwinit ALL reports nfs server working.

Any help would be greatly appreciated. I have been banging my head on this for a while now!

Vince

Chu Ruilin

unread,

Sep 23, 2015, 11:57:39 AM9/23/15

to ware...@lbl.gov

have you tried with wwinit ALL all passed and then regenerate the vnfs image from scratch? I usually solve my problems that way

--
You received this message because you are subscribed to the Google Groups "Warewulf" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
To post to this group, send email to ware...@lbl.gov.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/aeeff7d5-5fa9-4765-876c-1c721d1aa304%40lbl.gov.
For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

Jess Cannata

unread,

Sep 23, 2015, 12:12:27 PM9/23/15

to ware...@lbl.gov

Vince,

I have not done a lot with RHEL/Centos 7, but this does not look right:

[root@D1P-HYDRAEX01 ~]# /bin/mount nfsd /proc/fs/nfsd -t nfsd

mount: unknown filesystem type 'nfsd'

Usually it is a

mount -t nfs

I have not seen type nfsd as a mount option.

Also, it would be helpful if you differentiated between your server/master and the client node. Is D1P-HYDRAEX01 your master or client?

On the master side, send the output of

showmount -e

netstat -anpt |grep nfs

Are you sure that NFS server is working on the master? Have you tried mounting the NFS share on the master?

Thanks,

--
Jess Cannata
R Systems
512-410-9690

Vince Forgetta

unread,

Sep 23, 2015, 12:38:02 PM9/23/15

to ware...@lbl.gov

Thanks Jess.

Yes, mount command I got from somewhere I cannot recall. Canonical command produces same error:

[root@D1P-HYDRAEX01 ~]# mount -t nfsd nfsd /proc/fs/nfsd

mount: unknown filesystem type 'nfsd'

D1P-HYDRAEX01 (above) is wwexecution node i.e. provisioned node from wwmaster.

D1P-HYDRAMS01 is wwmaster node.

D1P-HYDRAFS01 is file server.

From wwmaster (CentOS7) I can mount NFS share served by file server.

From wwexecution node I am unable to mount NFS share served by file server.

Info you requested plus some extra from the execution node:

[root@D1P-HYDRAEX01 ~]# netstat -anpt |grep nfs (no output)

[root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.10

Export list for 192.168.13.10:

/mnt/mount1 192.168.13.10/24

/mnt/mount2 192.168.13.10/24

/mnt/mount3 192.168.13.10/24

/mnt/mount4 192.168.13.10/24

[root@D1P-HYDRAEX01 ~]# rpc.nfsd -d

rpc.nfsd: Checking netconfig for visible protocols.

rpc.nfsd: Enabling inet udp.

rpc.nfsd: Enabling inet tcp.

rpc.nfsd: Enabling inet6 udp.

rpc.nfsd: Enabling inet6 tcp.

rpc.nfsd: Unable to access /proc/fs/nfsd errno 2 (No such file or directory).

Please try, as root, 'mount -t nfsd nfsd /proc/fs/nfsd' and then restart rpc.nfsd to correct the problem

ww commands I use to provision OS to execution node:

# wwmkchroot centos-7 /var/chroots/hydraex-centos7

# wwvnfs --chroot /var/chroots/hydraex-centos7

# wwbootstrap `uname -r`

Warewulf> node new D1P-HYDRAEX01 --netdev=eth3 --hwaddr=0c:c4:7a:1f:05:67 --ipaddr=192.168.13.12 --groups=HYDRAEX --domain=ldi.lan

Warewulf> provision set --lookup groups HYDRAEX --vnfs=hydraex-centos7 --bootstrap=3.10.0-229.14.1.el7.x86_64

# wwsh dhcp update

# systemctl restart dhcpd

# systemctl restart httpd

Reboot execution node.

thanks,

Vince

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAHBpPoVJ5PxOoU6X8aNWjFA6FjF5WHtUMkwrbQoJdgdSoC40JA%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 12:41:52 PM9/23/15

to ware...@lbl.gov

Another small point, I agree that /proc/fs/nfsd file system would under normal circumstances not be mounted manually. I never knew nfsd was even a valid file system type! However, appears that the kernel module does this and it does appear as a file system type on the wwmaster node:

[root@D1P-HYDRAMS01 vforget]# mount |grep nfsd

nfsd on /proc/fs/nfsd type nfsd (rw,relatime)

But not on the execution node:

[root@D1P-HYDRAEX01 ~]# mount | grep nfsd

[root@D1P-HYDRAEX01 ~]#

I assume that this is required to get NFS mounting to work.

Ian Kaufman

unread,

Sep 23, 2015, 12:45:10 PM9/23/15

to ware...@lbl.gov

But this is an execution node, and should not need nfsd running.

Somehow, your execution node config/image/VNFS got messed up. You
might want to start over, but this time, only install what is
necessary for NFS clients.

Ian

> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsAjft2SQdE9kkKUfv8oXN2sJ%2B7isvzvCFbZX1Uj9zng-g%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

--

Ian Kaufman
Research Systems Administrator
UC San Diego, Jacobs School of Engineering ikaufman AT ucsd DOT edu

Jason Stover

unread,

Sep 23, 2015, 12:47:31 PM9/23/15

to ware...@lbl.gov

The nfsd "file system" is on the server side. nfs is the client side:

mount -t nfs server:/export /mnt

When I was fighting with NFS on 7 clients I think I needed to do a:
systemctl enable nfs

It was ignoring NFS entries in /etc/fstab, etc... until I did that.

-J

> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsAjft2SQdE9kkKUfv8oXN2sJ%2B7isvzvCFbZX1Uj9zng-g%40mail.gmail.com.

Jess Cannata

unread,

Sep 23, 2015, 12:49:49 PM9/23/15

to ware...@lbl.gov

Vince,

Jason beat me to it, but you should just try to mount the export on the client:

mount -t nfs 192.168.13.10:/mnt/mount1 /mnt (or whatever the mount point on the client should be)

I did not know about the need for

systemctl enable nfs

on the client side.

--
Jess Cannata
R Systems
512-410-9690

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt-begKNQ45syf6NKpQ0uL8ee3SLMUdoBefZkejM7zjyjw%40mail.gmail.com.

Chu Ruilin

unread,

Sep 23, 2015, 12:55:11 PM9/23/15

to ware...@lbl.gov

This is what I do usually:

wwinit ALL ## everything should be OK, including database, nfsd, dhcp, bootstrap, etc, except the last step vnfs since I did not give CHROOTDIR

wwmkchroot centos-7 /var/chroots/centos-7

and optional step:
yum --tolerant --installroot /var/chroots/centos-7 -y install stuff_you_want_to_put_in_vnfs

wwvnfs --chroot /var/chroots/centos-7

and then I do

wwnodescan --netdev=eth0 --ipaddr=blablabla --netmask=255.255.0.0 --vnfs=centos-7 --bootstrap=`uname -r` --groups=compute c[001-999]

this will do a stateless bootup for all compute nodes and record them to ww. WW will automatically export /home, /usr/local, /opt and /var/chroots on master nodes and mount them on compute nodes. I don't need to manually do anything unless I want something else.

If I want stateful provision, I install kernel and grub to chroot and re-image the vnfs, set bootlocal in ww, reboot all compute nodes with pdsh, and then unset bootlocal in ww, done.

has been working for me every time, that's why I love ww so much

On Wed, Sep 23, 2015 at 12:37 PM, Vince Forgetta <forg...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsBzqnSuWq9g2e0USZ3p%3DXUEaz4wjsABjw6Z3RYaqp-yBg%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 12:55:44 PM9/23/15

to ware...@lbl.gov

@Ian, I will rebuild the image. I hope it solves the issues below.

@Jess, mount reports:

[root@D1P-HYDRAEX01 ~]# mount -t nfs 192.168.13.10:/mnt/mount1 /mnt

mount.nfs: No such device

@Jason: systemctl enable nfs reports:

[root@D1P-HYDRAEX01 ~]# systemctl enable nfs.service

Failed to issue method call: No such file or directory

[root@D1P-HYDRAEX01 ~]# systemctl start nfs.service

A dependency job for nfs-server.service failed. See 'journalctl -xn' for details.

[root@D1P-HYDRAEX01 ~]# journalctl -xn

-- Logs begin at Wed 2015-09-23 05:34:44 EDT, end at Wed 2015-09-23 05:54:13 EDT. --

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan systemd[1]: Dependency failed for NFS Mount Daemon.

-- Subject: Unit nfs-mountd.service has failed

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit nfs-mountd.service has failed.

--

-- The result is dependency.

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan systemd[1]: Unit proc-fs-nfsd.mount entered failed state.

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan systemd[1]: Starting Notify NFS peers of a restart...

-- Subject: Unit rpc-statd-notify.service has begun with start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit rpc-statd-notify.service has begun starting up.

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan rpc.idmapd[4783]: main: open(/var/lib/nfs/rpc_pipefs//nfs): No such file or

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan sm-notify[4788]: Version 1.3.0 starting

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan sm-notify[4788]: Already notifying clients; Exiting!

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan mount[4779]: mount: unknown filesystem type 'nfsd'

Sep 23 05:54:13 D1P-HYDRAEX01.ldi.lan systemd[1]: Started Notify NFS peers of a restart.

-- Subject: Unit rpc-statd-notify.service has finished start-up

-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

--

-- Unit rpc-statd-notify.service has finished starting up.

--

-- The start-up result is done.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAHBpPoX4P%3DiXVdJV9EqHjE4C9MGvK7fW542HNnMJWm-hLH1gYw%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 1:22:19 PM9/23/15

to ware...@lbl.gov

I have rebuilt a new image like so:

# wwmkchroot centos-7 /var/chroots/hydraex-centos7.new

# wwvnfs --chroot /var/chroots/hydraex-centos7.new

# uname -r

3.10.0-229.14.1.el7.x86_64

# wwbootstrap `uname -r`

# wwsh

Warewulf> node new D1P-HYDRAEX01 --netdev=eth3 --hwaddr=0c:c4:7a:1f:05:67 --ipaddr=192.168.13.12 --groups=HYDRAEX --domain=ldi.lan

Warewulf> provision set --lookup groups HYDRAEX --vnfs=hydraex-centos7.new --bootstrap=3.10.0-229.14.1.el7.x86_64

# wwsh dhcp update

# systemctl restart dhcpd

# systemctl restart httpd

# wwinit ALL

nfsd: Activating Systemd service: nfsd

nfsd: + /bin/systemctl -q enable nfs-server.service OK

nfsd: + /bin/systemctl -q restart nfs-server.service OK

Same problem. Moreover /home, /usr/local/ and /var/chroots does not mount either:

Mounts exported by master:

[root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.11

Export list for 192.168.13.11:

/opt 192.168.13.0/255.255.255.0

/usr/local 192.168.13.0/255.255.255.0

/var/chroots 192.168.13.0/255.255.255.0

/home 192.168.13.0/255.255.255.0

On execution node:

root@D1P-HYDRAEX01 ~]# mount /homme

mount: can't find /homme in /etc/fstab

[root@D1P-HYDRAEX01 ~]# mount /home

mount.nfs: No such device

[root@D1P-HYDRAEX01 ~]# mount /usr/local

mount.nfs: No such device

[root@D1P-HYDRAEX01 ~]# mount /var/chroots

mount.nfs: No such device

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAC_eze8H_OHJz0S82Ecw_7DHCzHQ_nULNvGoFw5MVf3Q%3D-jHDA%40mail.gmail.com.

Jason Stover

unread,

Sep 23, 2015, 1:47:23 PM9/23/15

to ware...@lbl.gov

Okay ... as a test, on the client try:

mount -t nfs4 server:/export /mnt

I know I've seen that "No such device" ... and I'm wanting to say it
was a 4 only server on a vers 3 mount request.

-J

> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsDuV3dnfLyGTQbutpHPG1V29r%3DUo3U5vGJK_PX_5nfBww%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 1:55:12 PM9/23/15

to ware...@lbl.gov

Sadly I tried this:

[root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.11

Export list for 192.168.13.11:

/opt 192.168.13.0/255.255.255.0

/usr/local 192.168.13.0/255.255.255.0

/var/chroots 192.168.13.0/255.255.255.0

/home 192.168.13.0/255.255.255.0

[root@D1P-HYDRAEX01 ~]# mount -t nsf4 192.168.13.11:/opt /opt

mount: unknown filesystem type 'nsf4'

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt_9coKMetESJUQ386GQF3eUL5xAq4oBQ86W-i_rY7PW4w%40mail.gmail.com.

Ian Kaufman

unread,

Sep 23, 2015, 2:04:14 PM9/23/15

to ware...@lbl.gov

nfs4, not nsf4

Ian

> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsCw3Jy1Zg-WXeE6%3DyENOUy-M%2B3iwYK2bXK3O55TSXjx0g%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

--

Novosielski, Ryan

unread,

Sep 23, 2015, 2:12:27 PM9/23/15

to ware...@lbl.gov

--------
WARNING: At least one of the links in the message below goes to an IP address (e.g. 10.1.1.1), which could be malicious. To learn how to protect yourself, please go here: https://commons.lbl.gov/x/_591B
--------

Typo there? NSF?

--
____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences*
|| \\UTGERS |---------------------*O*---------------------
||_// Biomedical | Ryan Novosielski - Senior Technologist
|| \\ and Health | novo...@rutgers.edu - 973/972.0922 (2x0922)
|| \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark
`'
________________________________________
From: Vince Forgetta [forg...@gmail.com]
Sent: Wednesday, September 23, 2015 1:54 PM
To: ware...@lbl.gov
Subject: Re: [Warewulf] nfsd kernel module not loading

Sadly I tried this:

[root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.11

Export list for 192.168.13.11<http://192.168.13.11>:
/opt 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
/usr/local 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
/var/chroots 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
/home 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>

[root@D1P-HYDRAEX01 ~]# mount -t nsf4 192.168.13.11:/opt /opt
mount: unknown filesystem type 'nsf4'

On Wed, Sep 23, 2015 at 1:46 PM, Jason Stover <jason....@gmail.com<mailto:jason....@gmail.com>> wrote:
Okay ... as a test, on the client try:

mount -t nfs4 server:/export /mnt

I know I've seen that "No such device" ... and I'm wanting to say it
was a 4 only server on a vers 3 mount request.

-J

> Export list for 192.168.13.11<http://192.168.13.11>:
> /opt 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
> /usr/local 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
> /var/chroots 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>
> /home 192.168.13.0/255.255.255.0<http://192.168.13.0/255.255.255.0>

>
> On execution node:
>
> root@D1P-HYDRAEX01 ~]# mount /homme
> mount: can't find /homme in /etc/fstab
> [root@D1P-HYDRAEX01 ~]# mount /home
> mount.nfs: No such device
> [root@D1P-HYDRAEX01 ~]# mount /usr/local
> mount.nfs: No such device
> [root@D1P-HYDRAEX01 ~]# mount /var/chroots
> mount.nfs: No such device
>

> On Wed, Sep 23, 2015 at 12:55 PM, Chu Ruilin <ruil...@gmail.com<mailto:ruil...@gmail.com>> wrote:
>>
>> This is what I do usually:
>>
>> wwinit ALL ## everything should be OK, including database, nfsd, dhcp,
>> bootstrap, etc, except the last step vnfs since I did not give CHROOTDIR
>>
>> wwmkchroot centos-7 /var/chroots/centos-7
>>
>> and optional step:
>> yum --tolerant --installroot /var/chroots/centos-7 -y install
>> stuff_you_want_to_put_in_vnfs
>>
>> wwvnfs --chroot /var/chroots/centos-7
>>
>> and then I do
>>
>> wwnodescan --netdev=eth0 --ipaddr=blablabla --netmask=255.255.0.0
>> --vnfs=centos-7 --bootstrap=`uname -r` --groups=compute c[001-999]
>>
>> this will do a stateless bootup for all compute nodes and record them to
>> ww. WW will automatically export /home, /usr/local, /opt and /var/chroots on
>> master nodes and mount them on compute nodes. I don't need to manually do
>> anything unless I want something else.
>>
>> If I want stateful provision, I install kernel and grub to chroot and
>> re-image the vnfs, set bootlocal in ww, reboot all compute nodes with pdsh,
>> and then unset bootlocal in ww, done.
>>
>> has been working for me every time, that's why I love ww so much
>>

>> On Wed, Sep 23, 2015 at 12:37 PM, Vince Forgetta <forg...@gmail.com<mailto:forg...@gmail.com>>

>> wrote:
>>>
>>> Thanks Jess.
>>>
>>> Yes, mount command I got from somewhere I cannot recall. Canonical
>>> command produces same error:
>>>
>>> [root@D1P-HYDRAEX01 ~]# mount -t nfsd nfsd /proc/fs/nfsd
>>> mount: unknown filesystem type 'nfsd'
>>>
>>> D1P-HYDRAEX01 (above) is wwexecution node i.e. provisioned node from
>>> wwmaster.
>>> D1P-HYDRAMS01 is wwmaster node.
>>> D1P-HYDRAFS01 is file server.
>>>
>>> From wwmaster (CentOS7) I can mount NFS share served by file server.
>>>
>>> From wwexecution node I am unable to mount NFS share served by file
>>> server.
>>>
>>> Info you requested plus some extra from the execution node:
>>>
>>> [root@D1P-HYDRAEX01 ~]# netstat -anpt |grep nfs (no output)
>>> [root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.10

>>> Export list for 192.168.13.10<http://192.168.13.10>:
>>> /mnt/mount1 192.168.13.10/24<http://192.168.13.10/24>
>>> /mnt/mount2 192.168.13.10/24<http://192.168.13.10/24>
>>> /mnt/mount3 192.168.13.10/24<http://192.168.13.10/24>
>>> /mnt/mount4 192.168.13.10/24<http://192.168.13.10/24>

>>>> 512-410-9690<tel:512-410-9690>
>>>>
>>>> On Wed, Sep 23, 2015 at 10:51 AM, Vince Forgetta <forg...@gmail.com<mailto:forg...@gmail.com>>

>>>>> an email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
>>>>> To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/aeeff7d5-5fa9-4765-876c-1c721d1aa304%40lbl.gov.
>>>>> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Warewulf" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send

>>>> an email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
>>>> To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

>>>> To view this discussion on the web visit
>>>> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAHBpPoVJ5PxOoU6X8aNWjFA6FjF5WHtUMkwrbQoJdgdSoC40JA%40mail.gmail.com.
>>>>
>>>> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Warewulf" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an

>>> email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
>>> To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

>>> To view this discussion on the web visit
>>> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsBzqnSuWq9g2e0USZ3p%3DXUEaz4wjsABjw6Z3RYaqp-yBg%40mail.gmail.com.
>>>
>>> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Warewulf" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
>> To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

>> To view this discussion on the web visit
>> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAC_eze8H_OHJz0S82Ecw_7DHCzHQ_nULNvGoFw5MVf3Q%3D-jHDA%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
> To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

> To view this discussion on the web visit
> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsDuV3dnfLyGTQbutpHPG1V29r%3DUo3U5vGJK_PX_5nfBww%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

--
You received this message because you are subscribed to the Google Groups "Warewulf" group.

To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov<mailto:warewulf%2Bunsu...@lbl.gov>.
To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt_9coKMetESJUQ386GQF3eUL5xAq4oBQ86W-i_rY7PW4w%40mail.gmail.com.
For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

--
You received this message because you are subscribed to the Google Groups "Warewulf" group.

To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov<mailto:warewulf+u...@lbl.gov>.
To post to this group, send email to ware...@lbl.gov<mailto:ware...@lbl.gov>.
To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsCw3Jy1Zg-WXeE6%3DyENOUy-M%2B3iwYK2bXK3O55TSXjx0g%40mail.gmail.com<https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsCw3Jy1Zg-WXeE6%3DyENOUy-M%2B3iwYK2bXK3O55TSXjx0g%40mail.gmail.com?utm_medium=email&utm_source=footer>.

Vince Forgetta

unread,

Sep 23, 2015, 2:13:35 PM9/23/15

to ware...@lbl.gov

dagnabbit!

[root@D1P-HYDRAEX01 ~]# mount -v -t nfs4 192.168.13.11:/opt /opt

mount.nfs4: timeout set for Wed Sep 23 07:07:57 2015

mount.nfs4: trying text-based options 'addr=192.168.13.11,clientaddr=192.168.13.12'

mount.nfs4: mount(2): Protocol not supported

mount.nfs4: Protocol not supported

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAPJtH1hQBCMryxBVy1EFzV%2BGs-4PRBNKnChtwGD%2BqikbJPm3dA%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 2:14:30 PM9/23/15

to ware...@lbl.gov

Also, is it expected that the kernel modules on the execution node are at a different directory level as compared to the master node?

On master:

[root@D1P-HYDRAMS01 ~]# find /lib/modules/ -name nfsd.ko

/lib/modules/3.10.0-229.el7.x86_64/kernel/fs/nfsd/nfsd.ko

/lib/modules/3.10.0-229.14.1.el7.x86_64/kernel/fs/nfsd/nfsd.ko

On execution node:

[root@D1P-HYDRAEX01 ~]# find /lib/modules/ -name nfsd.ko

/lib/modules/kernel/fs/nfsd/nfsd.ko

Related maybe, but "depmod" does not work as well. Again, not sure if relevant to provisioned node.

Jess Cannata

unread,

Sep 23, 2015, 3:00:16 PM9/23/15

to ware...@lbl.gov

Vince,

I quickly got NFS working on the server and client sides with Centos 7. This is outside of Warewulf and a very simple test export, but you should be able to troubleshoot your install. I had to disable the firewall on the server because by default it was blocking NFS. You would need to open up the NFS ports in the firewall if you plan to use it.

I hope that this helps you understand how NFS works.

Jess

*** Master/NFS Server ***

IP: 192.168.122.200

yum install nfs-utils.x86_64

[root@c7-m ~]# cat /etc/exports
/opt *(rw)

[root@c7-m ~]# systemctl enable nfs-server.service
ln -s '/usr/lib/systemd/system/nfs-server.service' '/etc/systemd/system/multi-user.target.wants/nfs-server.service'

[root@c7-m ~]# systemctl start nfs-server.service

[root@c7-m ~]# systemctl status nfs-server.service
nfs-server.service - NFS server and services
   Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled)
   Active: active (exited) since Wed 2015-09-23 13:46:44 CDT; 1s ago
Process: 2035 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
Process: 2034 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
Main PID: 2035 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/nfs-server.service

Sep 23 13:46:43 c7-m systemd[1]: Starting NFS server and services...
Sep 23 13:46:44 c7-m systemd[1]: Started NFS server and services.

[root@c7-m ~]# systemctl status firewalld.service
firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)
   Active: active (running) since Wed 2015-09-23 13:44:59 CDT; 6min ago
Main PID: 327 (firewalld)
   CGroup: /system.slice/firewalld.service
           └─327 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Sep 23 13:44:56 c7-m systemd[1]: Starting firewalld - dynamic firewall daemon...
Sep 23 13:44:59 c7-m systemd[1]: Started firewalld - dynamic firewall daemon.

[root@c7-m ~]# systemctl stop firewalld.service

[root@c7-m ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

[root@c7-m ~]# ss -anpt | grep 2049
LISTEN     0      64                        *:2049                     *:*
ESTAB      0      0           192.168.122.200:2049       192.168.122.239:757

*** Client ***

yum install nfs-utils.x86_64

[root@c7-1 ~]# mount -t nfs 192.168.122.200:/opt /mnt

[root@c7-1 ~]# df -h /mnt
Filesystem            Size Used Avail Use% Mounted on
192.168.122.200:/opt 7.2G 897M 6.4G 13% /mnt

[root@c7-1 ~]# lsmod | grep -i nfs
nfsv4                 474203 1
dns_resolver           13140 1 nfsv4
nfs                   241261 2 nfsv4
lockd                  93977 1 nfs
sunrpc                293453 12 nfs,rpcsec_gss_krb5,auth_rpcgss,lockd,nfsv4
fscache                57813 2 nfs,nfsv4

--
Jess Cannata
R Systems
512-410-9690

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsCA9zzVURM5-9S%3DjHhQpfkU9C4n-YbMRb%3DrzGp8pzpy6A%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 3:36:09 PM9/23/15

to ware...@lbl.gov

Thanks, Jess. I can confirm that the server setup is as you describe.

Only issue is that I cannot use yum from the execution node as it does not have internet connection yet.

However, using yum from master to install into chroot appears to show nfs-utils is installed:

[root@D1P-HYDRAMS01 vforget]# yum --tolerant --installroot /var/chroots/hydraex-centos7.new -y install nfs-utils

Loaded plugins: fastestmirror

Loading mirror speeds from cached hostfile

* base: centos.mirror.gtcomm.net

* epel: mirrors.mit.edu

* extras: centos.mirrors.atwab.net

* updates: centos.mirrors.atwab.net

Package 1:nfs-utils-1.3.0-0.8.el7.x86_64 already installed and latest version

Nothing to do

On execution node appears that nfs tools are there:

[root@D1P-HYDRAMS01 vforget]# ls /sbin/nfs*

/sbin/nfsdcltrack /sbin/nfsidmap /sbin/nfsiostat /sbin/nfsstat

I can start rpcbind, but cannot start nfs:

[root@D1P-HYDRAEX01 ~]# systemctl restart rpcbind

[root@D1P-HYDRAEX01 ~]# systemctl start nfs

A dependency job for nfs-server.service failed. See 'journalctl -xn' for details.

I may just try and remove and re-compile warewulf on the master and re-build vnfs, etc. If this fails, I may just re-install the entire master node from scratch.

V

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAHBpPoW0bxsrdudVVDQZYE1L235nETMv%2BgoxXU%3DZtz1n1i44Lw%40mail.gmail.com.

Jess Cannata

unread,

Sep 23, 2015, 3:43:21 PM9/23/15

to ware...@lbl.gov

Why are you trying to start NFS Server on the compute node?

You should only need to mount the NFS share. See my client instructions for a guide.

--
Jess Cannata
R Systems
512-410-9690

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAEO3gsAF2TWa46_A6GUzTgMxm3NPMNOZS-keo9igq3kk2%3DvxWQ%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 3:57:21 PM9/23/15

to ware...@lbl.gov

OK. I see know. But even if I do not attempt to start the server ("systemctl start nfs") the issue persists.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAHBpPoW3%2BN17Tonf_Ek86cj62sOetjVn0DVfse2Wg_fSpAr6rQ%40mail.gmail.com.

Vince Forgetta

unread,

Sep 23, 2015, 3:58:45 PM9/23/15

to ware...@lbl.gov

From clean reboot of provisioned node:

[root@D1P-HYDRAEX01 ~]# showmount -e 192.168.13.11

Export list for 192.168.13.11:

/opt 192.168.13.0/255.255.255.0

/usr/local 192.168.13.0/255.255.255.0

/var/chroots 192.168.13.0/255.255.255.0

/home 192.168.13.0/255.255.255.0

[root@D1P-HYDRAEX01 ~]# mount -t nfs4 192.168.13.10:/opt /opt

mount.nfs4: No such device

[root@D1P-HYDRAEX01 ~]# mount -t nfs 192.168.13.10:/opt /opt

mount.nfs: No such device

Vince Forgetta

unread,

Sep 23, 2015, 8:54:43 PM9/23/15

to ware...@lbl.gov

I think I found the source of the problem. Seems that the kernel modules directory being provisioned in screwed up somehow. A recap, here is kernel module directory layout on the master vs provisioned node:

[root@D1P-HYDRAMS01 ~]# find /lib/modules/ -name nfsd.ko

/lib/modules/3.10.0-229.el7.x86_64/kernel/fs/nfsd/nfsd.ko

/lib/modules/3.10.0-229.14.1.el7.x86_64/kernel/fs/nfsd/nfsd.ko

On execution node:

[root@D1P-HYDRAEX01 ~]# find /lib/modules/ -name nfsd.ko

/lib/modules/kernel/fs/nfsd/nfsd.ko

Note missing kernel version number. When I "fix" this by creating this directory and adding symlink back to the kernel directory the nfs kernel modules load properly. Now I get a "protocal not supported" error from mount. At this point, I will stop trying to fix this as I assume something is severely wrong with how the kernel is getting bootstrapped. I will just reinstall the entire master node and WW3.7.

Thanks to everyone for your help, and sorry for the bother.

V

Vince Forgetta

unread,

Sep 28, 2015, 2:57:08 PM9/28/15

to ware...@lbl.gov

Hi all,

I have reinstalled the wwmaster node from scratch. See here for full log of what was done:

https://www.dropbox.com/s/38b3sqr6jbdd09q/Hydra-Master-CentOS7.pdf?dl=0

The problem remains in that I cannot mount any of the NFS exports from the wwmaster onto the provisioned node.

At this point, I will mull this over for some time, but will consider fallback to WW3.6 shortly as I need to get this setup within a short time period.

Thanks to all for the past help, and I hope you see something I did wrong in the current setup I have posted.