Getting Base Node configuration ................ on BIOS

[Arash Nemati Hayati]

I am trying to add a new node to the cluster using a vfns. I am stuck on the part where the node is trying to get the base node configurations. 

What should I check for to diagnose the issue?

Thanks

[Jason Stover]

Hi Arash,

Make sure that you don't have a firewall on the provisioner, or if
you do that Port 80 is open to the cluster network.

Hanging at that part, is usually either:
1) Firewall blocking port 80
2) Web server isn't running

-J

[Arash Nemati Hayati]

Hi Jason,

Thanks for your help. I opened the port on both internal and public zone networks. The problem still occurred. I even disabled the firewall on the provisioner but it didn't fix it. 

ping and wget both works as chroot on the image.

Can this be related to the version of Warewolf? I guess I am running out of ideas...

[Jason Stover]

Hi Arash,

Can you send a picture of the screen when you're getting the error?

Also, in the PXE file, can you add: wwdebug=5 -- to the kernel arguments.

When it's trying to get the "node configuration," what the node is
doing is making a HTTP request to the provisioner. So on the
provisioner you *should* be seeing a request similar to:

/WW/nodeconfig.pl?hwaddr=[MAC]

If you are seeing the request, but the node isn't getting a
response, I would verify the routing table on the provisioner is setup
right to talk to the IP's you're trying to.

-J

On Wed, Sep 12, 2018 at 1:32 PM Arash Nemati Hayati

[Arash Nemati Hayati]

Please find the error in the attachment. Would you please point me to where I should look for the PXE file and WW on the provisioner?

[Jason Stover]

The PXE file would be under...

/var/lib/tftpboot/warewulf/01-[MAC] (IIRC)

And in the normal HTTP logs, so something like: /var/log/httpd/access_log

-J

[Arash Nemati Hayati]

Ok, looking at the http logs, I don't see the request from that mac address. So the client is not sending http request to the provisioner..

[John Hearns]

Arash, a rather stupid question from me. On your network switch, have
you defined the port to which the new compute node is attached as an
Edge Port?
If not defined as an Edge Port the port can take a long time to come
up, sue to spanning tree being active. You then can miss request being
made.
I do not think this is the problem, but please check.
This is a general piece of advice for everyone though.
On Wed, 12 Sep 2018 at 20:27, Arash Nemati Hayati

<a.nemat...@gmail.com> wrote:
>
> Ok, looking at the http logs, I don't see the request from that mac address. So the client is not sending http request to the provisioner..
>
> On Wednesday, September 12, 2018 at 3:15:11 PM UTC-4, Jason Stover wrote:
>>
>> The PXE file would be under...
>>
>> /var/lib/tftpboot/warewulf/01-[MAC] (IIRC)
>>
>> And in the normal HTTP logs, so something like: /var/log/httpd/access_log
>>
>> -J
>>
>> On Wed, Sep 12, 2018 at 2:08 PM Arash Nemati Hayati
>> <a.nemat...@gmail.com> wrote:
>> >
>> > Please find the error in the attachment. Would you please point me to where I should look for the PXE file and WW on the provisioner?
>> >
>> >
>

> --
> You received this message because you are subscribed to the Google Groups "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
> To post to this group, send email to ware...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/9aee70df-00bc-444e-909f-72b12bb03028%40lbl.gov.
> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

[Arash Nemati Hayati]

Hi John,

Thanks for the note. The ports are set with spanning-tree portfast, which eliminates such issues. 
The master and clients nodes are set on a private VLAN. The clients cannot communicate with other VLANs by design. The clients talk only to their master node, which can talk to the rest of the network and beyond.  

Not sure why the http request is not sent to the master's node...

[Arash Nemati Hayati]

Jason,
Kernel ebugging message attached here.

[Jason Stover]

So, the provisioner can be pinged. From the Debug shell, can you try running:

/warewulf/bin/wwgetnodeconfig

I think that's the path... or maybe
/warewulf/transport-http/wwgetnodeconfig. wwgetnodeconfig may just be
in the path. Anyway, that's the script that should be trying to run
where it's hanging. As I previously mentioned, what it's going to do
is a wget to the provisioner, like:

wget -q -O - "http://$master/WW/nodeconfig?hwaddr=[MAC]"

Where in your case, $master will be set to 10.2.1.1. So the URL it's
trying to get is "/WW/nodeconfig" ... without the ".pl" extension.

As an aside, what is the MTU on the provisioner device, and is the MTU
different on the nodes? I have seen bad behavior when the MTU's didn't
match.

And from a previous comment that I can't remember if I answered. But
yes, the 3.6 release is... quite old now. It should still work, but
it's getting closer to the point that I would highly suggest updating
if you can. As mentioned, on the Github page there is a 3.8.1 release.

-J


On Wed, Sep 12, 2018 at 4:15 PM Arash Nemati Hayati

<a.nemat...@gmail.com> wrote:
>
> Jason,
> Kernel ebugging message attached here.
>
> On Wednesday, September 12, 2018 at 3:15:11 PM UTC-4, Jason Stover wrote:
> > The PXE file would be under...
> >
> > /var/lib/tftpboot/warewulf/01-[MAC] (IIRC)
> >
> > And in the normal HTTP logs, so something like: /var/log/httpd/access_log
> >
> > -J
> >
> > On Wed, Sep 12, 2018 at 2:08 PM Arash Nemati Hayati
> > <a.nemat...@gmail.com> wrote:
> > >
> > > Please find the error in the attachment. Would you please point me to where I should look for the PXE file and WW on the provisioner?
> > >
> > >
>

> --
> You received this message because you are subscribed to the Google Groups "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
> To post to this group, send email to ware...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/1cd1af0a-7bfc-49ab-9a6d-722e98016e5a%40lbl.gov.

[Ryan Novosielski]

My experience here is that, at least with some of the older versions of Warewulf, there were some lurking bugs where there were ways to modify aspects of nodes that somehow wound up not generating all of the right config stuff, and that I would typically find them while making rookie mistakes. I never got to the bottom of it, but my feeling was that it would happen when you modified an existing definition but not creating a new one or something like that. A made-up example would be that you’d be fine configuring a new node, but if you accidentally configured a new node with the wrong network interface that is not one of the ones that’s used for DHCP and then renamed it, some of the necessary config might be missed. Since I was in a hurry, I never got to the bottom of it and just recreated the definition for the node. When I’d see that scenario, I’d generally get a 404 in the httpd log for something that should have been present.

HTH,
=R

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt9uUi%3De9UW6iGmU6c%2B1faKMknnH%2BbajkiywkNm0VmcuBA%40mail.gmail.com.

> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

--
____
|| \\UTGERS, |---------------------------*O*---------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Research Computing - MSB C630, Newark
`'

[Arash Nemati Hayati]

Ryan, 

Thank you very much for your response. Did you get warewolf working on Centos, Redhat 7? What version did you use? 

Thanks,

Arash

[Meij, Henk]

Easily deployed via http://openhpc.community/downloads/ using the centos7/warewulf recipe.

-Henk

---

From: Arash Nemati Hayati <a.nemat...@gmail.com>
Sent: Thursday, September 13, 2018 9:57:49 AM
To: Warewulf
Cc: novo...@rutgers.edu
Subject: Re: [Warewulf] Getting Base Node configuration ................ on BIOS

 

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/cae9c30b-e180-4d6e-9924-67f703fe8b9d%40lbl.gov.

[Arash Nemati Hayati]

Jason,

It hangs at wwgetnodeconfig. Attached is the stage it stops in debugging mode.

mtu is 1500 on both master and client nodes. 

[Arash Nemati Hayati]

Henk,

Thanks for your suggestions. I followed this recipe. It depends on mariadb for warewolf provisoning. mariadb is obsoleted in yum repo and it has been replaced by mysql. May I ask when did you deploy the recipe? 

[Meij, Henk]

You got that in reverse. When Oracle acquired MySQL, MariaDB was born as the open source flavor.

My notes are located here...deployed test instance in August 2018

https://dokuwiki.wesleyan.edu/doku.php?id=cluster:170

-Henk

[root@ohpc0-test ~]# yum repolist
repo id                       repo name                                                   status
!OpenHPC                      OpenHPC-1.3 - Base                                             821
!OpenHPC-updates              OpenHPC-1.3 - Updates                                        1,322
!base/7/x86_64                CentOS-7 - Base                                              9,911
*!epel/x86_64                 Extra Packages for Enterprise Linux 7 - x86_64              12,647
!extras/7/x86_64              CentOS-7 - Extras                                              370
!updates/7/x86_64             CentOS-7 - Updates                                           1,054
repolist: 26,125

[root@ohpc0-test ~]# rpm -qa | grep ^maria
mariadb-server-5.5.56-2.el7.x86_64
mariadb-libs-5.5.56-2.el7.x86_64
mariadb-5.5.56-2.el7.x86_64

---

From: Arash Nemati Hayati <a.nemat...@gmail.com>

Sent: Thursday, September 13, 2018 10:20:54 AM

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/2974e1ff-e9f1-467c-806d-79e5197ecc52%40lbl.gov.

[Arash Nemati Hayati]

Interesting. Thanks for the notes. I will look into that. 

mysql-community was part of my repo lists on yum and the obsolete notes were from there I think:

Package 1:mariadb-5.5.60-1.el7_5.x86_64 is obsoleted by mysql-community-client-5.6.41-2.el7.x86_64 which is already installed

[Ryan Novosielski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This sounds to me like you may have installed a package directly from
MySQL, or have enabled a MySQL repo, whose package obsoletes mariadb:

[root@master ~]# yum list available | grep mysql-community
[root@master ~]#

> ----------------------------------------------------------------------
- --
>
>
*From:* Arash Nemati Hayati <a.nemat...@gmail.com <javascript:>>
> *Sent:* Thursday, September 13, 2018 10:20:54 AM *To:* Warewulf
> *Cc:* novo...@rutgers.edu <javascript:> *Subject:* Re: [Warewulf]

> Getting Base Node configuration ................ on BIOS
>
> Henk,
>
> Thanks for your suggestions. I followed this recipe. It depends on
> mariadb for warewolf provisoning. mariadb is obsoleted in yum repo
> and it has been replaced by mysql. May I ask when did you deploy
> the recipe?
>
> On Thursday, September 13, 2018 at 10:11:32 AM UTC-4, Meij, Henk
> wrote:
>
> Easily deployed via http://openhpc.community/downloads/
> <http://openhpc.community/downloads/> using the centos7/warewulf
> recipe.
>
>
> -Henk
>
>

> ----------------------------------------------------------------------
- --
>
>
*From:* Arash Nemati Hayati <a.nemat...@gmail.com>
> *Sent:* Thursday, September 13, 2018 9:57:49 AM *To:* Warewulf
> *Cc:* novo...@rutgers.edu *Subject:* Re: [Warewulf] Getting Base

<https://groups.google.com/a/lbl.gov/d/msgid/warewulf/1cd1af0a-7bfc-49ab
- -9a6d-722e98016e5a%40lbl.gov>.

>
>>> For more options, visit
> https://groups.google.com/a/lbl.gov/d/optout

> <https://groups.google.com/a/lbl.gov/d/optout>.

>>
>> -- You received this message because you are subscribed to
> the Google Groups "Warewulf" group.
>> To unsubscribe from this group and stop receiving emails
> from it, send an email to warewulf+u...@lbl.gov.
>> To post to this group, send email to ware...@lbl.gov. To view
>> this discussion on the web visit

> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt9uUi%3De9U
W6iGmU6c%2B1faKMknnH%2BbajkiywkNm0VmcuBA%40mail.gmail.com
>
>
<https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt9uUi%3De9UW
6iGmU6c%2B1faKMknnH%2BbajkiywkNm0VmcuBA%40mail.gmail.com>.

>
>> For more options, visit
> https://groups.google.com/a/lbl.gov/d/optout

> <https://groups.google.com/a/lbl.gov/d/optout>.

>
> -- ____ || \\UTGERS,
> |---------------------------*O*--------------------------- ||_//
> the State | Ryan Novosielski - novo...@rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~
> RBHS Campus || \\ of NJ | Office of Advanced Research
> Computing - MSB C630, Newark `'
>
> -- You received this message because you are subscribed to the
> Google Groups "Warewulf" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to
> warewulf+u...@lbl.gov. To post to this group, send email to
> ware...@lbl.gov. To view this discussion on the web visit
> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/cae9c30b-e180-4d6
e-9924-67f703fe8b9d%40lbl.gov
>
>

<https://groups.google.com/a/lbl.gov/d/msgid/warewulf/cae9c30b-e180-4d6e
- -9924-67f703fe8b9d%40lbl.gov?utm_medium=email&utm_source=footer>.

> For more options, visit
> https://groups.google.com/a/lbl.gov/d/optout

> <https://groups.google.com/a/lbl.gov/d/optout>.

>
> -- You received this message because you are subscribed to the
> Google Groups "Warewulf" group. To unsubscribe from this group and
> stop receiving emails from it, send an email to

> warewulf+u...@lbl.gov <javascript:>. To post to this group, send
> email to ware...@lbl.gov <javascript:>. To view this discussion on
> the web visit
> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/2974e1ff-e9f1-467
c-806d-79e5197ecc52%40lbl.gov
>
>
<https://groups.google.com/a/lbl.gov/d/msgid/warewulf/2974e1ff-e9f1-467c
- -806d-79e5197ecc52%40lbl.gov?utm_medium=email&utm_source=footer>.

> For more options, visit
> https://groups.google.com/a/lbl.gov/d/optout

> <https://groups.google.com/a/lbl.gov/d/optout>.
>

- --
____
|| \\UTGERS, |----------------------*O*------------------------

||_// the State | Ryan Novosielski - novo...@rutgers.edu

|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark
`'


-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAluakugACgkQmb+gadEcsb5y0ACgicOv2DF9fRMtMnM0rgGngkiR
MjMAoNctG44IaeA1M8/kJLiodk5P/1Xc
=7LHv
-----END PGP SIGNATURE-----

[Arash Nemati Hayati]

I upgraded warewolf to 3.8. Now I get "Operation not permitted" where the client tries to do http://.../WW/bootstrap/... 

I checked the permissions on the source file but it didn't help. Any suggestions?

[Jason Stover]

Hi Arash,

Did you restart the HTTP server after updating to use the new
warewulf config? From the iPXE error you posted on the OpenHPC list,
that "Operation not permitted" is generally when the HTTP server
returns a 403 Forbidden error.

-J

On Thu, Sep 13, 2018 at 3:00 PM Arash Nemati Hayati

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/4e6e7096-62f1-462d-979d-7d2335a83e42%40lbl.gov.
> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

[Ryan Novosielski]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

There's also a message on there suggesting it's SELinux. You might
check through that mailing list to see some of the suggestions.

BTW, It /is/ possible to run the head/master/SMS node with SELinux
enabled, if you're comfortable doing so (we do), but you've got to
have some understanding how it works/what to allow.

- --

>> You received this message because you are subscribed to the
>> Google Groups "Warewulf" group. To unsubscribe from this group
>> and stop receiving emails from it, send an email to
>> warewulf+u...@lbl.gov. To post to this group, send email
>> to ware...@lbl.gov. To view this discussion on the web visit

>> https://groups.google.com/a/lbl.gov/d/msgid/warewulf/4e6e7096-62f1-46

2d-979d-7d2335a83e42%40lbl.gov.
>>
>>
For more options, visit https://groups.google.com/a/lbl.gov/d/optout.
>

- --
____
|| \\UTGERS, |----------------------*O*------------------------
||_// the State | Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 ~*~ RBHS Campus
|| \\ of NJ | Office of Advanced Res. Comp. - MSB C630, Newark
`'
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAluazbsACgkQmb+gadEcsb4qkQCgwYvibpkhBDxl4drLSwp3tcXB
2YkAnjcrqZHAly7Jfy5iFS8NVqAwQHPv
=jle4
-----END PGP SIGNATURE-----

[Arash Nemati Hayati]

Hi Jason,

I did restart http server after the update. yes I see 403 Forbidden error even from the master's node. 

Hi Ryan,

I disabled SELINUX for now but the same thing happens.

Is warewolf supposed to create the bootstrap directory in /srv/wareolf?

when I look inside /srv/warewolf i just see ipxe folder and initramfs.

http://10.2.1.1/WWipxe.... ok
Now booting ...
http://10.2.1.1/WW/bootstrap/x86_64/2/initfs.gz ... Operation not permitted (http://ipxe.org/410c613c)

[Arash Nemati Hayati]

Ok, I was able to get stateless and stateful installation on a compute node. However, after making some changes to the vnfs and restarting the node,

It gets stuck on files (picture attached). I recreated the vnfs image but the issue wasn't resolved. 

Please let me know if you have seen this before or you have any suggestions.. I really appreciate it.

Starting the provision handler:

...

*kernelmodules .... OK

*files ---> get stuck here

[Jason Stover]

Hi Arash,

On the provisioner try running:

wwsh file sync

The "files" section there pulls any files defined (wwsh file list ,
and, wwsh provision print [node] | grep 'FILES'), and places it onto
the node. For example, by default there's a file named "dynamic_hosts"
that is a generated list of all the hosts and placed at /etc/hosts on
nodes. Again, you should see "/WW/file?hwaddr=" requests in the Apache
logs.

-J

On Tue, Sep 18, 2018 at 12:47 PM Arash Nemati Hayati

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/10cccc9c-cdd1-4d8d-a7d9-5b5813b2b97c%40lbl.gov.

[Arash Nemati Hayati]

wow! I see.. It worked! Thanks a lot. 

Isn't it a good practice to to this sync somewhere during the procedure of provisioning a node?

[Jason Stover]

Hi Arash,

> during the procedure of provisioning a node

No. Files are separate from the provisioning step. What happened was
there could have been a failed DB read, and caused it to hang. Doing
the `wwsh file sync` causes it to resync _every_ file it knows about.
If you knew exactly what file it was hanging on, you probably could
have done a: wwsh file sync [file name] -- and that would have
resync'd just that file, and probably would have fixed it just the
same.

The files are used during provisioning (and on live nodes --
wwgetfiles cron), but they aren't dependent. You could remove all
files from a node if you wanted. What would happen in that case is
that Warewulf would ask for the list of files, get an empty set and
continue on.

-J

[Arash Nemati Hayati]

So warewulf does not find the vnfs on reboot:

getvnfs .... Running

fetching gpu-centos7.5 ... Retrying...

I see the vnfs file and path in the list on the master node. Any ideas?

gpu-centos7.5        533.0      x86_64     /opt/ohpc/admin/images/gpu-centos7.5

#### gpu-6-11 #################################################################

       gpu-6-11: BOOTSTRAP        = 3.10.0-862.11.6.el7.x86_64

       gpu-6-11: VNFS             = gpu-centos7.5

       gpu-6-11: FILES            = group,munge.key,network,passwd,shadow,slurm.conf

       gpu-6-11: PRESHELL         = FALSE

       gpu-6-11: POSTSHELL        = FALSE

       gpu-6-11: CONSOLE          = UNDEF

       gpu-6-11: PXELINUX         = UNDEF

       gpu-6-11: SELINUX          = DISABLED

       gpu-6-11: KARGS            = "net.ifnames=0 biosdevname=0 quiet"

       gpu-6-11: FS               = "select /dev/sda,mklabel gpt,mkpart primary 1MiB 3MiB,mkpart primary ext4 3MiB 513MiB,mkpart primary linux-swap 513MiB 50%,mkpart primary ext4 50% 100%,name 1 grub,name 2 boot,name 3 swap,name 4 root,set 1 bios_grub on,set 2 boot on,mkfs 2 ext4 -L boot,mkfs 3 swap,mkfs 4 ext4 -L root,fstab 4 / ext4 defaults 0 0,fstab 2 /boot ext4 defaults 0 0,fstab 3 swap swap defaults 0 0"

       gpu-6-11: BOOTLOADER       = sda

       gpu-6-11: BOOTLOCAL        = FALSE

Thanks,

[Ryan Novosielski]

It must be be attempting to download something from somewhere. You’ll need to do some of the standard troubleshooting stuff. Is there anything in the HTTPD logs, if not, can you confirm where it’s attempting to contact somehow. I’ll say that I’ve not bothered much with stateful images. You might try enabling the pre or post shell if you find you need to to get more troubleshooting in.

Also, if you are using RDMA, not apropos of Warewulf, but avoid kernel 3.10.0-862.11.6; it will not work (for RDMA; you won’t notice otherwise).

> --
> You received this message because you are subscribed to the Google Groups "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
> To post to this group, send email to ware...@lbl.gov.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/79e74994-5abd-47b8-a667-eb1fe565946d%40lbl.gov.

> For more options, visit https://groups.google.com/a/lbl.gov/d/optout.

> <20180925_125420.jpg>

[Arash Nemati Hayati]

It's trying to find the cache directory for the image but it cannot??

http error:

[Tue Sep 25 14:23:06.108343 2018] [:error] [pid 121793] mkdir /var/tmp/warewulf_cache: No such file or directory at /usr/libexec/warewulf/cgi-bin/vnfs.pl line 88.\n

                            if (! -d "$vnfs_cachedir/$vnfs_name") {

                                mkpath("$vnfs_cachedir/$vnfs_name");

                            }

10.2.255.48 - - [25/Sep/2018:14:28:34 -0400] "GET /WW/vnfs?hwaddr=0c:c4:7a:6e:7b:0c HTTP/1.1" 500 532 "-" "Wget"

[Rodrigo Freitas]

Did you ever got a solution on this? I'm facing the same problem