Failed to fetch kickstart error

[Mike Walters]

Hi,

I finally got stacki frontend setup and seems to be working as it should. I however am having issues kickstarting a backend node from discovery and spreadsheet. I figured out the network issue for hyper-v requiring you to have a legacy network adapter for PXE. Although I have the network communicating I am still having issues kickstarting a test VM. I will try to post all the needed info here:

Frontend: 192.168.55.141

Backend: 192.168.55.142

I loaded a spreadsheet with the legacy MAC address set into stacki frontend.  Spreadsheet looks as so:

NAME,INTERFACE HOSTNAME,DEFAULT,APPLIANCE,RACK,RANK,IP,MAC,INTERFACE,NETWORK,CHANNEL,OPTIONS,VLAN,INSTALLACTION,RUNACTION,GROUPS,BOX
backendnode01,backendnode01,TRUE,backend,1,1,192.168.55.142,00:15:5D:0E:FC:50,eth0,private,,,,install,os,,default

Error on VM after startup (Server keeps retrying the connection):

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.55.141/install/sbin/profile.cgi

Frontend Log: /var/log/httpd/ssl_request_log

[30/Oct/2017:23:54:15 -0400] 192.168.55.142 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 25

I also tried disabling iptables, but did not help. 

Server Info:

[root@stacki ~]# stack list network

NETWORK ADDRESS      MASK          GATEWAY      MTU   ZONE DNS   PXE

private 192.168.55.0 255.255.255.0 192.168.55.1 1500  ---- False True

[root@stacki ~]# stack list host

HOST          RACK RANK CPUS APPLIANCE BOX     ENVIRONMENT RUNACTION INSTALLACTION

stacki        0    0    2    frontend  default ----------- os        install

backendnode01 1    1    1    backend   default ----------- os        install

[root@stacki ~]# stack list pallet

NAME   VERSION              RELEASE ARCH   OS     BOXES

os     7.3_11122da          7.x     x86_64 redhat

stacki 4.0_20170414_c4aff2a 7.x     x86_64 redhat default

CentOS 7                    7.x     x86_64 redhat default

[root@stacki ~]# stack list host boot

HOST          ACTION

stacki

backendnode01 install

[Joe Kaiser]

Check the md5sum of your iso media.

Check a "stack list host profile backendnode01" 

And did you set "nukedisks" to true for the first install?

Thanks,

Joe

--
You received this message because you are subscribed to the Google Groups "stacki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Walters]

Hi Joe,

1. I guess you are talking about the md5sum for the Frontend stackios installation?  

SHA1 hash of file stackios-4.0_c4aff2a-7.x.x86_64.disk1.iso:

04659d15b7e205f27838341cd045c2837ad87f66

2. What exactly do you want me to check in the stack list host profile?  The response from the command displays a very long output.

3. I just tried to set to nukedisks, but still having same error. 

[root@stacki ~]# stack set host boot backendnode01 attr=nukedisks action=install

-- 

You received this message because you are subscribed to the Google Groups "stacki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+un...@googlegroups.com.

[Anoop Rajendra]

What does /var/log/httpd/ssl_error_log say for this request?

-a

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.

[Mike Walters]

Hi Anoop,

That log file is empty. :(

I can see the server is trying to connect:

[root@stacki ~]# ss -an -t

TIME-WAIT  0      0                    ::ffff:192.168.55.141:443                              ::ffff:192.168.55.142:1

 

 

[Anoop Rajendra]

Is there a ssl_access_log perhaps?

--

[Mike Walters]

Hi Anoop,

Yes, there is entries in the ssl_access_log. Many lines like this:

192.168.55.142 - - [31/Oct/2017:15:17:07 -0400] "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 503 25

192.168.55.142 - - [31/Oct/2017:15:17:11 -0400] "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 503 25

192.168.55.142 - - [31/Oct/2017:15:17:13 -0400] "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 503 25

 

[Mike Walters]

Also here is my storage configuration just in case:

[root@stacki stacki]#  stack list storage partition

SCOPE         DEVICE PARTID MOUNTPOINT SIZE   FSTYPE OPTIONS

global        sda    1      /          30000  ext4   -------

global        sda    2      /var       30000  ext4   -------

global        sda    3      swap       32768  swap   -------

global        sda    4      /scratch   0      xfs    -------

backendnode01 sda    1      /boot      500    ext3   -------

backendnode01 sda    2      swap       32768  swap   -------

backendnode01 sda    3      /          30000  ext4   -------

backendnode01 sda    4      /var       20000  ext4   --grow

backendnode01 sda    5      /home      5000   ext4   -------

backendnode01 sda    6      /tmp       5000   ext4   -------

backendnode01 sda    7      /var/log   5000   ext4   -------

 

I am running into a lot of errors, should I be concerned? I am tempted to use Chef, but I like the simplicity of Stacki. I'm also hearing many people asking about Stacki 5.  Will that be dramatically different?

[Joe Kaiser]

stacki 4.0 for centos 7.x? 

You need a biosboot partition in your csv or none of it will work. 

,sda,biosboot,1,biosboot,

Thanks,

Joe

[Anoop Rajendra]

The above error indicates that there seems to be an httpd timeout.

To fix this, run

echo 64 > /var/tmp/profile.semaphore

Restart the apache server, and try again.

-Anoop

[Mike Walters]

Okay, I have updated the spreadsheet to include the biosboot:

[root@stacki ~]# stack list storage partition

SCOPE         DEVICE PARTID MOUNTPOINT SIZE   FSTYPE   OPTIONS

global        sda    1      biosboot   1      biosboot -------

global        sda    2      /          30000  ext4     -------

global        sda    3      /var       30000  ext4     -------

global        sda    4      swap       32768  swap     -------

global        sda    5      /scratch   0      xfs      -------

backendnode01 sda    1      biosboot   1      biosboot -------

backendnode01 sda    2      /boot      500    ext3     -------

backendnode01 sda    3      swap       16384  swap     -------

backendnode01 sda    4      /          30000  ext4     -------

backendnode01 sda    5      /var       20000  ext4     --grow

backendnode01 sda    6      /home      5000   ext4     -------

backendnode01 sda    7      /tmp       5000   ext4     -------

backendnode01 sda    8      /var/log   5000   ext4     -------

I also ran the command provided by Anoop.

The kickstart seems to start now, but I am seeing a different issues. The screen moves very fast with errors, here is a screenshot during this:

Here is how far the httpd logs get this time around:

==> /var/log/httpd/ssl_access_log <==
192.168.55.142 - - [01/Nov/2017:13:52:45 -0400] "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 200 81513

==> /var/log/httpd/ssl_request_log <==

[01/Nov/2017:13:52:45 -0400] 192.168.55.142 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/profile.cgi?os=redhat&arch=x86_64&np=1 HTTP/1.1" 81513

==> /var/log/httpd/access_log <==

192.168.55.142 - - [01/Nov/2017:13:52:50 -0400] "GET /install/pallets/stacki/4.0_20170414_c4aff2a/7.x/redhat/x86_64/.treeinfo HTTP/1.1" 404 269 "-" "-"

192.168.55.142 - - [01/Nov/2017:13:52:50 -0400] "GET /install/pallets/stacki/4.0_20170414_c4aff2a/7.x/redhat/x86_64/LiveOS/squashfs.img HTTP/1.1" 200 332746752 "-" "-"

192.168.55.142 - - [01/Nov/2017:13:53:06 -0400] "GET /install/pallets/stacki/4.0_20170414_c4aff2a/7.x/redhat/x86_64/images/updates.img HTTP/1.1" 200 120160256 "-" "-"

Any other thoughts? Am I doing something wrong, I thought this was a simple test for Stacki, but I cannot get past the first use case :( 

Also, just so you know the VM has a 100GB OS Drive attached to it.  I made sure to lower the partition dramatically to be sure that wasn't the cause. 

 

[Anoop Rajendra]

Your backend has too little memory. Raise it up to the minimum requirements.

--

[Mike Walters]

I have it set to the minimum as specified on the backend page. 

3072MB ~ 3GB 

I set the VM to have 4GB of RAM now the OS installation has begun :) 

Error Note:  I had to restart httpd on Frontend node to get the server to grab the kickstart file. It was having same error as above before restarting service:

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.55.141/install/sbin/profile.cgi

Will Stacki 5.0 support lower RAM backend servers?  We currently use Spacewalk here to create all of our VM's and Physical servers. Some VM's are only running 2GB of RAM.

[Anoop Rajendra]

We're looking into shrinking the installer. It's an unknown at this point if we can. If we can shrink it, then yes - Stacki (both frontend and backend) can run on less that 4 GB of memory. If not, then not.

[Joe Kaiser]

If you add inst.ks to your bootaction, you can reduce the memory required by the backend install. 

We are considering doing that by default in the next release.

Well, I am, I haven't talked to anyone else about that yet. 

Thanks,

Joe

[Kellen O'Connor]

This post has helped me troubleshoot my setup pretty well. 

To be clear, what exactly fixed this issue:

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.55.141/install/sbin/profile.cgi

I've attempted to change the bootaction, but it doesn't seem to resolve the issue.

On Thursday, November 2, 2017 at 11:59:45 AM UTC-4, Joe Kaiser wrote:

If you add inst.ks to your bootaction, you can reduce the memory required by the backend install. 

We are considering doing that by default in the next release.

Well, I am, I haven't talked to anyone else about that yet. 

Thanks,

Joe

On Wed, Nov 1, 2017 at 11:35 AM, Anoop Rajendra <anoop.r...@gmail.com> wrote:

On Wed, Nov 1, 2017 at 11:31 AM, Mike Walters <wmi...@gmail.com> wrote:

I have it set to the minimum as specified on the backend page. 

3072MB ~ 3GB 

I set the VM to have 4GB of RAM now the OS installation has begun :) 

Error Note:  I had to restart httpd on Frontend node to get the server to grab the kickstart file. It was having same error as above before restarting service:

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.55.141/install/sbin/profile.cgi

Will Stacki 5.0 support lower RAM backend servers?  We currently use Spacewalk here to create all of our VM's and Physical servers. Some VM's are only running 2GB of RAM.

We're looking into shrinking the installer. It's an unknown at this point if we can. If we can shrink it, then yes - Stacki (both frontend and backend) can run on less that 4 GB of memory. If not, then not.

--
You received this message because you are subscribed to the Google Groups "stacki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "stacki" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+un...@googlegroups.com.

[Mike Walters]

Adding more RAM(4GB) to the VM and restarting httpd service on frontend fixed my issue.

[Joe Kaiser]

How have you tried to change the bootaction?

You essentially have to overwrite the existing bootaction:

stack add bootaction <all the stuff>

And then you have to toggle the backend install after you've changed the bootaction.

# stack set host boot backend action=install

sometimes it's necessary to do this:

# stack set host boot backend action=os

# stack set host boot backend action=install

There was a weird bug in some of the earlier versions for 4.0. 

Thanks,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.

[Kellen O'Connor]

I'll respond to this tomorrow morning after I test these fixes. Thanks so much for the speedy responses.

You received this message because you are subscribed to a topic in the Google Groups "stacki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stacki/qELt69PjtZw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stacki+unsubscribe@googlegroups.com.

[Kellen O'Connor]

Okay, so I did a couple of things. I increased the RAM allocation to the VM that stacki is running on (VirtualBox). I also attempted the 'stack add bootaction' fix, but now it's giving me a different error. 

I get a "TFTP prefix: Unable to locate configuration file". 

I ran 'stack list bootaction' to find the vmlinuz version and the initrd version to use with 'stack add bootaction'.

So I'm not entirely sure if I messed up the syntax of the 'stack add bootaction' fix, or if something else is going on.

--

Kellen O'Connor

(215) 970 6250

kellen....@temple.edu

[Joe Kaiser]

send a:

stack list bootaction

Thanks,

Joe

[Kellen O'Connor]

ACTION                     KERNEL                                  RAMDISK                                     ARGS
install                    vmlinuz-4.0_20170414_c4aff2a-7.x-x86_64 initrd.img-4.0_201740414_cfaff2a-7.x-x86_64 ip=bootif:dhcp inst.ks=https://192.168.0.19/install/sbin/kickstart.cgi inst.noverifyssl inst.ks.sendmac ramdisk_size=400000
install headless           vmlinuz-4.0_20170414_c4aff2a-7.x-x86_64 initrd.img-4.0_20170414_c4aff2a-7.x-x86_64  ip=dhcp inst.ks=https://192.168.0.19/install/sbin/profile.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac inst.vnc ramdisk_size=300000
install no-parallel-format vmlinuz-4.0_20170414_c4aff2a-7.x-x86_64 initrd.img-4.0_20170414_c4aff2a-7.x-x86_64  ip=dhcp inst.ks=https://192.168.0.19/install/sbin/profile.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac no-parallel-format ramdisk_size=300000
memtest                    kernel memtest                          ------------------------------------------- 
os                         com32 chain.c32                         ------------------------------------------- hd0
os.hplocalboot             localboot -1                            ------------------------------------------- 
os.localboot               localboot 0                             ------------------------------------------- 
pxeflash                   kernel memdisk bigraw                   pxeflash.img                                keeppxe
rescue                     vmlinuz-4.0_20170414_c4aff2a-7.x-x86_64 initrd.img-4.0_20170414_c4aff2a-7.x-x86_64  ip=dhcp inst.ks=https://192.168.0.19/install/sbin/profile.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac rescue ramdisk_size=300000

You received this message because you are subscribed to a topic in the Google Groups "stacki" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stacki/qELt69PjtZw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to stacki+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kellen O'Connor

(215) 970 6250

kellen....@temple.edu

--
You received this message because you are subscribed to the Google Groups "stacki" group.

[Joe Kaiser]

That's looks fine for the bootactions.

do a:

# stack set host boot action backend action=os

# stack set host boot action backend action=install

and then:

# cat /tftpboot/pxelinux/pxelinux.cfg/*

To make sure it took. 

Try again.

If it happens again, send a screen shot of the exact error you are seeing. 

Thanks,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.

[Kellen O'Connor]

It looks like the directory /tftpboot/pxelinux/pxelinux.cfg/ is empty. 

Again, thanks so much for the replies. 

To unsubscribe from this group and all its topics, send an email to stacki+unsubscribe@googlegroups.com.

[Joe Kaiser]

Okay, send a:

# stack list host

and a:

# df -h

Thanks,

Joe

[Kellen O'Connor]

I removed backend host yesterday after the failed attempt using "Stack remove host backend-0-0" so only my frontend machine is listed in stack list host.

HOST    RACK RANK CPUS APPLIANCE BOX     ENVIRONMENT RUNACTION INSTALLACTION
keltest 0    0    4    frontend  default ----------- os        install      

I get this when I run "df -h"

Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/cl_keltest-root  6.2G  3.2G  3.1G  52% /
devtmpfs                     2.8G     0  2.8G   0% /dev
tmpfs                        2.8G     0  2.8G   0% /dev/shm
tmpfs                        2.8G  8.4M  2.8G   1% /run
tmpfs                        2.8G     0  2.8G   0% /sys/fs/cgroup
/dev/sda1                   1014M  138M  877M  14% /boot
tmpfs                        570M     0  570M   0% /run/user/0

[Joe Kaiser]

and that would be why your tftpboot directory is empty. There is nothing to install. 

So you'll have to try installing a backend node before we know if it works. 

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.

[Kellen O'Connor]

So this error that I said I was recieving: 

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.55.141/install/sbin/profile.cgi

I got this error when I was using "discover-nodes" to add the back end machine. 

Is discover-nodes still utilized, or is using the CSV method pretty much standard?

To unsubscribe from this group and all its topics, send an email to stacki+unsubscribe@googlegroups.com.

[Joe Kaiser]

discover-nodes is still working. but I'm wondering why you have a different ip address than what you want in that url.

So send the output of:

# stack list network

# stack list host interface

and 

# stack list attr | grep Kickstart

Thanks,

Joe

[Kellen O'Connor]

Oh no, I copied and pasted the error message from Mike Walters. My error message reads out 

[  80.042503] dracut-initqueue[580]: Warning: failed to fetch kickstart from https://192.168.0.19/install/sbin/profile.cgi

My apologies, I was on my cellphone.

[Joe Kaiser]

send the things in the previous email. 

thanks,

Joe

[Kellen O'Connor]

Output of stack list network:

NETWORK ADDRESS     MASK          GATEWAY     MTU   ZONE DNS   PXE
private 192.168.0.0 255.255.255.0 192.168.1.0 1500  ---- False True

Output of stack list host interface

HOST    INTERFACE DEFAULT NETWORK MAC               IP           NAME    MODULE VLAN OPTIONS CHANNEL
keltest eth0      True    private 08:00:27:c2:59:96 192.168.0.19 keltest ------ ---- ------- 

Output of stack list attr | grep Kickstart

Kickstart_Keyboard             -------- us
Kickstart_Lang                 -------- en_US
Kickstart_Langsupport          -------- en_US
Kickstart_PrivateAddress       True     192.168.0.19
Kickstart_PrivateBroadcast     True     192.168.0.255
Kickstart_PrivateDNSDomain     True     
Kickstart_PrivateDNSServers    -------- 8.8.8.8
Kickstart_PrivateEthernet      -------- 08:00:27:c2:59:96
Kickstart_PrivateGateway       -------- 192.168.1.0
Kickstart_PrivateHostname      True     keltest
Kickstart_PrivateInterface     -------- eth0
Kickstart_PrivateKickstartHost True     192.168.0.19
Kickstart_PrivateNTPHost       -------- 192.168.0.19
Kickstart_PrivateNetmask       True     255.255.255.0
Kickstart_PrivateNetmaskCIDR   True     24
Kickstart_PrivateNetwork       True     192.168.0.0
Kickstart_PrivateRootPassword  -------- $1$q9OTtkx8$n0m5bkeg.483TUR8810JC1
Kickstart_PublicNTPHost        -------- pool.ntp.org
Kickstart_Timezone             -------- America/New_York

[Joe Kaiser]

Okay, is this really your gateway?

192.168.1.0

That's usually the network ID and is reserved as special. Try pinging that ip, it should barf at you. 

Thanks,

Joe

To unsubscribe from this group and stop receiving emails from it, send an email to stacki+unsubscribe@googlegroups.com.

[Kellen O'Connor]

So yes, the gateway was incorrect. I'm using an unmanaged switch. On the front end I tried changing the gateway to 0.0.0.0

Also, here is a photo of the error I get when trying to add a node to the backend. It repeats this cycle until someone stops it.

To unsubscribe from this group and all its topics, send an email to stacki+unsubscribe@googlegroups.com.

[Joe Kaiser]

Try this first. Set the gateway for the private subnet in the "stack list network" to the ip of the frontend.

# stack set network gateway private gateway=192.168.0.19

Then:

# stack sync config

# stack sync host network frontend restart=true

The problem is that your Gatway in the attributes table is set to the .0 address and this may cause problems. You can't change it, even at the host level,  because it's immutable. 

Try resetting the gatway in the networks table as above, and if that doesn't work, reinstall the frontend and use the private ip address of the frontend as the gateway.

You can always add more networks after the install. 

Thanks,

Joe

[Kellen O'Connor]

Thanks! It's up and running, was able to provision my backend appliance with CentOS7. 

I wound up having to reinstall the frontend while setting the gateway address as my frontend machine's IP.

Thanks for all of the help.

[Joe Kaiser]

A pleasure, sometimes the right answer is just to start over.