Transient Problem

[Jeremy Miller]

I have a new installation of stacki 3.0-7 running on centos 7.1.  I'm getting the following errors that seem to randomly come and go.  Rebooting the stacki frontend will sometimes fix the issue.

I've tried the fixes from this thread without any progress: https://groups.google.com/forum/#!searchin/stacki/transient/stacki/5-999peP0wI/OQMz980QDwAJ

Any ideas how I can troubleshoot this further?

Thanks!

Here is the output of a few commands:

#     stack list bootaction

ACTION                      KERNEL                RAMDISK               ARGS                                                                                                                                    

install:                    vmlinuz-3.0-x86_64    initrd.img-3.0-x86_64 ip=bootif:dhcp inst.ks=https://10.84.108.80/install/sbin/kickstart.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac                   

install headless:           vmlinuz-3.0-x86_64    initrd.img-3.0-x86_64 ip=bootif:dhcp inst.ks=https://10.84.108.80/install/sbin/kickstart.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac headless          

install no-parallel-format: vmlinuz-3.0-x86_64    initrd.img-3.0-x86_64 ip=bootif:dhcp inst.ks=https://10.84.108.80/install/sbin/kickstart.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac no-parallel-format

memtest:                    kernel memtest        --------------------- ----------------------------------------------------------------------------------------------------------------------------------------

os:                         com32 chain.c32       --------------------- hd0                                                                                                                                     

os.hplocalboot:             localboot -1          --------------------- ----------------------------------------------------------------------------------------------------------------------------------------

os.localboot:               localboot 0           --------------------- ----------------------------------------------------------------------------------------------------------------------------------------

pxeflash:                   kernel memdisk bigraw pxeflash.img          keeppxe                                                                                                                                 

rescue:                     vmlinuz-3.0-x86_64    initrd.img-3.0-x86_64 ip=bootif:dhcp inst.ks=https://10.84.108.80/install/sbin/kickstart.cgi inst.geoloc=0 inst.noverifyssl inst.ks.sendmac rescue       

tail -f /var/log/httpd/log*

==> /var/log/httpd/ssl_access_log <==

10.84.101.52 - - [19/May/2016:08:23:32 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:32 -0400] 10.84.101.52 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.68 - - [19/May/2016:08:23:34 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:34 -0400] 10.84.108.68 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.101.52 - - [19/May/2016:08:23:34 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:34 -0400] 10.84.101.52 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.68 - - [19/May/2016:08:23:35 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:35 -0400] 10.84.108.68 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.101.52 - - [19/May/2016:08:23:35 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:35 -0400] 10.84.101.52 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.58 - - [19/May/2016:08:23:36 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:36 -0400] 10.84.108.58 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.68 - - [19/May/2016:08:23:37 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:37 -0400] 10.84.108.68 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=1 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.101.52 - - [19/May/2016:08:23:37 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:37 -0400] 10.84.101.52 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=8 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.58 - - [19/May/2016:08:23:38 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:38 -0400] 10.84.108.58 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 25

==> /var/log/httpd/ssl_access_log <==

10.84.108.58 - - [19/May/2016:08:23:39 -0400] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 503 25

==> /var/log/httpd/ssl_request_log <==

[19/May/2016:08:23:39 -0400] 10.84.108.58 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 "GET /install/sbin/kickstart.cgi?arch=x86_64&np=2 HTTP/1.1" 25

^C

# cat /var/tmp/kickstart.semaphore

32

[Jeremy Miller]

Error on terminal: 

Transient problem: HTTP error Will retry in 4 seconds. 1 retries

Warning: left.

Throwing away 25 bytes

[Jeremy Miller]

Also, it seems that restarting httpd.service will kick it into place and allow the backend to install but not consistently.

[Greg DuVall Bruno]

I don't have a good idea on what might be happening.

If the problem happens again, and if you are ok with me logging into your frontend as root, then send me a note and I'll debug it.

 - gb

[Jeremy Miller]

My system is behind a firewall so that will be difficult, can we debug over chat or on this board for the benefit of others?

Thanks

[Greg DuVall Bruno]

Yes, we can do that.

 - gb

[Kellen O'Connor]

I am having this exact same issue as well, exact same error message when PXE Booting. Same setup, just Stacki 4.0.