simp bootstrap fails on puppet server starting for 5.2.0-0

218 views
Skip to first unread message

Brodie McDougald

unread,
Nov 2, 2016, 2:45:20 PM11/2/16
to SIMP Q&A Forum
I'm basically seeing the same issue described here https://groups.google.com/d/msg/simp/rE4z015TjJ4/oYr9RP9TBAAJ  however I'm not using FIPS in the setup, so the digest_algorithm trick doesn't help.  I'm attempting 5.2.0-0 setup.  I go through the 'simp config' portion, restart, then attempt 'simp bootstrap'.  It plugs along for a while then dies with the following:

Track => #### Done!
... with tag 'group'
Track => #### Done!

Relabeling filesystem for selinux...
Cleaning out /tmp
*** Running Puppet Finalization ***

Track => ########### Done!
Waiting for Puppet Server to Start  @
The Puppet Server did not start within 5 minutes. Please start puppetserver by hand and inspect any issues.

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:63:in `rescue in ensure_running'
/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:41:in `ensure_running'
/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:71:in `track_output'
/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:228:in `run'
/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli.rb:86:in `start'
/usr/local/share/gems/gems/simp-cli-1.0.20/bin/simp:5: in `<top (required)>'
/bin/simp:23:in `load'
/bin/simp:23:in `<main>'

Trevor Vaughan

unread,
Nov 2, 2016, 5:48:02 PM11/2/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

My guess is that you don't have enough RAM on your system (2.4G *free*).

Can you check /var/log/puppetlabs/puppetserver/puppetserver.log and see if there's anything useful in there?

Thanks,

Trevor


--
You received this message because you are subscribed to the Google Groups "SIMP Q&A Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/377ef05c-d13d-4773-b229-1e3eb51e8876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Trevor Vaughan
Vice President, Onyx Point, Inc

-- This account not approved for unencrypted proprietary information --

Brodie McDougald

unread,
Nov 3, 2016, 7:34:01 AM11/3/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Good morning Trevor.  I took a look at the things you suggested and here's what I found:

$cat /proc/meminfo
performed at the prompt after the failure shows I have just over 6.1GB free  (8GB system)

I checked out the puppetserver.log (slightly different location than what you referenced) and it has the following repeated several times toward the end of the log:

INFO      [puppet-server] Puppet Caching node for puppet.lab.net
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'
INFO      [puppet-server] Puppet Compiled catalog for puppet.lab.net in environment production in 169.44 seconds

Message repeats numerous times as mentioned with the times being the only noticeable change.


To unsubscribe from this group and stop receiving emails from it, send an email to simp+uns...@googlegroups.com.

Trevor Vaughan

unread,
Nov 4, 2016, 4:22:06 PM11/4/16
to Brodie McDougald, SIMP Q&A Forum
Ah, that.

I'm surprised that this wasn't set by default when you ran simp config and bootstrap but do you have 'trusted_node_data = true' in the [main] section of your puppet.conf?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/4499241e-3283-43e4-8096-d49050d2ff5f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 7, 2016, 8:30:01 AM11/7/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Good morning Trevor,
     Thanks for the feedback.  I just took a look at /etc/puppet/puppet.conf and 'trusted_node_data' is already set to true in that file.  Any other ideas?

Brodie

Brodie McDougald

unread,
Nov 7, 2016, 8:34:30 AM11/7/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Also thought I'd add to my previous remark... I was looking at the file and also saw something else that seemed odd (though unlikely to have any effect on the current issue).  environmentpath is called out twice in the [main] section.  Both instances are set to /etc/puppet/environments, so they don't conflict, but may be something to clean up next time you're working in that area.

Trevor Vaughan

unread,
Nov 7, 2016, 3:15:30 PM11/7/16
to Brodie McDougald, SIMP Q&A Forum
The environmetpath won't cause any issues.

Are you using Puppet 3 or Puppet 4? I've only seen that error in Puppet 4 so far.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0bcc824f-6fdf-4524-9d43-1d0ad8426c65%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 7, 2016, 4:55:16 PM11/7/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Well, I used the Cent7 iso and simp files on the page for 5.2.0-0 and checked out that tag.  So I haven't installed anything unusual or different beyond that that I recall.  To answer your question I did 'puppet --version' which returned 3.8.6.

Trevor Vaughan

unread,
Nov 8, 2016, 9:53:12 AM11/8/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

The only place where we explicitly use this variable is in the hiera.yaml file itself.

Can you try changing 'clientcert' to 'certname' in the hiera.yaml file and restarting the Puppet Server?

This appears to be a bug in our setup but I'm not sure why it's not evidencing in all cases.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/dba768f1-8f50-42c0-abc7-f0a2debf7b61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 8, 2016, 1:44:05 PM11/8/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Gave that a try and gave it another go.  Still fails the same way in the same spot.  After the failure I navigated back to /etc/puppet/ and looked at the hiera.yaml file to ensure the change I made was still there and hadn't been overwritten somehow in the restart and the change was still present.  So I guess that wasn't it either.  :(

Brodie

Trevor Vaughan

unread,
Nov 8, 2016, 2:01:05 PM11/8/16
to Brodie McDougald, SIMP Q&A Forum
Did you make sure to restart the puppetserver process?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/bc9cc184-4bac-4927-bf05-9aefc5977f62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Trevor Vaughan

unread,
Nov 8, 2016, 2:01:38 PM11/8/16
to Brodie McDougald, SIMP Q&A Forum
Sorry, just re-read all of that.

Can you grep for 'clientcert' in the /etc/puppet directory and see if it's anywhere else?

Thanks,

Trevor

Brodie McDougald

unread,
Nov 8, 2016, 3:29:40 PM11/8/16
to SIMP Q&A Forum, brodie.m...@gmail.com
No problem,
Did a grep recursively in /etc/puppet/ for the 'clientcert' and found it in the following file locations as well:
/etc/puppet/environments/simp/modules/mcollective/README.md
/etc/puppet/environments/simp/modules/mcollective/examples/mco_profile/manifests/params.pp
/etc/puppet/environments/simp/modules/puppetdb/spec/unit/classes/master/report_processor_spec.rb

Brodie



--
Trevor Vaughan
Vice President, Onyx Point, Inc

-- This account not approved for unencrypted proprietary information --

Trevor Vaughan

unread,
Nov 9, 2016, 9:44:22 AM11/9/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

Were any of those in conjunction with the 'trusted' hash? They should not have been.

$::clientcert is a valid variable, but it's not valid inside of the trusted hash.

Can you also check /etc/puppet/puppet.conf just to make sure it's not stuck in there somewhere?

Other than that, I can't actually think of a reason that this would be showing up in your manifest at all!

I suppose you could try 'grep -r clientcert /etc/puppet | grep trusted'. That's the only way that this would be called.

I just grepped through our entire codebase and could only find it in the hiera.yaml file.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/1f442e14-24cd-4ab0-8136-79beae4ade58%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 9, 2016, 10:38:49 AM11/9/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Good morning Trevor,
     Not exactly sure what you're looking for, but in trying to answer your questions...I did the grep you mentioned and it didn't show any results.  I then looked at params.pp and the mention of 'clientcert' was mentioned in 3 lines:

$ssl_server_public  =  "${::settings::ssldir}/public_keys/${::clientcert}.pem"
$ssl_server_private =  "${::settings::ssldir}/private_keys/${::clientcert}.pem"
$ssl_server_cert      =  "${::settings::ssldir}/certs/${::clientcert}.pem"

in the Ruby file it's located in the 'let(:facts) do' section as:
:clientcert    =>    'test.domain.local',

Brodie

Trevor Vaughan

unread,
Nov 9, 2016, 11:16:10 AM11/9/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

Those are OK, the issue, per the error message is with trusted['clientcert'] which is not a thing.

Can you try the following?

1) systemctl stop puppetserver
2) ps -ef | grep puppet

Kill anything you see running from #2 above then

1) systemctl start puppetserver
2) Wait for it to start
3) puppet agent -t
4) See if you have different errors in the puppetserver.log

Thanks,

Trevor


To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/45698166-dd49-420b-a62c-9f95f0e59c16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 9, 2016, 4:03:15 PM11/9/16
to SIMP Q&A Forum, brodie.m...@gmail.com
OK, ran through those steps and here are the results.  After running puppet agent -t, you see a series of messages on the screen during the start (agentFeedback.txt attached).  Puppetserver.log also attached.  Interestingly, it seems the cert errors from before were not in the log, but the agent -t reports warning and errors in that execution on screen as shown in the file.  Progress?

Thanks,
Brodie
agentFeedback.txt
puppetserver.log

Brodie McDougald

unread,
Nov 9, 2016, 4:42:03 PM11/9/16
to SIMP Q&A Forum, brodie.m...@gmail.com
As an addition I went ahead and tried 'simp bootstrap' again to see if it failed still, and it fails the same as before.  I am sure that was expected but thought it worth a try.

Brodie

Trevor Vaughan

unread,
Nov 9, 2016, 5:12:30 PM11/9/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

Can you try the following:

systemctl stop puppetserver

Wait until it stops

systemctl start puppetserver
watch -n1 'netstat -tlpn | grep 81'

When you see the puppetserver actually bind to the ports (may take a couple of minutes), then run 'puppet agent -t' by hand.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/cf4c3020-80cd-4e2f-adaf-78beefe218fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 10, 2016, 11:56:58 AM11/10/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Good morning.  I gave that a try.  The "watch" for netstat sits in listen for now over an hour and as far as I can tell, nothing new pops up.  All I get is the following which came up immediately after hitting enter on the watch command:

Every 1.0s: netstat -tlpn | grep 81
tcp6     0     0 :::8150       :::*     LISTEN     1865/java


Brodie

Trevor Vaughan

unread,
Nov 10, 2016, 7:17:43 PM11/10/16
to Brodie McDougald, SIMP Q&A Forum
OK, well, at least that's OK-ish.

When that's running, try the following:

puppet agent -t --masterport=8150

'simp bootstrap' normally does this, but I'm hoping that this will give us a better clue as to what's actually going wrong.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0fd1e746-43a9-49c0-b6c8-0d5def0ee882%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 14, 2016, 2:08:07 PM11/14/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Sorry about the delay, I was out of the office and am just getting back to this.

So I'm not sure how to run that "while the watch is running".  But I broke out of the watch command and ran 'puppet agent -t --masterport=8150'
That gave me the following:

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Net::ReadTimeout
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

(see updated puppetserver.log)
puppetserver.log

Trevor Vaughan

unread,
Nov 14, 2016, 2:39:53 PM11/14/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

Can you try puppet agent -t --masterport=8150 --server=<your.host.name>

It's starting to smell like either your host doesn't know itself (/etc/hosts or hostname) or there's something else on your network with a conflicting IP.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/da795acc-8b3f-4a37-853b-ed7e170d8bb2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 15, 2016, 11:23:44 AM11/15/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Sadly no luck either.  I did 'puppet agent -t --masterport=8150 --server=puppet.lab.net' and got the same error as before.  I then typed 'hostname' to make sure that was my actual hostname and sure enough I got 'puppet.lab.net' returned by the command.

Brodie

Trevor Vaughan

unread,
Nov 15, 2016, 12:13:14 PM11/15/16
to Brodie McDougald, SIMP Q&A Forum
Can you make sure that your reverse DNS lookup for your hostname also returns what you expect?

Also, try puppet agent -t --masterport=8150 --server=<your.ip.address> and see if that makes a difference.

Are you seeing any updates in the puppetserver's log when the client attempts to check in?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/519a9ea8-0d71-4cf1-a5c6-8b2392583f72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 15, 2016, 2:00:40 PM11/15/16
to SIMP Q&A Forum, brodie.m...@gmail.com
I think we may be getting focused in on what might be the root of the problem....maybe haha.
So, first off let me start with the second part.  I ran it with the IP address and it complains that 192.168.1.1 did not match the server certificate; expected one of puppet.lab.net, DNS:puppet, DNS:puppet.lab.net.
Nothing gets added to the puppetserver log for that run.
Now back to the first item.  This machine is all by itself.  It's not on a network, or on the internet.  It has no DNS defined other than perhaps itself...but DNS Server is not installed since that's not part of the SIMP installation for CENTOS7 as far as I can tell.  So doing dig, nslookup, etc doesn't find any servers with the name of puppet.lab.net.  That being said, /etc/hosts has an entry for '192.168.1.1 puppet.lab.net puppet' so I'd think it would at least resolve the name to the IP locally.

Trevor Vaughan

unread,
Nov 15, 2016, 3:42:40 PM11/15/16
to Brodie McDougald, SIMP Q&A Forum
OK, all of that actually sounds fine and what I would expect from a fresh install.

Are you still getting the same error when you stop the puppetserver and then run 'puppetserver foreground' and connect to it with the agent?

Thanks,

Trevor


To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/6baffd46-748e-46cb-a66a-d76c89ecdd3d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 15, 2016, 4:20:54 PM11/15/16
to SIMP Q&A Forum, brodie.m...@gmail.com
You confused me a little on that one.  If I run 'puppetserver foreground' as you suggest, then my session is locked into that process because you made it run in the foreground, and puppetserver is not something that ever "completes" to return me to a prompt while it's running.  So there's really no way I can then run agent from the allocated session prompt, unless I'm missing something.

However, yes, if I stop puppetserver and then start it again followed by running puppet agent, I'm getting the same errors as I always have since nothing has changed.

Perhaps I'm misunderstanding what you're wanting me to do?

Brodie

Trevor Vaughan

unread,
Nov 15, 2016, 5:09:05 PM11/15/16
to Brodie McDougald, SIMP Q&A Forum
You'll need to use two terminals.

Login to the second terminal <Alt>-<Ctrl>-F2, etc...

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/8ef036e5-5308-4ba0-928c-d05793736db7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 16, 2016, 9:38:14 AM11/16/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Ah ok.  Well I tried that and it still fails the same.  Newest log attached.

Brodie
puppetserver.log

Trevor Vaughan

unread,
Nov 16, 2016, 10:29:26 AM11/16/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

So, I see warnings, but I don't actually see any issues and it looks like it successfully compiled a catalog.

If you disable the Puppet server, then run it in the foreground in terminal 1 and run the Puppet agent in terminal 2 to connect to it, do you get any errors?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/fe1aa399-0839-47e2-a403-8b35037f5902%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 22, 2016, 11:39:53 AM11/22/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Hi Trevor,
    Sorry for the delay, but I had other projects demanding my time.  So yeah, even if I run them in two separate terminals I still get the "could not retrieve catalog" "skipping run" error as I always have.  It's strange you guys don't see this there as well.  I've had another guy here look at it as well.  He tried on two different VMs and mine on a laptop (so environments are not hardware unique) and both of us can repro the exact same issue in all 3 places.  Very frustrating.

Brodie

Trevor Vaughan

unread,
Nov 22, 2016, 2:12:02 PM11/22/16
to Brodie McDougald, SIMP Q&A Forum
Hi Brodie,

Would it be possible to try one of the ISOs from https://simp-project.com/ISO?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/66eace83-7fb7-4b6c-96be-be6629709537%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 22, 2016, 3:05:25 PM11/22/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Sure,
    Trying to finish up another task before everyone leaves for the holidays, but I'm downloading the new ISO now and will give it a shot on Monday and let you know.

Brodie

Trevor Vaughan

unread,
Nov 22, 2016, 3:14:21 PM11/22/16
to Brodie McDougald, SIMP Q&A Forum
OK, sorry about that. I have no idea how you're getting the error that you are with that value removed from the hiera files.

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/4c1c641f-8340-4858-8047-d87114a0be26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Brodie McDougald

unread,
Nov 28, 2016, 3:54:42 PM11/28/16
to SIMP Q&A Forum, brodie.m...@gmail.com
Looked at the link you sent.  I was using the 5.2.1-0 currently.  That's the one that's having these problems sadly.  One of our IT guys got a different build working on his VM... (think 5.1 maybe?).  Going to check with him and see if that helps.

Brodie
Brodie
Reply all
Reply to author
Forward
0 new messages