simp bootstrap fails on puppet server starting for 5.2.0-0

[Brodie McDougald]

I'm basically seeing the same issue described here https://groups.google.com/d/msg/simp/rE4z015TjJ4/oYr9RP9TBAAJ  however I'm not using FIPS in the setup, so the digest_algorithm trick doesn't help.  I'm attempting 5.2.0-0 setup.  I go through the 'simp config' portion, restart, then attempt 'simp bootstrap'.  It plugs along for a while then dies with the following:

Track => #### Done!

... with tag 'group'

Track => #### Done!

Relabeling filesystem for selinux...

Cleaning out /tmp

*** Running Puppet Finalization ***

Track => ########### Done!

Waiting for Puppet Server to Start  @

The Puppet Server did not start within 5 minutes. Please start puppetserver by hand and inspect any issues.

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:63:in `rescue in ensure_running'

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:41:in `ensure_running'

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:71:in `track_output'

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli/commands/bootstrap.rb:228:in `run'

/usr/local/share/gems/gems/simp-cli-1.0.20/lib/simp/cli.rb:86:in `start'

/usr/local/share/gems/gems/simp-cli-1.0.20/bin/simp:5: in `<top (required)>'

/bin/simp:23:in `load'

/bin/simp:23:in `<main>'

[Trevor Vaughan]

Hi Brodie,

My guess is that you don't have enough RAM on your system (2.4G *free*).

Can you check /var/log/puppetlabs/puppetserver/puppetserver.log and see if there's anything useful in there?

Thanks,

Trevor

--
You received this message because you are subscribed to the Google Groups "SIMP Q&A Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/377ef05c-d13d-4773-b229-1e3eb51e8876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Brodie McDougald]

Good morning Trevor.  I took a look at the things you suggested and here's what I found:

$cat /proc/meminfo

performed at the prompt after the failure shows I have just over 6.1GB free  (8GB system)

I checked out the puppetserver.log (slightly different location than what you referenced) and it has the following repeated several times toward the end of the log:

INFO      [puppet-server] Puppet Caching node for puppet.lab.net

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::hostgroup';

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

WARN    [puppet-server] Scope(Class[main]) Could not look up qualified variable '::trusted["clientcert"]'

INFO      [puppet-server] Puppet Compiled catalog for puppet.lab.net in environment production in 169.44 seconds

Message repeats numerous times as mentioned with the times being the only noticeable change.

To unsubscribe from this group and stop receiving emails from it, send an email to simp+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/377ef05c-d13d-4773-b229-1e3eb51e8876%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Ah, that.

I'm surprised that this wasn't set by default when you ran simp config and bootstrap but do you have 'trusted_node_data = true' in the [main] section of your puppet.conf?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/4499241e-3283-43e4-8096-d49050d2ff5f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Good morning Trevor,

     Thanks for the feedback.  I just took a look at /etc/puppet/puppet.conf and 'trusted_node_data' is already set to true in that file.  Any other ideas?

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/4499241e-3283-43e4-8096-d49050d2ff5f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Also thought I'd add to my previous remark... I was looking at the file and also saw something else that seemed odd (though unlikely to have any effect on the current issue).  environmentpath is called out twice in the [main] section.  Both instances are set to /etc/puppet/environments, so they don't conflict, but may be something to clean up next time you're working in that area.

[Trevor Vaughan]

The environmetpath won't cause any issues.

Are you using Puppet 3 or Puppet 4? I've only seen that error in Puppet 4 so far.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0bcc824f-6fdf-4524-9d43-1d0ad8426c65%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Well, I used the Cent7 iso and simp files on the page for 5.2.0-0 and checked out that tag.  So I haven't installed anything unusual or different beyond that that I recall.  To answer your question I did 'puppet --version' which returned 3.8.6.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0bcc824f-6fdf-4524-9d43-1d0ad8426c65%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Hi Brodie,

The only place where we explicitly use this variable is in the hiera.yaml file itself.

Can you try changing 'clientcert' to 'certname' in the hiera.yaml file and restarting the Puppet Server?

This appears to be a bug in our setup but I'm not sure why it's not evidencing in all cases.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/dba768f1-8f50-42c0-abc7-f0a2debf7b61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Gave that a try and gave it another go.  Still fails the same way in the same spot.  After the failure I navigated back to /etc/puppet/ and looked at the hiera.yaml file to ensure the change I made was still there and hadn't been overwritten somehow in the restart and the change was still present.  So I guess that wasn't it either.  :(

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/dba768f1-8f50-42c0-abc7-f0a2debf7b61%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Did you make sure to restart the puppetserver process?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/bc9cc184-4bac-4927-bf05-9aefc5977f62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Sorry, just re-read all of that.

Can you grep for 'clientcert' in the /etc/puppet directory and see if it's anywhere else?

Thanks,

Trevor

[Brodie McDougald]

No problem,

Did a grep recursively in /etc/puppet/ for the 'clientcert' and found it in the following file locations as well:

/etc/puppet/environments/simp/modules/mcollective/README.md

/etc/puppet/environments/simp/modules/mcollective/examples/mco_profile/manifests/params.pp

/etc/puppet/environments/simp/modules/puppetdb/spec/unit/classes/master/report_processor_spec.rb

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/bc9cc184-4bac-4927-bf05-9aefc5977f62%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Trevor Vaughan
Vice President, Onyx Point, Inc

(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

[Trevor Vaughan]

Hi Brodie,

Were any of those in conjunction with the 'trusted' hash? They should not have been.

$::clientcert is a valid variable, but it's not valid inside of the trusted hash.

Can you also check /etc/puppet/puppet.conf just to make sure it's not stuck in there somewhere?

Other than that, I can't actually think of a reason that this would be showing up in your manifest at all!

I suppose you could try 'grep -r clientcert /etc/puppet | grep trusted'. That's the only way that this would be called.

I just grepped through our entire codebase and could only find it in the hiera.yaml file.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/1f442e14-24cd-4ab0-8136-79beae4ade58%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Good morning Trevor,

     Not exactly sure what you're looking for, but in trying to answer your questions...I did the grep you mentioned and it didn't show any results.  I then looked at params.pp and the mention of 'clientcert' was mentioned in 3 lines:

$ssl_server_public  =  "${::settings::ssldir}/public_keys/${::clientcert}.pem"

$ssl_server_private =  "${::settings::ssldir}/private_keys/${::clientcert}.pem"

$ssl_server_cert      =  "${::settings::ssldir}/certs/${::clientcert}.pem"

in the Ruby file it's located in the 'let(:facts) do' section as:

:clientcert    =>    'test.domain.local',

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/1f442e14-24cd-4ab0-8136-79beae4ade58%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Hi Brodie,

Those are OK, the issue, per the error message is with trusted['clientcert'] which is not a thing.

Can you try the following?

1) systemctl stop puppetserver

2) ps -ef | grep puppet

Kill anything you see running from #2 above then

1) systemctl start puppetserver

2) Wait for it to start

3) puppet agent -t

4) See if you have different errors in the puppetserver.log

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/45698166-dd49-420b-a62c-9f95f0e59c16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

OK, ran through those steps and here are the results.  After running puppet agent -t, you see a series of messages on the screen during the start (agentFeedback.txt attached).  Puppetserver.log also attached.  Interestingly, it seems the cert errors from before were not in the log, but the agent -t reports warning and errors in that execution on screen as shown in the file.  Progress?

Thanks,

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/45698166-dd49-420b-a62c-9f95f0e59c16%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

As an addition I went ahead and tried 'simp bootstrap' again to see if it failed still, and it fails the same as before.  I am sure that was expected but thought it worth a try.

Brodie

[Trevor Vaughan]

Hi Brodie,

Can you try the following:

systemctl stop puppetserver

Wait until it stops

systemctl start puppetserver

watch -n1 'netstat -tlpn | grep 81'

When you see the puppetserver actually bind to the ports (may take a couple of minutes), then run 'puppet agent -t' by hand.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/cf4c3020-80cd-4e2f-adaf-78beefe218fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Good morning.  I gave that a try.  The "watch" for netstat sits in listen for now over an hour and as far as I can tell, nothing new pops up.  All I get is the following which came up immediately after hitting enter on the watch command:

Every 1.0s: netstat -tlpn | grep 81

tcp6     0     0 :::8150       :::*     LISTEN     1865/java

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/cf4c3020-80cd-4e2f-adaf-78beefe218fe%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

OK, well, at least that's OK-ish.

When that's running, try the following:

puppet agent -t --masterport=8150

'simp bootstrap' normally does this, but I'm hoping that this will give us a better clue as to what's actually going wrong.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0fd1e746-43a9-49c0-b6c8-0d5def0ee882%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Sorry about the delay, I was out of the office and am just getting back to this.

So I'm not sure how to run that "while the watch is running".  But I broke out of the watch command and ran 'puppet agent -t --masterport=8150'

That gave me the following:

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: Net::ReadTimeout

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

(see updated puppetserver.log)

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/0fd1e746-43a9-49c0-b6c8-0d5def0ee882%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Hi Brodie,

Can you try puppet agent -t --masterport=8150 --server=<your.host.name>

It's starting to smell like either your host doesn't know itself (/etc/hosts or hostname) or there's something else on your network with a conflicting IP.

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/da795acc-8b3f-4a37-853b-ed7e170d8bb2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Sadly no luck either.  I did 'puppet agent -t --masterport=8150 --server=puppet.lab.net' and got the same error as before.  I then typed 'hostname' to make sure that was my actual hostname and sure enough I got 'puppet.lab.net' returned by the command.

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/da795acc-8b3f-4a37-853b-ed7e170d8bb2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

Can you make sure that your reverse DNS lookup for your hostname also returns what you expect?

Also, try puppet agent -t --masterport=8150 --server=<your.ip.address> and see if that makes a difference.

Are you seeing any updates in the puppetserver's log when the client attempts to check in?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/519a9ea8-0d71-4cf1-a5c6-8b2392583f72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

I think we may be getting focused in on what might be the root of the problem....maybe haha.

So, first off let me start with the second part.  I ran it with the IP address and it complains that 192.168.1.1 did not match the server certificate; expected one of puppet.lab.net, DNS:puppet, DNS:puppet.lab.net.

Nothing gets added to the puppetserver log for that run.

Now back to the first item.  This machine is all by itself.  It's not on a network, or on the internet.  It has no DNS defined other than perhaps itself...but DNS Server is not installed since that's not part of the SIMP installation for CENTOS7 as far as I can tell.  So doing dig, nslookup, etc doesn't find any servers with the name of puppet.lab.net.  That being said, /etc/hosts has an entry for '192.168.1.1 puppet.lab.net puppet' so I'd think it would at least resolve the name to the IP locally.

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/519a9ea8-0d71-4cf1-a5c6-8b2392583f72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

OK, all of that actually sounds fine and what I would expect from a fresh install.

Are you still getting the same error when you stop the puppetserver and then run 'puppetserver foreground' and connect to it with the agent?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/6baffd46-748e-46cb-a66a-d76c89ecdd3d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

You confused me a little on that one.  If I run 'puppetserver foreground' as you suggest, then my session is locked into that process because you made it run in the foreground, and puppetserver is not something that ever "completes" to return me to a prompt while it's running.  So there's really no way I can then run agent from the allocated session prompt, unless I'm missing something.

However, yes, if I stop puppetserver and then start it again followed by running puppet agent, I'm getting the same errors as I always have since nothing has changed.

Perhaps I'm misunderstanding what you're wanting me to do?

Brodie

To view this discussion on the web visit https://groups.google.com/d/msgid/simp/6baffd46-748e-46cb-a66a-d76c89ecdd3d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Trevor Vaughan]

You'll need to use two terminals.

Login to the second terminal <Alt>-<Ctrl>-F2, etc...

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/8ef036e5-5308-4ba0-928c-d05793736db7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Ah ok.  Well I tried that and it still fails the same.  Newest log attached.

Brodie

[Trevor Vaughan]

Hi Brodie,

So, I see warnings, but I don't actually see any issues and it looks like it successfully compiled a catalog.

If you disable the Puppet server, then run it in the foreground in terminal 1 and run the Puppet agent in terminal 2 to connect to it, do you get any errors?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/fe1aa399-0839-47e2-a403-8b35037f5902%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Hi Trevor,

    Sorry for the delay, but I had other projects demanding my time.  So yeah, even if I run them in two separate terminals I still get the "could not retrieve catalog" "skipping run" error as I always have.  It's strange you guys don't see this there as well.  I've had another guy here look at it as well.  He tried on two different VMs and mine on a laptop (so environments are not hardware unique) and both of us can repro the exact same issue in all 3 places.  Very frustrating.

Brodie

[Trevor Vaughan]

Hi Brodie,

Would it be possible to try one of the ISOs from https://simp-project.com/ISO?

Thanks,

Trevor

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/66eace83-7fb7-4b6c-96be-be6629709537%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Sure,

    Trying to finish up another task before everyone leaves for the holidays, but I'm downloading the new ISO now and will give it a shot on Monday and let you know.

Brodie

[Trevor Vaughan]

OK, sorry about that. I have no idea how you're getting the error that you are with that value removed from the hiera files.

To unsubscribe from this group and stop receiving emails from it, send an email to simp+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/simp/4c1c641f-8340-4858-8047-d87114a0be26%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Brodie McDougald]

Looked at the link you sent.  I was using the 5.2.1-0 currently.  That's the one that's having these problems sadly.  One of our IT guys got a different build working on his VM... (think 5.1 maybe?).  Going to check with him and see if that helps.

Brodie

Brodie

To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/simp/0fd1e746-43a9-49c0-b6c8-0d5def0ee882%40googlegroups.com?utm_me