Puppet server stopped working

[Bret Wortman]

I accidentally ran puppet agent on our puppet master and now puppet server won't start up any more. Multiple reboots have failed to clear the situation and I can't figure out what file changed.

Here's the tail end of /var/log/messages | grep puppetserver, minus the datestamps:

:

puppet puppetserver: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

puppet puppetserver: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

puppet puppetserver: at java.lang.Thread.run(Thread.java:748)

puppet puppetserver: Background process 4918 exited before start had completed

puppet systemd: puppetserver.service: control process exited, code=exited status=1

puppet systemd: Failed to start puppetserver Service.

puppet systemd: Unit puppetserver.service entered failed state.

puppet systemd: puppetserver.service failed.

puppet systemd: puppetserver.service holdoff time over, scheduling restart

puppet systemd: Starting puppetserver Service...

puppet puppetserver: OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0

Any thoughts about where to look next for a solution, or at least a more in-depth understanding of what's going on? This server has been in operation for several years without any issues until we stomped something...

Thanks,

[Patrick Lesher]

One of our guys did something similar a couple weeks ago and we didn’t have a backup.  It sounds like ours was a little different in that the server was running fine, remote agents could connect, process, etc, but the puppet master’s agent refused to run.

Ours is a mono deployment on a single VM so I could take a snapshot and play around with it.

In the end I did an upgrade to the latest 2018 ( it was 2017.x ) and everything came up working.  I’m not sure if that will help you but it solved our problem.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/39635033-2e37-49de-8e86-493c7829654a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Bret Wortman]

Further to this, journalctl -xe has this to contribute when I try to start it up:

:

systemd[1]: Unit puppetserver.service entered failed state.

systemd[1]: puppetserver.service failed.

polkitd[632]: Unregistered Authentication Agent for unix-process:16477:1258070 (system bus name :1.652, object pathy /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_us.UTF

:

[Bret Wortman]

Aha! According to the puppetserver.log, I've got a weird keypar:

ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!

java.lang.IllegalArgumentException: Expected a KeyPair or PrivateKey, got org.bouncycastle.openssl.PEMEncryptedKeyPair@748f579b

:

:

What is this thing, and how can I recreate my server's original (or a new) keypair?

[Scott Hazelhurst]

Were you able to resolve this issue? I am now getting the same problem....

Thanks

Scott

[Bret Wortman]

I did, by building a new server. That said, I'd try this advice before starting over: https://puppet.com/docs/puppet/5.5/ssl_regenerate_certificates.html

For us it was also a change to move from a monolithic, everything on one server architecture to something a bit more distributed.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/c9tpVjpF4sc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/23914a82-0812-4aa8-92cc-ae35dbd6b6be%40googlegroups.com.

[Christopher Wood]

If you're reaching expiry you might consider this thing, it worked for me to refresh the CA cert:

https://forge.puppet.com/puppetlabs/certregen

(NB, check all your puppetserver/puppetmaster hosts for stray puppet/ssl/ca directories, having extra ones around can cause a bit of pain. Make sure you only have a "ca" dir on your CA host before starting, test first, etc.)

On Thu, Jul 19, 2018 at 01:55:33PM -0400, Bret Wortman wrote:
> I did, by building a new server. That said, I'd try this advice before
> starting

> over: [1]https://puppet.com/docs/puppet/5.5/ssl_regenerate_certificates.html

> For us it was also a change to move from a monolithic, everything on one
> server architecture to something a bit more distributed.
> On Thu, Jul 19, 2018 at 11:35 AM, Scott Hazelhurst
> <[2]scott.ha...@gmail.com> wrote:
>
> Were you able to resolve this issue? I am now getting the same
> problem....
> Thanks
> Scott
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit

> [3]https://groups.google.com/d/topic/puppet-users/c9tpVjpF4sc/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to

> [4]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [5]https://groups.google.com/d/msgid/puppet-users/23914a82-0812-4aa8-92cc-ae35dbd6b6be%40googlegroups.com.
> For more options, visit [6]https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to [7]puppet-users...@googlegroups.com.

> To view this discussion on the web visit

> [8]https://groups.google.com/d/msgid/puppet-users/CAN9oxgSh_G5qStiQd6DaJjZ%3DoTtQB0ms%3DcoyoD1Z5G%3DxSJ_pZQ%40mail.gmail.com.
> For more options, visit [9]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. https://puppet.com/docs/puppet/5.5/ssl_regenerate_certificates.html
> 2. mailto:scott.ha...@gmail.com
> 3. https://groups.google.com/d/topic/puppet-users/c9tpVjpF4sc/unsubscribe
> 4. mailto:puppet-users...@googlegroups.com
> 5. https://groups.google.com/d/msgid/puppet-users/23914a82-0812-4aa8-92cc-ae35dbd6b6be%40googlegroups.com?utm_medium=email&utm_source=footer
> 6. https://groups.google.com/d/optout
> 7. mailto:puppet-users...@googlegroups.com
> 8. https://groups.google.com/d/msgid/puppet-users/CAN9oxgSh_G5qStiQd6DaJjZ%3DoTtQB0ms%3DcoyoD1Z5G%3DxSJ_pZQ%40mail.gmail.com?utm_medium=email&utm_source=footer
> 9. https://groups.google.com/d/optout

[Scott Hazelhurst]

Many thanks for the help -- I managed to get it going

The other problem I had for the record was that I set the server_url to localhost while the server's CA was in its real name and that caused all sorts of confusion.

Regards

Scott