Jira (PDB-5211) Check whether PuppetDB still doesn't complete shutdown after migration fails

14 views
Skip to first unread message

Rob Browning (Jira)

unread,
Jul 27, 2021, 2:25:04 PM7/27/21
to puppe...@googlegroups.com
Rob Browning created an issue
 
PuppetDB / Task PDB-5211
Check whether PuppetDB still doesn't complete shutdown after migration fails
Issue Type: Task Task
Assignee: Unassigned
Created: 2021/07/27 11:24 AM
Priority: Normal Normal
Reporter: Rob Browning

Check whether the issue described in PDB-3715 (ticket describes a way to reproduce the issue) still exists. If not, let's close that ticket, otherwise, we can just leave it alone for now. i.e. right now, just wanted to see if we've fixed the issue.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Bogdan Irimie (Jira)

unread,
Jul 28, 2021, 4:02:03 AM7/28/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Jul 28, 2021, 4:02:04 AM7/28/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Jul 28, 2021, 4:06:04 AM7/28/21
to puppe...@googlegroups.com

Andrei Filipovici (Jira)

unread,
Aug 11, 2021, 9:01:04 AM8/11/21
to puppe...@googlegroups.com

Andrei Filipovici (Jira)

unread,
Aug 13, 2021, 7:36:02 AM8/13/21
to puppe...@googlegroups.com
Andrei Filipovici commented on Task PDB-5211
 
Re: Check whether PuppetDB still doesn't complete shutdown after migration fails

Tried on a centos-7-x86_64 machine with the puppet agent 7.9.0 and puppetdb versions 6.12.0 and 7.9.0.
I first installed puppetdb version 6.12.0 and modified a table that breaks migration 78.

The exact steps are as follows:

rpm -Uvh https://yum.puppet.com/puppet7-release-el-7.noarch.rpm
yum install -y puppet agent

/opt/puppetlabs/bin/puppet module install puppetlabs-puppetdb --version 7.9.0

yum install java-1.8.0-openjdk-headless
rpm -ivh http://builds.delivery.puppetlabs.net/puppetdb/6.12.0/artifacts/el/7/puppet6/x86_64/puppetdb-6.12.0-1.el7.noarch.rpm

/opt/puppetlabs/puppet/bin/puppet apply << EOF
class

{'puppetdb::globals': version => '6.12.0'}

class

{'puppetdb': disable_ssl => true}

EOF

sudo -u postgres /usr/bin/psql -d puppetdb -c "ALTER TABLE certnames ADD COLUMN catalog_inputs_hash bytea"

wget http://builds.delivery.puppetlabs.net/puppetdb/7.5.2/artifacts/el/7/puppet7/x86_64/puppetdb-7.5.2-1.el7.noarch.rpm
/opt/puppetlabs/bin/puppet resource --evaltrace package puppetdb ensure=7.5.2 provider=rpm source=/root/puppetdb-7.5.2-1.el7.noarch.rpm
tail -f /var/log/puppetlabs/puppetdb/puppetdb.log
journalctl -u puppetdb | grep 'Failed to start puppetdb Service'
cat /usr/lib/systemd/system/puppetdb.service

Andrei Filipovici (Jira)

unread,
Aug 13, 2021, 7:54:03 AM8/13/21
to puppe...@googlegroups.com

Also the interesting thing is that PuppetDB is started in a loop, but it won't be able to start.
So, if you do a:

journalctl -u puppetdb | grep 'Failed to start puppetdb Service'

you'll see a bunch of:

Aug 13 09:53:34 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:53:43 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:53:53 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:02 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:12 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:21 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:31 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:40 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:50 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:54:59 well-clover systemd[1]: Failed to start puppetdb Service.
Aug 13 09:55:08 well-clover systemd[1]: Failed to start puppetdb Service.

Took a look at the puppetdb.service file located in /usr/lib/systemd/system/puppetdb.service and we saw the settings:

TimeoutStartSec=14400
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5

They might be the cause of the continuous starting of PuppetDB.

Andrei Filipovici (Jira)

unread,
Aug 16, 2021, 7:58:03 AM8/16/21
to puppe...@googlegroups.com

Andrei Filipovici (Jira)

unread,
Aug 16, 2021, 7:59:01 AM8/16/21
to puppe...@googlegroups.com

Andrei Filipovici (Jira)

unread,
Aug 16, 2021, 8:03:02 AM8/16/21
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages