Jira (PDB-4422) initial migration at first start fails puppetdb hangs.

[Felipe Garcia (JIRA)]

Felipe Garcia created an issue

PuppetDB / PDB-4422 initial migration at first start fails puppetdb hangs. Issue Type: Bug Affects Versions: PDB 5.2.8 Assignee: Unassigned Components: PuppetDB Created: 2019/06/25 12:43 AM Environment: CentOS Linux release 7.6.1810   rpm -qa | grep puppet *puppet*5-release-5.0.0-4.el7.noarch *puppet*db-5.2.8-1.el7.noarch puppet-agent-5.5.14-1.el7.x86_64 *puppet*db-termini-5.2.8-1.el7.noarch *postgre*sql96-server-9.6.14-1PGDG.rhel7.x86_64 *postgre*sql96-9.6.14-1PGDG.rhel7.x86_64 *postgre*sql96-contrib-9.6.14-1PGDG.rhel7.x86_64 *postgre*sql96-libs-9.6.14-1PGDG.rhel7.x86_64 Priority: Normal Reporter: Felipe Garcia installing puppetserver and hangs at puppetdb start. logs snip 2019-06-25 17:23:53,655 INFO  [p.p.s.migrate] [4/8] Reconstructing facts to refer to fact_values... 2019-06-25 17:23:53,659 INFO  [p.p.s.migrate] [5/8] Cleaning up duplicate null values... 2019-06-25 17:23:53,662 INFO  [p.p.s.migrate] [6/8] Computing fact value hashes... 2019-06-25 17:23:53,666 INFO  [p.p.s.migrate] [7/8] Indexing fact_values table... 2019-06-25 17:23:53,682 INFO  [p.p.s.migrate] [8/8] Indexing facts table... 2019-06-25 17:23:53,705 INFO  [p.p.s.migrate] Applied database migration version 64 in 70 ms 2019-06-25 17:23:53,706 INFO  [p.p.s.migrate] Applying database migration version 65 2019-06-25 17:23:53,746 ERROR [p.p.s.migrate] Caught SQLException during migration java.sql.BatchUpdateException: Batch entry 1 alter table resource_events      alter column status type text,      alter column property type text,      alter column containing_class type text,      alter column file type text was aborted.  Call getNextException to see the cause.         at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:133)         at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2004)         at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:360)         at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:1019)   full log avaliable if needed. Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Fredrik Larsson (JIRA)]

Fredrik Larsson commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs. I see the same issue, and it seems to be related to the PostgreSQL version. The migrations work with PostgreSQL versions up to and including 9.6.13, but they fail with version 9.6.14. I don't know PostgreSQL or PuppetDB well enough to understand exactly what fix in 9.6.14 breaks the migrations: https://www.postgresql.org/docs/9.6/release-9-6-14.html This is trivially reproducible using the Pupperware repository, by pinning the postgres image to version 9.6.13 PuppetDB starts as expected, using 9.6.14 (or simply 9.6 which points to 9.6.14), PuppetDB fails to start due to the SQLException during migration 65.

Add Comment

[Fredrik Larsson (JIRA)]

Fredrik Larsson updated an issue

PuppetDB / PDB-4422

initial migration at first start fails puppetdb hangs.

Change By: Fredrik Larsson Affects Version/s: PDB 6.3.2

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs. Postgres versions released last Thursday June 20th, which includes 9.6.14, contain a regression in the behavior of the ALTER TABLE statement that is causing this issue by preventing PuppetDB from initializing its database schema. A concise summary of the issue can be found in this message to the Postgres mailing list: https://www.postgresql.org/message-id/20190621004554.GB1626%40paquier.xyz There is a preliminary patch up for PuppetDB to work around the issue: https://github.com/puppetlabs/puppetdb/pull/3010 Until PuppetDB or Postgres release a new version, the best workaround is to use an older version of Postgres such as 9.6.13.

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs.

This fix has been merged and we are in the process of releasing versions of PuppetDB that avoid hitting the bug in PostgreSQL. You can follow the release Epic for your PuppetDB version stream if you'd like, but the packages will likely go live at the same time for all three. 5.2.9 - PDB-4425 6.0.4 - PDB-4431 6.3.4 - PDB-4437

Add Comment

[Felipe Garcia (JIRA)]

Felipe Garcia commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs.

I pinned the version of postgres to .13 and it works fine. Note. when puppetdb failed to migrate systemctl was unable to stop the service and it had to be killed off.

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt assigned an issue to Austin Blatt

PuppetDB / PDB-4422

initial migration at first start fails puppetdb hangs.

Change By: Austin Blatt Assignee: Austin Blatt Release Notes Summary: Due to a bug in the most recent releases of PostgreSQL (Bug #15865 https://www.postgresql.org/message-id/15865-17940eacc8f8b081%40postgresql.org)  9.4.23, 9.5.18, 9.6.14, 10.9, and 11.4 a PuppetDB database migration would fail. PuppetDB's database migrations are now re-written to avoid triggering that bug (with no substantive change to what the migrations do). Team: PuppetDB Release Notes: Bug Fix

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4422 initial migration at first start fails puppetdb hangs.

Change By: Austin Blatt Release Notes Summary: Due to a bug in the most recent releases of PostgreSQL ( [ Bug #15865 | https://www.postgresql.org/message-id/15865-17940eacc8f8b081%40postgresql.org ] )  9.4.23, 9.5.18, 9.6.14, 10.9, and 11.4 a PuppetDB database migration would fail. PuppetDB's database migrations are now re-written to avoid triggering that bug (with no substantive change to what the migrations do).

Add Comment

[Austin Blatt (JIRA)]

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4422 initial migration at first start fails puppetdb hangs.

Change By: Austin Blatt Release Notes Summary: Due to a bug in the most recent releases of PostgreSQL (Bug #15865 https://www.postgresql.org/message-id/15865-17940eacc8f8b081%40postgresql.org)   9.4.23, 9.5.18, 9.6.14, 10.9, and 11.4   (Bug #15865 https://www.postgresql.org/message-id/15865-17940eacc8f8b081%40postgresql.org) a PuppetDB database migration would fail. PuppetDB's database migrations are now re-written to avoid triggering that bug (with no substantive change to what the migrations do).

Add Comment

[Jean Bond (JIRA)]

Jean Bond updated an issue

PuppetDB / PDB-4422 initial migration at first start fails puppetdb hangs.

Change By: Jean Bond Labels: resolved-issue-added

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs. New versions of PuppetDB have been released that will work around the bug in postgres. Let us know if you run into any other issues.

Add Comment

[LeLutin (JIRA)]

LeLutin commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs.

Hi there! for what it's worth, we've just hit exactly the same problem with puppet 6.2.0 paired with postgresql 11.4 on debian buster (using debian packages for both). Downgrading to 11.3 lets puppetdb finish its schema upgrade, so the same change that happened in the 9.6.x branch must have also happened somewhere between 11.3 and 11.4 We're going to try importing the fix for puppetdb from one of the versions mentioned above to see if it fixes the issue for the 11.x branch of postgresql, too. The debian bug report is here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932135

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs.

LeLutin correct, we believe the bug likely affects all their newest Postgres versions (which are 11.4, 10.9, 9.6.14, 9.5.18, 9.4.23). Importing the fix from PuppetDB should fix the problem on 11.4 as well, please let us know if it doesn't.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PDB-4422

Re: initial migration at first start fails puppetdb hangs.

David Alexander A fix for this is already in 2018.1.9. The ship day is close enough, this week, that waiting for that is going to be your lowest-risk option compared to spending days testing and validating a one-off backport.

Add Comment