Jira (PDB-4280) Ensure PE upgrade will work properly with reports migration

[Austin Blatt (JIRA)]

Austin Blatt created an issue

PuppetDB / PDB-4280 Ensure PE upgrade will work properly with reports migration Issue Type: New Feature Assignee: Unassigned Created: 2019/02/28 2:10 PM Priority: Normal Reporter: Austin Blatt To support PDB-3911, a migration to the reports table will land in Kearney that changes how we generate the report hash (and therefore changes all the report hashes currently in PuppetDB). PuppetDB's HA sync uses the report hashes to determine what is different between the two PuppetDB's. This means that if one PuppetDB is left online (running the old code), and the other PuppetDB is shutdown and upgraded, it will perform the migration, change all the report hashes and then attempt to perform an initial sync with the old PuppetDB. Since every report hash has changed, it will attempt to sync ever report again. Once it completes that sync, it'll be able to start up. Then the next periodic sync from the old PuppetDB will do the same thing, but in reverse.   The initial idea of disabling sync between these two versions of PuppetDB checking the schema_migration number seems to be inadequate to prevent the old PuppetDB from syncing with the new PuppetDB.   Outstanding questions (that are more specific than the general, how does the installer upgrade an HA deployment of PuppetDB?) Do both PuppetDB's get shutdown during an upgrade at the same time? Or are they entirely separate? Can we use the installer to disable HA sync between the two PuppetDB's before we begin upgrading and then have the installer re-enable sync once both PuppetDB's are upgraded? Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Austin Blatt (JIRA)]

Austin Blatt assigned an issue to Austin Blatt

PuppetDB / PDB-4280 Ensure PE upgrade will work properly with reports migration

Change By: Austin Blatt Assignee: Austin Blatt

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4280 Ensure PE upgrade will work properly with reports migration

Change By: Austin Blatt Team: PuppetDB

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4280 Ensure PE upgrade will work properly with reports migration Change By: Austin Blatt

To support PDB-3911, a migration to the reports table will (hgopefully) land in Kearney that changes how we generate the report hash (and therefore changes all the report hashes currently in PuppetDB). PuppetDB's HA sync uses the report hashes to determine what is different between the two PuppetDB's. This means that if one PuppetDB is left online (running the old code), and the other PuppetDB is shutdown and upgraded, it will perform the migration, change all the report hashes and then attempt to perform an initial sync with the old PuppetDB. Since every report hash has changed, it will attempt to sync ever report again. Once it completes that sync, it'll be able to start up. Then the next periodic sync from the old PuppetDB will do the same thing, but in reverse.

The initial idea of disabling sync between these two versions of PuppetDB checking the schema_migration number seems to be inadequate to prevent the old PuppetDB from syncing with the new PuppetDB.   Outstanding questions (that are more specific than the general, how does the installer upgrade an HA deployment of PuppetDB?)

* Do both PuppetDB's get shutdown during an upgrade at the same time? Or are they entirely separate? * Can we use the installer to disable HA sync between the two PuppetDB's before we begin upgrading and then have the installer re-enable sync once both PuppetDB's are upgraded?

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4280 Ensure PE upgrade will work properly with reports migration Change By: Austin Blatt

To support PDB-3911, a migration to the reports table will ( hgopefully hopefully ) land in Kearney that changes how we generate the report hash (and therefore changes all the report hashes currently in PuppetDB). PuppetDB's HA sync uses the report hashes to determine what is different between the two PuppetDB's. This means that if one PuppetDB is left online (running the old code), and the other PuppetDB is shutdown and upgraded, it will perform the migration, change all the report hashes and then attempt to perform an initial sync with the old PuppetDB. Since every report hash has changed, it will attempt to sync ever report again. Once it completes that sync, it'll be able to start up. Then the next periodic sync from the old PuppetDB will do the same thing, but in reverse.

The initial idea of disabling sync between these two versions of PuppetDB checking the schema_migration number seems to be inadequate to prevent the old PuppetDB from syncing with the new PuppetDB.   Outstanding questions (that are more specific than the general, how does the installer upgrade an HA deployment of PuppetDB?) * Do both PuppetDB's get shutdown during an upgrade at the same time? Or are they entirely separate? * Can we use the installer to disable HA sync between the two PuppetDB's before we begin upgrading and then have the installer re-enable sync once both PuppetDB's are upgraded?

Add Comment

[Joshua Partlow (JIRA)]

Joshua Partlow commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration Upgrade is one node at a time. The installer itself only handles upgrade of primary infrastructure (all the services that you would have a basic monolithic install of PE). In some configurations, you may have most services except postgres on one node, and a separately managed postgresql node. In older configurations, you might have separate primary master, puppetdb (and postgres), and console nodes. But I believe the specific installation we are concerned with are HA installations of PE, which, for the most part, look like a single monolithic primary and a single monolithic replica. There's a puppetdb instance backed to a postgres instance on each of those nodes. The replica is secondary infrastructure. It is not upgraded by the installer, but by a separate upgrade script. But, regardless, each node is upgraded individually by the admin running the installer on each node in sequence, or in the case of secondary infrastructure like the replica, running the upgrade script on that node after the primary infrastructure is upgraded and running fully again.

Add Comment

[Joshua Partlow (JIRA)]

Joshua Partlow commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

What's the blocker preventing puppetdb from recognizing that it it's counterpart is a different version?

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

We could add it to Kearney, and it'd be able to identify that it shouldn't sync from the older PuppetDB, but the old PuppetDB wouldn't know how to check with the upgraded one and our HA sync uses entirely standard queries, so the upgraded PuppetDB would be unaware that it was being synced from by an old PuppetDB.

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

If we made the changes necessary to prevent the old PuppetDB from syncing with a newer one in the upcoming PE 2018.1.8, 2019.0.3 releases, is it acceptable to require a customer to be on the newest z release of their track before upgrading to 2019.1.0?

Add Comment

[Joshua Partlow (JIRA)]

Joshua Partlow commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

Hmm, that's a good point; I see the problem now. I'll have to think about the z requirements. Nick Walker What are the upgrade requirements? We could possibly tighten that?

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

Based on what I'm seeing in PE we only configure sync on the replica node. Looking something like this: [root@pe-replica ~]# cat /etc/puppetlabs/puppetdb/conf.d/sync.ini   [sync] server_urls = https://master201903-centos.puppetdebug.vlan:8081 intervals = 2m And sync is not configured on the primary. So if the primary is upgraded to a version of PuppetDB that has the new hash then I think it can require that the initiating PuppetDB sync sends along its schema_migration number. If no schema_migration number is sent along then the new PuppetDB version should not sync. Make sense?

Add Comment

[Nick Walker (JIRA)]

Nick Walker commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

In order to allow a newer PuppetDB to initiate a sync with an older PuppetDB I suppose we could require that the non-initiating PuppetDB send some new field in it's response to the initiating PuppetDB and if it's not present then don't sync either. This new field could be something like "sync_version" that when present we could use to determine if the PuppetDBs should sync with each other and when not present we know we shouldn't sync with that version of PuppetDB.

Add Comment

[Joshua Partlow (JIRA)]

Joshua Partlow commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

Nick Walker I have a 2019.1 primary and replica configured, and both have /etc/puppetlabs/puppetdb/conf.d/sync.ini. On the primary it's pointing to the replica, and on the replica it's pointing to the primary. primary: nptm5vsc0i0y618.delivery.puppetlabs.net replica: yyhgstm8f74vq91.delivery.puppetlabs.net The primary master's sync peers are set in the PE HA Master node group: https://yyhgstm8f74vq91.delivery.puppetlabs.net/#/configure/groups/13a5d374-ee67-4a91-ab1f-22321ea55d67/classes That system was from a `frankenbuilder 2019.1 --install --provision`

Add Comment

[Joshua Partlow (JIRA)]

Joshua Partlow commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

Austin Blatt I know nothing about how two puppetdb's negotiate a sync. Are there two cases here? 1. primary puppetdb wants to sync to the replica puppetdb, initiates contact with the replica 2. replica puppetdb wants to sync to the primary puppetdb, initiates contact with the primary In case #1, is it reasonably for puppetdb to check the status endpoint of the host it is going to sync with and stop if it is < 6.3 (assuming that's the API change version)? In case #2, is there a point where primary puppetdb, having been contacted to establish a sync, could again check the syncing nodes status endpoint and abort? Or is what you are saying that there is no negotiation, and a 'sync' is just one instance pushing a bunch of SQL to another?

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4280

Re: Ensure PE upgrade will work properly with reports migration

There's no negotiation, in order to facilitate bringing up a brand new PuppetDB without changing configuration to the existing PuppetDB, sync is controlled entirely from the side that is asking for the data. And it asks for data in through standard query paths so the queries don't appear as sync specific.

Add Comment

[Claudia Petty (Jira)]

Claudia Petty updated an issue

PuppetDB / PDB-4280

Ensure PE upgrade will work properly with reports migration

Change By: Claudia Petty Labels: new-feature

Add Comment

This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)