Jira (PDB-4606) Partition the reports table

21 views
Skip to first unread message

Robert Roland (JIRA)

unread,
Dec 18, 2019, 1:40:04 PM12/18/19
to puppe...@googlegroups.com
Robert Roland created an issue
 
PuppetDB / Story PDB-4606
Partition the reports table
Issue Type: Story Story
Assignee: Robert Roland
Components: PuppetDB
Created: 2019/12/18 10:39 AM
Priority: Normal Normal
Reporter: Robert Roland

The report table needs to be partitioned by day, like the resource_events table, to optimize storage and make it easier to deal with additional data being added from Bolt.

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Nick Walker (JIRA)

unread,
Dec 19, 2019, 4:37:03 PM12/19/19
to puppe...@googlegroups.com
Nick Walker commented on Story PDB-4606
 
Re: Partition the reports table

Could we add a configuration setting that truncates the reports table before performing this migration?

The reason I ask is because we suggest to many users that they truncate the reports table before large migrations with the reports table ( or postgresql upgrades) and performing the truncation requiring dropping an FK and recreating it unless you want to delete the entire contents of your database.

If it seems reasonable then adding documentation like "add this setting to puppetdb before upgrading" is a lot more palatable than run these SQL commands to remove all reports before the long migration.

Robert Roland (JIRA)

unread,
Dec 19, 2019, 6:10:03 PM12/19/19
to puppe...@googlegroups.com
Robert Roland commented on Story PDB-4606

I would prefer this to be handled entirely via documentation, where we document how to truncate this table, maybe even provide a script that assists them with it.

Adding a setting adds additional complexity to PuppetDB (the setting itself, validation, code to use it, unit tests, integration tests) that may not be extra maintenance in the future. I don't know of any other migrations where we have done this.

This isn't without precedent - there are already scripted truncation of tables via cron running at customer sites.

Eric Thompson (JIRA)

unread,
Dec 19, 2019, 6:25:03 PM12/19/19
to puppe...@googlegroups.com
Eric Thompson commented on Story PDB-4606

FWIW, i agree with Rob here. It feels like this can be handled with documentation and with the expert guidance our CS and SEs provide to our larger customers.
If we have issues later, perhaps we can talk about a setting or providing truncation automation.

This change is critical to Plans in PE supporting Plan reports in PDB. I wouldn't want to hold-up that critical product priority.

Austin Blatt (Jira)

unread,
Mar 11, 2020, 2:22:03 PM3/11/20
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Team: PuppetDB HA
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Austin Blatt (Jira)

unread,
Mar 11, 2020, 2:23:04 PM3/11/20
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Release Notes: Enhancement

Austin Blatt (Jira)

unread,
Mar 11, 2020, 2:23:06 PM3/11/20
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Sprint: HA Team 2020-03-25
Story Points: 3

Austin Blatt (Jira)

unread,
Mar 11, 2020, 2:24:03 PM3/11/20
to puppe...@googlegroups.com
Austin Blatt updated an issue
Change By: Austin Blatt
Story Points: 3 2

Austin Blatt (Jira)

unread,
Mar 11, 2020, 2:56:03 PM3/11/20
to puppe...@googlegroups.com
Austin Blatt assigned an issue to Austin Blatt
Change By: Austin Blatt
Assignee: Robert Roland Austin Blatt

Zachary Kent (Jira)

unread,
Apr 24, 2020, 5:38:04 PM4/24/20
to puppe...@googlegroups.com
Zachary Kent updated an issue
Change By: Zachary Kent
Fix Version/s: PDB 6.10.0

Margaret Lee (Jira)

unread,
Apr 27, 2020, 5:59:03 PM4/27/20
to puppe...@googlegroups.com
Margaret Lee commented on Story PDB-4606

Claire Cadman can we add something to the release notes for the platform this week to cover the following:

 
FOSS users, if they have a very large database, will need to change the start timeout manually if they do not delete their reports prior to the upgrade. We estimate it will take roughly 80-85 minutes per 10GB, but that’s an extremely loose number that may not reflect reality for everyone.

Claire Cadman (Jira)

unread,
Apr 28, 2020, 6:18:03 AM4/28/20
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages