Jira (PDB-5095) Add a tool to timeshift archive exports

32 views
Skip to first unread message

Rob Browning (Jira)

unread,
Apr 6, 2021, 8:30:03 PM4/6/21
to puppe...@googlegroups.com
Rob Browning created an issue
 
PuppetDB / Improvement PDB-5095
Add a tool to timeshift archive exports
Issue Type: Improvement Improvement
Assignee: Unassigned
Components: PuppetDB
Created: 2021/04/06 5:29 PM
Priority: Normal Normal
Reporter: Rob Browning

Add a (for now) dev-only lein alias to timeshift wire format exports (produced by the pdb archive endpoint). Perhaps something like this:

lein timeshift-export --timeshift-to now -o timeshifted.tgz original.tgz

See the :install-gems alias in project.clj for an example:

:install-gems {:source-paths ^:replace ["src-gems"]
               :target-path "target-gems"
               :dependencies ~puppetserver-test-deps}

We might be fine with the normal (non-test) deps (not sure), and might want to include the main tree in the source paths, i.e. maybe :source-paths ^:replace ["src-shift-export" "src"] or something. Could also use https://github.com/clojure/tools.cli for the argument processing. Alhough if the above options are all we need for now, then handling the arguments manually may be sufficient.

In terms of the processing, we'll presumably open the archive, traverse it once, parsing the commands to collect the min and max timestamps (just like the SQL shifting code we're adding in PDB-4771 does). Then we'll open the output file and traverse input archive again to read, timeshift and write the commands to the output archive. See the code related to the archive endpoint for examples of opening, reading, and writing the archive files.

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)
Atlassian logo

Bogdan Irimie (Jira)

unread,
Apr 7, 2021, 9:15:04 AM4/7/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Apr 7, 2021, 9:37:03 AM4/7/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
-- Add a (for now) dev-only {{lein}} alias to timeshift wire format exports (produced by the pdb archive endpoint).  Perhaps something like this: --

{code:sh}

lein timeshift-export --timeshift-to now -o timeshifted.tgz original.tgz
{code}


See the {{:install-gems}} alias in {{project.clj}} for an example:

{code:clojure}

:install-gems {:source-paths ^:replace ["src-gems"]
               :target-path "target-gems"
               :dependencies ~puppetserver-test-deps}
{code}

-- We might be fine with the normal (non-test) deps (not sure), and might want to include the main tree in the source paths, i.e. maybe {{:source-paths ^:replace ["src-shift-export" "src"]}} or something.  Could also use https://github.com/clojure/tools.cli for the argument processing.  Alhough if the above options are all we need for now, then handling the arguments manually may be sufficient. --

In terms of the processing, we'll presumably open the archive, traverse it once, parsing the commands to collect the min and max timestamps (just like the SQL shifting code we're adding in PDB-4771 does).  Then we'll open the output file and traverse input archive again to read, timeshift and write the commands to the output archive. See the code related to the [archive endpoint|https://puppet.com/docs/puppetdb/latest/api/admin/v1/archive.html] for examples of opening, reading, and writing the archive files.

Bogdan Irimie (Jira)

unread,
Apr 7, 2021, 9:38:03 AM4/7/21
to puppe...@googlegroups.com

Oana Tanasoiu (Jira)

unread,
Apr 7, 2021, 10:37:03 AM4/7/21
to puppe...@googlegroups.com
Oana Tanasoiu updated an issue
 
Change By: Oana Tanasoiu
Labels: ghost tsr-pdb-backlog

Florin Dragos (Jira)

unread,
Apr 7, 2021, 10:40:03 AM4/7/21
to puppe...@googlegroups.com
Florin Dragos updated an issue
Change By: Florin Dragos
Remaining Estimate: 3 minutes
Original Estimate: 3 minutes

Florin Dragos (Jira)

unread,
Apr 7, 2021, 10:40:03 AM4/7/21
to puppe...@googlegroups.com
Florin Dragos updated an issue
Change By: Florin Dragos
Story Points: 3
Remaining Estimate: 3 minutes
Original Estimate: 3 minutes

Florin Dragos (Jira)

unread,
Apr 7, 2021, 10:40:04 AM4/7/21
to puppe...@googlegroups.com
Florin Dragos updated an issue
Change By: Florin Dragos
Sprint: HAHA/Grooming 2

Andrei Filipovici (Jira)

unread,
Apr 15, 2021, 3:55:02 AM4/15/21
to puppe...@googlegroups.com

Bogdan Irimie (Jira)

unread,
Apr 21, 2021, 3:27:04 AM4/21/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021 , HAHA/Grooming 2

Bogdan Irimie (Jira)

unread,
May 5, 2021, 9:44:03 AM5/5/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021, Ghost-5.05.2021 , HAHA/Grooming 2

Bogdan Irimie (Jira)

unread,
May 5, 2021, 10:29:01 AM5/5/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021, Ghost-5.05.2021, ghost-19.05.2021 , HAHA/Grooming 2

Bogdan Irimie (Jira)

unread,
May 19, 2021, 3:35:01 AM5/19/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021, Ghost-5.05.2021, ghost-19.05.2021, ghost-19.05.2021 , HAHA/Grooming 2

Andrei Filipovici (Jira)

unread,
May 20, 2021, 10:43:01 AM5/20/21
to puppe...@googlegroups.com
Andrei Filipovici commented on Improvement PDB-5095
 
Re: Add a tool to timeshift archive exports

The time shift export utility accepts 3 parameters:

  • [Mandatory] -i / --input
    Path to the .tgz pdb export, which will be shifted.
  • [Optional] -o / --output
    Path to the where the shifted export will be saved. If no path is given, the archive will be stored in the same place as the input export, but the export name will be prefixed with "shifted". (e.g input-export.tgz will become shifted-input-export.tgz). If the output path doesn't end with .tgz it will be added.
  • [Optional]-t / --shift-to-time
    Timestamp to which all the export timestamp will be shifted. If it's not provided, the system's current timestamp will be used.

The time shifting tool will convert all timestamps to a Zero timezone format. e.g timestamp like: 2015-03-26T10:58:51+10:00 will become 2015-03-26T11:58:51Z

Bogdan Irimie (Jira)

unread,
Jun 2, 2021, 3:30:01 AM6/2/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021, Ghost-5.05.2021, ghost-19.05.2021, ghost-19.05.2021, ghost-2.06.2021 , ready for triage 3

Bogdan Irimie (Jira)

unread,
Jun 16, 2021, 3:17:03 AM6/16/21
to puppe...@googlegroups.com
Bogdan Irimie updated an issue
Change By: Bogdan Irimie
Sprint: ghost-21.04.2021, Ghost-5.05.2021, ghost-19.05.2021, ghost-19.05.2021, ghost-2.06.2021, ghost-16.06.2021 , ready for triage 3

Rob Browning (Jira)

unread,
Jul 15, 2021, 12:23:03 PM7/15/21
to puppe...@googlegroups.com
Rob Browning updated an issue
Change By: Rob Browning
Fix Version/s: PDB 7.5.0

Rob Browning (Jira)

unread,
Jul 15, 2021, 12:23:04 PM7/15/21
to puppe...@googlegroups.com
Rob Browning updated an issue
Change By: Rob Browning
Fix Version/s: PDB 6.18.0
Reply all
Reply to author
Forward
0 new messages