Design Doc for a new Prometheus Migration Tool

[Harkishen Singh]

Hey everyone!

The prometheus monitoring metric data is very useful for a wide range of use-cases. Prometheus users often have data that can be several years old. In order to store all of this data, prometheus integrates with third-party services that provide long-term data storage functionality. Currently, Prometheus has 27 officially mentioned remote-storage integration solutions. With so many remote storage platforms out there along with a long data retention periods, it is very likely that users at some point might want/have to switch from one remote storage solution to another. It then becomes necessary and desired to transfer (old) data from one storage system to another to avoid having to maintain, query, etc. two systems. This is a big deal for users who want a simplified data storage and query system. It also avoids lock-in.

Towards that goal, we (at Timescale/Promscale) decided to come up with an open-source universal data migration tool, that can migrate data from one remote storage to another. This tool is able to read and write to a diverse set of storage systems because it uses the almost-universally-implemented remote_read/remote_write APIs. This migration process is designed to be highly scalable and reliable so that there remains no issue while migrating several terabytes of prometheus monitoring data, even if the process stops for some reason (and hence we support the ability to resume the process). At the same time, this process should remain stateless to ease deployment and operations. Finally, the process informs the user as to the progress being made with interactive outputs at runtime. With the above idea, we have designed a document around this and would be very happy if we can get more eyes on this and get additional suggestions/feedback on it. 

Link to the design document: https://tsdb.co/prom-migrator-design-doc

We want this tool to be open-source, community-driven, and free to use for everyone in the Prometheus community.

Thank you,
The TimescaleDB/Promscale team