Prometheus 1.8 -> 2.0 data migration tool

[Alexey “AlekSi” Palazhchenko]

Hi,

I wrote a small tool to migrate all data from running Prometheus 1.8 instance to a new tsdb directory for Prometheus 2.0. It can be used to move all data to the new Prometheus instead of running two versions simultaneously. It is here: https://github.com/Percona-Lab/prom-migrate

It wasn't properly tested yet, etc. etc. Please do try it, but do not remove your old data just yet. I would be especially thankful if someone from Prometheus team will take a look if I miss some important tsdb options, tweaks or something else.

–-–
Alexey «AlekSi» Palazhchenko

[Ben Kochie]

Thanks, this is a pretty interesting migration tool.

I did some testing, it seems to work, but only when the server is pretty small, and the index of all metrics can be queried easily.  This seems to top out at around 10,000 metrics before it got really slow.

In talking with some of the other Prometheus developers, it would help to query the list of all instances and shard the work over a number of go routines on a per instance basis.

Alexey «AlekSi» Palazhchenko

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/B6CCB2E2-F50B-4FCC-ADEF-3D69F40BCF86%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Wilkie]

Do you think it would be possible to have this directly read the 1.x data from disk and not rely on the remote read API? I’d love to have a stab at that but won’t have time this week...

On Sat, 11 Nov 2017 at 09:05, Ben Kochie <sup...@gmail.com> wrote:

Thanks, this is a pretty interesting migration tool.

I did some testing, it seems to work, but only when the server is pretty small, and the index of all metrics can be queried easily.  This seems to top out at around 10,000 metrics before it got really slow.

In talking with some of the other Prometheus developers, it would help to query the list of all instances and shard the work over a number of go routines on a per instance basis.

On Thu, Nov 9, 2017 at 10:43 PM, Alexey “AlekSi” Palazhchenko <alexey.pa...@gmail.com> wrote:

Hi,

I wrote a small tool to migrate all data from running Prometheus 1.8 instance to a new tsdb directory for Prometheus 2.0. It can be used to move all data to the new Prometheus instead of running two versions simultaneously. It is here: https://github.com/Percona-Lab/prom-migrate

It wasn't properly tested yet, etc. etc. Please do try it, but do not remove your old data just yet. I would be especially thankful if someone from Prometheus team will take a look if I miss some important tsdb options, tweaks or something else.

–-–

Alexey «AlekSi» Palazhchenko

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/B6CCB2E2-F50B-4FCC-ADEF-3D69F40BCF86%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmp3Q1RQ8SjYKLjtPx12PqqFF6GYPpyJRjzTHkmQErDJqQ%40mail.gmail.com.

[Tobias Schmidt]

As most others, SoundCloud would have an interest in such a tool as well ;-) I have a bit too many other things on my plate already, but we would happy to test and provide detailed feedback.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAB58Z12qovAzKqeXgsVU8FG2M%3DER-QhnB2jdwV10YDNgbvwCAg%40mail.gmail.com.

[Ben Kochie]

I think Julius is working on a PoC for direct disk conversation

On Nov 11, 2017 19:51, "Tobias Schmidt" <tob...@gmail.com> wrote:

As most others, SoundCloud would have an interest in such a tool as well ;-) I have a bit too many other things on my plate already, but we would happy to test and provide detailed feedback.

On Sat, Nov 11, 2017, 19:40 Tom Wilkie <tom.w...@gmail.com> wrote:

Do you think it would be possible to have this directly read the 1.x data from disk and not rely on the remote read API? I’d love to have a stab at that but won’t have time this week...

On Sat, 11 Nov 2017 at 09:05, Ben Kochie <sup...@gmail.com> wrote:

Thanks, this is a pretty interesting migration tool.

I did some testing, it seems to work, but only when the server is pretty small, and the index of all metrics can be queried easily.  This seems to top out at around 10,000 metrics before it got really slow.

In talking with some of the other Prometheus developers, it would help to query the list of all instances and shard the work over a number of go routines on a per instance basis.

On Thu, Nov 9, 2017 at 10:43 PM, Alexey “AlekSi” Palazhchenko <alexey.pa...@gmail.com> wrote:

Hi,

I wrote a small tool to migrate all data from running Prometheus 1.8 instance to a new tsdb directory for Prometheus 2.0. It can be used to move all data to the new Prometheus instead of running two versions simultaneously. It is here: https://github.com/Percona-Lab/prom-migrate

It wasn't properly tested yet, etc. etc. Please do try it, but do not remove your old data just yet. I would be especially thankful if someone from Prometheus team will take a look if I miss some important tsdb options, tweaks or something else.

–-–

Alexey «AlekSi» Palazhchenko

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/B6CCB2E2-F50B-4FCC-ADEF-3D69F40BCF86%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmp3Q1RQ8SjYKLjtPx12PqqFF6GYPpyJRjzTHkmQErDJqQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

[Julius Volz]

Yeah, this just prompted me to throw my first (naive) approach onto GitHub

https://github.com/juliusv/prom-data-migrator

It queries *all* series at configurable step widths from the old storage and then inserts them into the new one. Since it queries all series, I imagine it wouldn't work well on very large servers, so the selection should probably be sharded (e.g. on instance label?), and perhaps parallelized.

Does anyone have a large-ish data directory for me to play around with? A couple of hundred MB up to few GB would be ideal for trying things out locally...

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqeeQRsYZUuxK4edHWrgA3uQa2gr25f3Uk4f5sV8aW91A%40mail.gmail.com.

[David Karlsen]

It queries *all* series at configurable step widths from the old storage and then inserts them into the new one. Since it queries all series, I imagine it wouldn't work well on very large servers, so the selection should probably be sharded (e.g. on instance label?), and perhaps parallelized.

Does anyone have a large-ish data directory for me to play around with? A couple of hundred MB up to few GB would be ideal for trying things out locally...

I can't really hand the dataset out - but I could run migrations to test.

Our test-instance has 67G worth of data:

du -sh /srv/prometheus/data/

67G     /srv/prometheus/data/

--

--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[Julius Volz]

Thanks, you could try it out, but I'm pretty sure at it's current completely non-optimized state it would just choke on that much data.

I'm getting datasets from someone tomorrow, so I'll be able to study and hopefully optimize the tool somewhat. But I'm not sure it can ever be really performant for large datasets.