migrating from 1.x to 2.x

[David Karlsen]

I am about to move to 2.x, and I use the guide: https://prometheus.io/docs/prometheus/2.0/migration/

but I still wonder about storage params:

existing 1.x:

-storage.local.path=/prometheus -storage.local.retention={{ prometheus.settings.retension }} -storage.local.target-heap-size={{ prometheus.settings.targetHeapSizeInBytes }}"

Looking at guide:

https://prometheus.io/docs/prometheus/2.0/migration/#storage

it does not mention flags for retension or heap - how do I set these in 2.x and have they been renamed to storage.tstb ?

coexistence:

The data format in Prometheus 2.0 has completely changed and is not backwards compatible with 1.8. To retain access to your historic monitoring data we recommend you run a non-scraping Prometheus instance running at least version 1.8.1 in parallel with your Prometheus 2.0 instance, and have the new server read existing data from the old one via the remote read protocol.

Previously on the mailinglist I have seen that the recommended approach was to run 2.x in paralell with 1.x, and do a switchover when 2.x had filled with data - is this now replaced by running 2.x as primary and scraping version and letting it do "remote read" towards 1.x?

What is the remote read stuff? Will grafana to v2, read what it has, and transparently v2 will pass a query to v1 and merge the results on the fly?

Any other things I should think of that's not in the migration guide?

[David Karlsen]

https://prometheus.io/docs/prometheus/2.0/storage/

I missed that this is versioned now - so I have settings for retension and path. How about memory use?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fde7375f-33b0-4baf-ab45-d13e61a166bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[Conor Broderick]

The retention flag is now --storage.tsdb.retention and the heap-size flag has been removed with Prometheus 2.0.

You can view all of the available flags running Prometheus with --help.

What is the remote read stuff?

Remote read allows Prometheus to transparently receive samples from another data source and is primarily intended for long term storage.

You can find a guide on accessing data from Prometheus 1.x in Prometheus 2.0 here.

 

Will grafana to v2, read what it has, and transparently v2 will pass a query to v1 and merge the results on the fly?

Yes this will work.

On 29 November 2017 at 13:29, Conor Broderick <conor.b...@robustperception.io> wrote:

The retention flag is now --storage.tsdb.retention and the heap-size flag has been removed with Prometheus 2.0.

You can view all of the available flags running Prometheus with --help.

What is the remote read stuff?

Remote read allows Prometheus to transparently receive samples from another data source and is primarily intended for long term storage.

You can find a guide on accessing data from Prometheus 1.x in Prometheus 2.0 here.

 

Will grafana to v2, read what it has, and transparently v2 will pass a query to v1 and merge the results on the fly?

Yes this will work.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fde7375f-33b0-4baf-ab45-d13e61a166bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

--

Conor Broderick

www.robustperception.io

[David Karlsen]

Hm - I am not seing historical data in grafana at all.

I see the /config section having 

remote_read:
- url: http://prometheus1:9090/api/v1/read
  remote_timeout: 1m
  read_recent: true

in my new server config. It can connect ok. I am not seing any errors in either the v1 or v2 logs.

I can browse the v1 and v2 consoles and see the data there through queries.

The external_labels are the same.

Any tips?

2017-11-29 14:29 GMT+01:00 Conor Broderick <conor.b...@robustperception.io>:

The retention flag is now --storage.tsdb.retention and the heap-size flag has been removed with Prometheus 2.0.

You can view all of the available flags running Prometheus with --help.

What is the remote read stuff?

Remote read allows Prometheus to transparently receive samples from another data source and is primarily intended for long term storage.

You can find a guide on accessing data from Prometheus 1.x in Prometheus 2.0 here.

 

Will grafana to v2, read what it has, and transparently v2 will pass a query to v1 and merge the results on the fly?

Yes this will work.

[David Karlsen]

What is even stranger is that some of my dashboards work OK, while others are missing the data that should appear over remote-read.

Also if I tcpdump and netstat I see queries are running seemingly OK for both cases.

[David Karlsen]

Any suggestions? I'm a bit stuck in the upgrade as I basically cannot get the v1 metrics to show.

[irfan....@gmail.com]

Hi David et all,

I am facing the same issue. Followed the Prometheus 1->2 migration guide and the 'remote_read' doesn't seem to be working as expected. e.g. sending graph query for metric 'up' produces a merged response (as expected), however sending 'up{job="server}' only returns results from Prometheus2 instance. I've tried playing with the 'read_recent' toggle, but 1) made no difference 2) don't really understand it's use (pointer to any documentation on this would be much appreciated.)

Has the guide worked seamlessly for anyone else?

Regards,
Irfan

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fde7375f-33b0-4baf-ab45-d13e61a166bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Conor Broderick

www.robustperception.io

--

Conor Broderick

www.robustperception.io

--

--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

[sim...@ooyala.com]

Hello,

I've just spent a good while trying to figure this one out - looks like if the external_labels are _different_ between the 1.8.x and 2.0.0 prometheus, remote_read doesn't work. Once I've changed external_labels to be identical between both instances I can see all of my historical metrics.

[David Karlsen]

Interesting!

I have external_labels in prometheus.yml that match in both v1 and v2 - but one of these labels have the same name and value as I have in some of my targets targets defined by service-discovery - if I remove this label from prometheus.yml it works.

Is this a bug?

Is it not valid to have an external_label key/value that matches what's already set for some targets?

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ae4a2a49-a8fe-4fcd-b92e-cdca782676b1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Brian Brazil]

On 6 December 2017 at 20:52, David Karlsen <davidk...@gmail.com> wrote:

Interesting!

I have external_labels in prometheus.yml that match in both v1 and v2 - but one of these labels have the same name and value as I have in some of my targets targets defined by service-discovery - if I remove this label from prometheus.yml it works.

Is this a bug?

 

Is it not valid to have an external_label key/value that matches what's already set for some targets?

It's valid, but you should avoid it as it causes oddities like the above.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGO7Ob1bJGFdLi7LsmpOLiopJAKGkzknzkxvuNr%2BSisvn%3DQNTg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io