Save Prometheus Data to external DB

[Junaid Subhani]

I am trying to setup a HA/backup solution for Prometheus.

I would like to write all the data Prometheus generates into an external DB like postgres/mysql/influxDB etc.

Does Prometheus provide any such option to connect to an external DB ? Right now I see the data being written in ~/prometheus-1.6.3.linux-amd64/data/ folder. 

[Ben Kochie]

For HA, you should run two identical Prometheus server that both scrape all targets.

For backup, your best bet is to use the generic write API.

https://www.robustperception.io/using-the-remote-write-path/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/07eb8070-593a-4e3d-a09e-d12d01772052%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julius Volz]

Caution: the latest write adapter in HEAD already has an update to the protocol that will require a version of Prometheus built from master to work. Or wait until Prometheus 1.7 is released :)

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpGat4TC6A-gvga911MCUshEd9HnhzRPxMT7cibnuT9hg%40mail.gmail.com.

[Junaid Subhani]

Thanks for the recommendation guys. Right now Im looking into the option of writing the data into InfluxDB. Checking the scattered documentation now :) 

On Tuesday, 30 May 2017 08:53:18 UTC-4, Julius Volz wrote:

Caution: the latest write adapter in HEAD already has an update to the protocol that will require a version of Prometheus built from master to work. Or wait until Prometheus 1.7 is released :)

On Tue, May 30, 2017 at 2:46 PM, Ben Kochie <sup...@gmail.com> wrote:

For HA, you should run two identical Prometheus server that both scrape all targets.

For backup, your best bet is to use the generic write API.

https://www.robustperception.io/using-the-remote-write-path/

On Tue, May 30, 2017 at 2:41 PM, Junaid Subhani <ijunaid...@gmail.com> wrote:

I am trying to setup a HA/backup solution for Prometheus.

I would like to write all the data Prometheus generates into an external DB like postgres/mysql/influxDB etc.

Does Prometheus provide any such option to connect to an external DB ? Right now I see the data being written in ~/prometheus-1.6.3.linux-amd64/data/ folder. 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/07eb8070-593a-4e3d-a09e-d12d01772052%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Julius Volz]

On Tue, May 30, 2017 at 3:41 PM, Junaid Subhani
<ijunaid...@gmail.com> wrote:
> Thanks for the recommendation guys. Right now Im looking into the option of
> writing the data into InfluxDB. Checking the scattered documentation now :)

See also:

https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage/remote_storage_adapter
https://www.youtube.com/watch?v=MuHkckZg5L0

:)

[Junaid Subhani]

Thanks Julius.

Im working on it now. I guess I am having a little trouble starting up the project.

I did ::

wget https://github.com/prometheus/prometheus/archive/master.zip

unzip master.zip
cd prometheus-master/documentation/examples/remote_storage/remote_storage_adapter/

go build

This is what I get ::

# pwd

/root/influxdb/prometheus-master/documentation/examples/remote_storage/remote_storage_adapter

# go version

go version go1.8.3 linux/amd64

# go build
main.go:28:2: cannot find package "github.com/gogo/protobuf/proto" in any of:
 /usr/local/go/src/github.com/gogo/protobuf/proto (from $GOROOT)
 /root/go/src/github.com/gogo/protobuf/proto (from $GOPATH)
main.go:29:2: cannot find package "github.com/golang/snappy" in any of:
 /usr/local/go/src/github.com/golang/snappy (from $GOROOT)
 /root/go/src/github.com/golang/snappy (from $GOPATH)
main.go:34:2: cannot find package "github.com/influxdata/influxdb/client/v2" in any of:
 /usr/local/go/src/github.com/influxdata/influxdb/client/v2 (from $GOROOT)
 /root/go/src/github.com/influxdata/influxdb/client/v2 (from $GOPATH)
main.go:30:2: cannot find package "github.com/prometheus/client_golang/prometheus" in any of:
 /usr/local/go/src/github.com/prometheus/client_golang/prometheus (from $GOROOT)
 /root/go/src/github.com/prometheus/client_golang/prometheus (from $GOPATH)
main.go:31:2: cannot find package "github.com/prometheus/common/log" in any of:
 /usr/local/go/src/github.com/prometheus/common/log (from $GOROOT)
 /root/go/src/github.com/prometheus/common/log (from $GOPATH)
main.go:32:2: cannot find package "github.com/prometheus/common/model" in any of:
 /usr/local/go/src/github.com/prometheus/common/model (from $GOROOT)
 /root/go/src/github.com/prometheus/common/model (from $GOPATH)
main.go:36:2: cannot find package "github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/graphite" in any of:
 /usr/local/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/graphite (from $GOROOT)
 /root/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/graphite (from $GOPATH)
main.go:37:2: cannot find package "github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/influxdb" in any of:
 /usr/local/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/influxdb (from $GOROOT)
 /root/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/influxdb (from $GOPATH)
main.go:38:2: cannot find package "github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/opentsdb" in any of:
 /usr/local/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/opentsdb (from $GOROOT)
 /root/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/opentsdb (from $GOPATH)
main.go:39:2: cannot find package "github.com/prometheus/prometheus/storage/remote" in any of:
 /usr/local/go/src/github.com/prometheus/prometheus/storage/remote (from $GOROOT)
 /root/go/src/github.com/prometheus/prometheus/storage/remote (from $GOPATH)

Would you be able to guide me as to what am I doing wrong here ? Sorry this is the first time Im trying to setup a project from GITHUB.

[Junaid Subhani]

Got this sorted out. Just needed to give the proper PATH :) 

[pradip.ra...@gmail.com]

Hi Junaid,

I saw that you were able to write Prometheus data to Influxdb. I am wondering why i am not able to get data in Influxdb. I have configured Prometheus .yml file similar to yours and when i run Prometheus i don't see any error but also i don't see data written in Influxdb. Appreciate if you could give some direction on this issue.

Thanks,

Pradip.

[Julius Volz]

Are you using compatible versions of Prometheus + the read/write adapter, as I wrote a couple replies earlier in this thread?

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/90f62503-0ba5-4d8a-90ed-4b2ba97aadd0%40googlegroups.com.

[pradip.ra...@gmail.com]

Hi Julius,

Thank you very much.

This is what i have done so far:

Installed prometheus:

prometheus, version 2.0.0-alpha.2 (branch: master, revision: ab0ce4a8d9858956b37b545cfb84bb4edb5d7776)

  build user:       root@fda0efffe2cf

  build date:       20170524-15:34:23

  go version:       go1.8.1

# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  remote_write:

    - url: "http://xx.xx.xx.xx:8086/write"

  remote_read:

    - url: "http://xx.xx.xx.xx:8086/read"

# scrape_timeout is set to the global default (10s).

Installed Influxdb:

$ influx

Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.

Connected to http://localhost:8086 version 1.0.0

InfluxDB shell version: 1.0.0

Installed adapter 

https://github.com/prometheus/prometheus/tree/master/documentation/examples/remote_storage/remote_storage_adapter(Adapter installed in Influxdb node)

ERROR:

ro...@ip-192.168.1.1:~/.go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/example_write_adapter# ./remote_storage_adapter -influxdb-url=http://localhost:8086/ -influxdb.database=prometheus -influxdb.retention-policy=autogen

bash: ./remote_storage_adapter: No such file or directory

On Thursday, June 1, 2017 at 2:16:42 PM UTC-7, Julius Volz wrote:

Are you using compatible versions of Prometheus + the read/write adapter, as I wrote a couple replies earlier in this thread?

On Thu, Jun 1, 2017 at 11:09 PM, <pradip.ra...@gmail.com> wrote:

Hi Junaid,

I saw that you were able to write Prometheus data to Influxdb. I am wondering why i am not able to get data in Influxdb. I have configured Prometheus .yml file similar to yours and when i run Prometheus i don't see any error but also i don't see data written in Influxdb. Appreciate if you could give some direction on this issue.

Thanks,

Pradip.

On Tuesday, May 30, 2017 at 5:41:21 AM UTC-7, Junaid Subhani wrote:

I am trying to setup a HA/backup solution for Prometheus.

I would like to write all the data Prometheus generates into an external DB like postgres/mysql/influxDB etc.

Does Prometheus provide any such option to connect to an external DB ? Right now I see the data being written in ~/prometheus-1.6.3.linux-amd64/data/ folder. 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Brian Brazil]

On 2 June 2017 at 17:40, <pradip.ra...@gmail.com> wrote:

Hi Julius,

Thank you very much.

This is what i have done so far:

Installed prometheus:

prometheus, version 2.0.0-alpha.2 (branch: master, revision: ab0ce4a8d9858956b37b545cfb84bb4edb5d7776)

2.0 does not yet support remote read/write.

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/84653536-ae41-44b8-ba9c-266a6c086684%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[pradip.ra...@gmail.com]

Hi Brian,

I even tried for version 1.6.3 and still no data in influxdb.

:~/Prometheus/prometheus-1.6.3.linux-amd64$ ./prometheus -version

prometheus, version 1.6.3

  build user:       root@a6410e65f5c7

  build date:       20170522-09:15:06

  go version:       go1.8.1

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/84653536-ae41-44b8-ba9c-266a6c086684%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Junaid Subhani]

Based on you ERROR I see ::

root@ip-192.168.1.1:~/.go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/example_write_adapter# ./remote_storage_adapter -influxdb-url=http://localhost:8086/ -influxdb.database=prometheus -influxdb.retention-policy=autogen

bash: ./remote_storage_adapter: No such file or directory

Did you create this file ? Seems like this file does not exist. You will have to goto the Source directory and run the go build command.

[Junaid Subhani]

Secondly 

global:



  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  remote_write:

    - url: "http://xx.xx.xx.xx:8086/write"

  remote_read:

    - url: "http://xx.xx.xx.xx:8086/read"

This is incorrect. remote_read and remote_write should not be part of global. Try below config pattern.

global:



  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

remote_write:

  - url: "http://xx.xx.xx.xx:8086/write"

remote_read:

  - url: "http://xx.xx.xx.xx:8086/read"

[pradip.ra...@gmail.com]

Hi Junaid,

Thanks for your reply.

Actually i didn't create any file for remote storage adapter. I copied it from github.

Thanks,

Pradip.

[Junaid Subhani]

Im going to assume you downloaded the GIT repository. The GIT repository does not have this file. You need to build it. 

[jer...@doupe.com]

Is there any discussion or work being done on updating the existing adapter to work with 2.0 as of yet?

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/84653536-ae41-44b8-ba9c-266a6c086684%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Tom Wilkie]

> Is there any discussion or work being done on updating the existing adapter to work with 2.0 as of yet?

There is a PR to add it: https://github.com/prometheus/prometheus/pull/2850

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/db036609-06b5-4050-92db-0edefe23c10b%40googlegroups.com.