Prometheus not writing data via remote_storage_adapter into InfluxDB

[Junaid Subhani]

My InfluxDB is setup properly ::

# curl -XPOST 'http://172.29.225.32:8086/query?pretty=true' --data-urlencode "q=SHOW DATABASES"
{
    "results": [
        {
            "statement_id": 0,
            "series": [
                {
                    "name": "databases",
                    "columns": [
                        "name"
                    ],
                    "values": [
                        [
                            "_internal"
                        ],
                        [
                            "prometheus"
                        ],
                        [
                            "mydb"
                        ]
                    ]
                }
            ]
        }
    ]
}

Next I started the Remote Storage Adapter

./remote_storage_adapter -influxdb-url=http://localhost:8086/ -influxdb.database=prometheus -influxdb.retention-policy=autogen

I can see that the port is now up and running ::

# netstat -tulpn | grep 9201
tcp6       0      0 :::9201                 :::*                    LISTEN      14322/./remote_stor

Next I configured my prometeus.yaml file accordingly ::

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.


# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
  remote_write:
    - url: "http://localhost:9201/write"


# Remote read configuration (for InfluxDB only at the moment).
  remote_read:
    - url: "http://localhost:9201/read"

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

And I see that the Prometheus process starts up successfully ::

# ./prometheus -config.file=prometheus.yml -log.level=debug
INFO[0000] Starting prometheus (version=1.6.3, branch=master, revision=c580b60c67f2c5f6b638c3322161bcdf6d68d7fc)  source=main.go:88
INFO[0000] Build context (go=go1.8.1, user=root@a6410e65f5c7, date=20170522-09:15:06)  source=main.go:89
INFO[0000] Loading configuration file prometheus.yml     source=main.go:251
INFO[0000] Loading series map and head chunks...         source=storage.go:421
INFO[0000] 3513 series loaded.                           source=storage.go:432
INFO[0000] Starting target manager...                    source=targetmanager.go:61
INFO[0000] Listening on :9090                            source=web.go:259
INFO[0300] Checkpointing in-memory metrics and chunks...  source=persistence.go:633
INFO[0300] Done checkpointing in-memory metrics and chunks in 279.603534ms.  source=persistence.go:665
 

But looking at the DB, it is still empty. No data is being written into InfluxDB. Is there some config that I have missed ? If not, what troubleshooting mechanism can I use to find out whats wrong ? 

[Julius Volz]

Are you using the remote_storage_adapter built from latest master? The
problem is that 1.6.3 doesn't include a change to the generic remote
read/write protocol yet that is already present in master (both in
Prometheus and the adapter).

You'll either have to use Prometheus *and* the adapter both built from
master, or build the adapter from the 1.6.3 branch as well, OR wait
for Prometheus 1.7 to be released to work with the current master
adapter version again.

However, I think Prometheus should log errors if that is the problem?

Also, are you sure that the indentation is fine in your config file?
The remote_write and remote_read sections are indented on the global
level, but they shouldn't be...

> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a8389918-b459-4e4a-b81a-4a5ac847217f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Junaid Subhani]

Actually I just build both Prometeus & remote_storage_adapter today from the repo.

git clone https://github.com/prometheus/prometheus.git


# ./prometheus -version
prometheus, version 1.6.2 (branch: master, revision: e0f046396a163913ee3c40030a7f42fbdcf1cbd5)
  build user:       root@supportserver
  build date:       20170530-16:43:14
  go version:       go1.8.3

Ive started remote_storage_adapter using ::

# /usr/local/go/src/github.com/prometheus/prometheus/documentation/examples/remote_storage/remote_storage_adapter/remote_storage_adapter -influxdb-url=http://localhost:8086/ -influxdb.database=prometheus -influxdb.retention-policy=autogen -log.level=debug -log.format=logger:stdout?json=true

As far as remote_write and remote_read sections sections are concerened, they are indeed under global_config. Are they meant to be somewhere else ? As per documentation on the github website, all that it says is ::

Configuring Prometheus

To configure Prometheus to send samples to this binary, add the following to your prometheus.yml:

# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
remote_write:
  - url: "http://localhost:9201/write"

# Remote read configuration (for InfluxDB only at the moment).
remote_read:
  - url: "http://localhost:9201/read"

[Junaid Subhani]

I just had a closer look at the documentation. https://prometheus.io/docs/operating/configuration/#<remote_write>

I indented remote_write to the left (no longer under global) and now it works ! Thanks for your help !

[Julius Volz]

Thank you, that actually made me question why Prometheus didn't
complain about the config, and I found a bug. Fix:
https://github.com/prometheus/prometheus/pull/2783

On Tue, May 30, 2017 at 7:41 PM, Junaid Subhani

> https://groups.google.com/d/msgid/prometheus-users/dc781735-f45b-4c39-afb8-bea48fe9a0cb%40googlegroups.com.