Prometheus Remote Write/Read + InfluxDB

[Josh Minor]

Hello,

I am playing with a system composed of netdata(collects node metrics), prometheus, influxdb, and grafana. In evaluating the prometheus remote read/write api, I thought it may be possible to provide metrics to prometheus via a "push" like method by writing them into influxdb directly using the prometheus remote write feature for netdata. This is done in netdata by specifying a remote write URL such as "http://influxdb:8086/api/v1/prom/write?db=prometheus", and I figured that prometheus would pick up on the new metric data available to it in the influxdb database. It does not seem like this is the case however, as no data shows up in the prometheus UI, although i've verified it does exist in the influxdb prometheus database. I have tested the "remote_read" option with "read_recent" set to both true and false to no avail.

So knowing this, is it ever possible to subvert prometheus scraping, and just push data directly into remote storage where prometheus can reference it later? Or does prometheus maintain it's own view of the metrics available in the system, and exclusively use the remote storage as a supplement to it's own local storage? It seems that in the prometheus UI search bar, there is an autocomplete feature which must query over all available metrics, I figured that with "remote_read" set to true, it would query the influxdb prometheus database for all measurements as a supplement to what it knows locally, but this wasn't the case.

Thanks,

Josh

[Brian Candler]

For doing this sort of stuff, VictoriaMetrics might suit you better.  It implements the same PromQL and remote_write APIs as prometheus, but you can push and backfill data to it in various formats - whilst still letting prometheus remote_write to it as well.  You can also choose to use its own scraper and alerter if you want to replace the whole stack.