InfluxQL queries for Data collected by Prometheus

[Yogesh Jadhav]

Hi All,
I am using Prometheus with SNMP exporter to fetch metrics from network devices like router, switch, etc. After creating dashboards in Grafana, I realized that Prometheus does not provide downsampling for long term storage over years such that footprint is fixed and small. So InfluxDB is installed along with Prometheus with the intention of using it for long term storage for Prometheus data employing different RPs & CQs for downsampling data depending on retention period.

I want Grafana to now display data from InfluxDB instead of Prometheus.

I find porting PromQL dashboards to InfluxQL for Prometheus collected data stored into InfluxDB a very challenging task. For example, In all dashboards, I use template variable Job (say with value=ubiquiti) to select from available devices and then use another template variable Instance ( value=ip.address.of.device) to select particular device among the pool of Ubiquiti devices.

But in InfluxQL I can either query Jobs or Instances not both.

show tag values from "ifInOctets" with key="instance" 
show tag values from "ifInOctets" with key="job"

Any pointers are welcome.

I have enabled both remote read and write in Prometheus for InfluxDB. Is it possible to keep existing PromQL dashboards with the ability to read historic and current data simultaneously?

Thanks in advance

[Brian Candler]

If you are using prometheus' remote_write feature to write to influxdb, then I would have thought you would be able to query it via prometheus' remote_read via the same PromQL interface - that is, you should not have to write influxQLqueries (I'm not sure why you've found this).  But you may end up having to write recording rules to do downsampling.

I'd suggest you consider using something else for long term storage which supports the same PromQL language natively, so your dashboards don't have to change.  Popular options include VictoriaMetrics (very easy to set up) and Thanos (a bit more work).

Thanos includes down-sampling as a built-in feature.  With VictoriaMetrics you can scrape the /federate endpoint as described here.  So either way, you don't have to write recording rules.

[Yogesh Jadhav]

Thanks for the reply, Brian.

I don't mind writing recording rules. My only worry is will I be able to use Grafana dashboards without any issues (especially when we access data over years) as InfluxDB stores it internally in a different format.

Our's is a small organization with a small network ( a few thousand devices to monitor) and a small team ( I am the only one working on this). Keeping disk space usage (relatively) small and fixed even after storing data for years is  

Thanos has a lot of overheads, it supports only object/cloud storage. I don't think it supports custom resolutions/retentions. The most important point is it's downsampling is intended for query performance optimization instead of reducing storage footprint. In fact, it increases it by 3 times.

VictoriaMetrics can't be considered for production systems (yet) as it is basically a one-man show. Also similar to other tools like Thanos, cortex, m3, etc, it has installation and maintenance overheads. 

Are you suggesting that a sophisticated long term solution setup is better than using an external DB with remote read/write?

[Aliaksandr Valialkin]

Hi Yogesh,

On Sun, May 10, 2020 at 3:51 PM Yogesh Jadhav <yog...@denovogroup.org> wrote:

Thanks for the reply, Brian.

I don't mind writing recording rules. My only worry is will I be able to use Grafana dashboards without any issues (especially when we access data over years) as InfluxDB stores it internally in a different format.

Our's is a small organization with a small network ( a few thousand devices to monitor) and a small team ( I am the only one working on this). Keeping disk space usage (relatively) small and fixed even after storing data for years is  

Thanos has a lot of overheads, it supports only object/cloud storage. I don't think it supports custom resolutions/retentions. The most important point is it's downsampling is intended for query performance optimization instead of reducing storage footprint. In fact, it increases it by 3 times.

VictoriaMetrics can't be considered for production systems (yet) as it is basically a one-man show. Also similar to other tools like Thanos, cortex, m3, etc, it has installation and maintenance overheads. 

VictoriaMetrics is successfully used in production by many happy users - see https://github.com/VictoriaMetrics/VictoriaMetrics/wiki/CaseStudies .

One of the most important selling points for VictoriaMetrics is easy installation and maintenance. Could you share more details on installation and maintenance overhead related to VictoriaMetrics?