Query Multiple Metrics?

[efa...@gmail.com]

Hey,

So I am trying to query multiple metrics at once.  I am using {__name__=~"apcupsd.+voltage.+",instance="isoflurane"}  to query a bunch of voltages from my UPS via netdata.  The issue is that the voltages are mV and I need them in V.  So I tried to just do {__name__=~"apcupsd.+voltage.+",instance="isoflurane"} / 100 .... but when I do that the result of the query seems to loose all of the metric names.....   Am I doing something wrong?

-Eric

[Ben Kochie]

It looks like you're using matching to get the metric names instead of using the actual metric name like this:

apcupsd_voltage{instance="isoflurane"}

This is probably why you're having issues.

It also looks like your instances names are missing fqdn:port as is recommended.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bda5506a-3139-493a-a8f5-bb21b09b4324%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[efa...@gmail.com]

What I was trying to do was query multiple metrics at once in a single query from Grafana.  Is that possible or do I need a different query per metric?  Basically I have a set of metrics apcupsd_*_voltage_* etc.... that I want to graph all at once, but I need to divide them by 100.  According to the documentation I can do the query that way... and it works correctly.... but the issue is when I do the / 100... 

-Eric

On Wednesday, June 28, 2017 at 2:06:52 AM UTC-4, Ben Kochie wrote:

It looks like you're using matching to get the metric names instead of using the actual metric name like this:

apcupsd_voltage{instance="isoflurane"}

This is probably why you're having issues.

It also looks like your instances names are missing fqdn:port as is recommended.

On Wed, Jun 28, 2017 at 2:58 AM, <efa...@gmail.com> wrote:

Hey,

So I am trying to query multiple metrics at once.  I am using {__name__=~"apcupsd.+voltage.+",instance="isoflurane"}  to query a bunch of voltages from my UPS via netdata.  The issue is that the voltages are mV and I need them in V.  So I tried to just do {__name__=~"apcupsd.+voltage.+",instance="isoflurane"} / 100 .... but when I do that the result of the query seems to loose all of the metric names.....   Am I doing something wrong?

-Eric

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Nicholas Capo]

Grafana usually handles the unit conversions for me.

For example if I graph a _seconds metric I will set the Grafana axis unit to seconds. Grafana will then display it in minutes/hours/day/etc depending on the value.

I have a feeling you can just set the unit to mV and Grafana will do the rest.

Nicholas

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e991dca2-5e19-4886-8a4e-d37228ebac26%40googlegroups.com.

[Ben Kochie]

Right, but I'm wondering what is between those asterisks that is causing you to have to do it this way.  What I'm getting at is you possibly have a metric name / labeling issue that can be fixed, completely eliminating the root cause of the problem.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e991dca2-5e19-4886-8a4e-d37228ebac26%40googlegroups.com.

[Brian Brazil]

On 30 June 2017 at 18:21, Ben Kochie <sup...@gmail.com> wrote:

Right, but I'm wondering what is between those asterisks that is causing you to have to do it this way.  What I'm getting at is you possibly have a metric name / labeling issue that can be fixed, completely eliminating the root cause of the problem.

https://www.robustperception.io/whats-in-a-__name__/ explains what's going on. https://www.robustperception.io/extracting-labels-from-legacy-metric-names/ is a workaround, but fixing the data at source is best.

Brian

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo6UeRKPi%3D2Mwmj2OSGNDZMQg1ys3RTCRFcUmhtw%3DoT5w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Eric Faden]

Thanks for the information.  That's definately the problem.  These metrics come from NetData so I don't have a lot of control over them.

-Eric

-Eric

[Ben Kochie]

Yes, but it's pretty easy to fix them.  How exactly (which exporter) are you ingesting these?

This can either be fixed with the exporter, or with a relabel config.  

On Sat, Jul 1, 2017 at 3:20 AM, Eric Faden <efa...@gmail.com> wrote:

Thanks for the information.  That's definately the problem.  These metrics come from NetData so I don't have a lot of control over them.

-Eric

-Eric

[Eric Faden]

These are being directly exported via NetData.  See: https://github.com/firehol/netdata/wiki/Using-Netdata-with-Prometheus

Basically in my setup I have a single NetData instance which gathers data from 3 other computers.    NetData exports the metrics via the URL

http://{ip_of_vm}:19999/api/v1/allmetrics?format=prometheus_all_hosts

I can easily fix the instance name not being fully qualified since that is just whatever I set the netdata hostname to.  

I have little control over the names themselves.  And actually looking through the rest of the metrics via netdata it seems that most, if not all, don't really utilize labels correctly and instead have metric names e.g.

system_cpu_cpu0   instead of system_cpu(cpu="cpu0")

I posted on the netdata github and am actually looking into their base to see if I could modify their code to give better labels/names. 

Relabeling would be difficult due to the number of metrics and the fact that they don't have a format that would be "easily" regexed.

I can post the output if your curious.

-Eric

-Eric

You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/fOGpOd97tAo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrpkjx%3D0ZZU2F%2BqN8YGkvbaFQKaeOKXYcEM_xqxtafgag%40mail.gmail.com.

[Brian Brazil]

On 1 July 2017 at 12:43, Eric Faden <efa...@gmail.com> wrote:

These are being directly exported via NetData.  See: https://github.com/firehol/netdata/wiki/Using-Netdata-with-Prometheus

Basically in my setup I have a single NetData instance which gathers data from 3 other computers.    NetData exports the metrics via the URL

http://{ip_of_vm}:19999/api/v1/allmetrics?format=prometheus_all_hosts

I can easily fix the instance name not being fully qualified since that is just whatever I set the netdata hostname to.  

I have little control over the names themselves.  And actually looking through the rest of the metrics via netdata it seems that most, if not all, don't really utilize labels correctly and instead have metric names e.g.

system_cpu_cpu0   instead of system_cpu(cpu="cpu0")

I posted on the netdata github and am actually looking into their base to see if I could modify their code to give better labels/names. 

Relabeling would be difficult due to the number of metrics and the fact that they don't have a format that would be "easily" regexed.

I advised them on their Prometheus output. The challenge is that netdata doesn't have a clean notion of labels, some are what we'd consider labels others are what we'd put in the metric name.  In such situations, the safest thing is not to export labels.

For general machine stats, the node_exporter would be recommended as that's designed with the Prometheus data model in mind.

Brian

--

Brian Brazil

www.robustperception.io

[Eric Faden]

Yeah.  I think I may convert over and use that.  And then use NetData and relabel just the ones I need.

-Eric

-Eric

[Ben Kochie]

Thanks for the info, it's really helpful to understand where the data is coming from.

If there are any system metrics that the node exporter is missing, I would love to know what you find useful.  We can add these to the exporter.

On Jul 1, 2017 1:55 PM, "Eric Faden" <efa...@gmail.com> wrote:

Yeah.  I think I may convert over and use that.  And then use NetData and relabel just the ones I need.

-Eric

-Eric

[Eric Faden]

I'll let you know.  Right now it is the metrics from the UPS that I'm trying to keep track of.  But the hosts are running proxmox and ideally I'd like to be able to keep track of disk usage, cpu, etc for the VMs.  Also IPMI and uptime for the machine itself.

-Eric

[Eric Faden]

Similarly the instance names show up as hosname_domain_tld instead of using periods when I tried to switch to TLDs.

-Eric