Recommended / best way to expose metrics from multiple sources

[Dominic Cioccarelli]

Hi all,

we have an existing monitoring application that needs to expose metrics (from the multiple monitored hosts) to Prometheus. For various reasons, it is not currently possible to instrument the end points directly, and thus the metrics need to be exposed from the central monitoring system instead.

Given that this is the case, is it better to expose all metrics during a single scrape, whereby the hosts are differentiated by labels or, alternatively, could we write the exporter in such a way that we could differentiate between the hosts by a supplementary path in the URI or an additional parameter?

With the first approach, we would have metrics exposed in a format similar to that below. During a single scrape, a single metric type for multiple endpoints would be exposed:

MBAvailable{component="HOST1234",instance="Memory\\Available Bytes",} 14022.0
MBAvailable{component="HOST6789",instance="Memory\\Available Bytes",} 13777.0
PoolNonpagedMBytes{component="HOST1234",instance="Memory\\Pool Nonpaged Bytes",} 356.47
PoolNonpagedMBytes{component="HOST6789",instance="Memory\\Pool Nonpaged Bytes",} 311.49

With the second approach, we would differentiate between the targets directly in the URL:

http://monitoring_platform:9090/metrics/HOST1234/

Which would then simply deliver the metrics for the specific target (HOST1234):

MBAvailable{instance="Memory\\Available Bytes",} 14022.0
PoolNonpagedMBytes{instance="Memory\\Pool Nonpaged Bytes",} 356.47

Which approach is more in the "Prometheus style"? I'd assume the latter as we are not providing any location information within the metrics and this is being added based on the scrape target.

That said, is it possible to specify different metrics paths per target (or to append an additional path on a per target basis)?

Thanks!

Dominic.

[Brian Brazil]

That's a question really of how this other monitoring system is designed. Usually you'd do it all in one scrape.

In addition, you should not expose a label like instance="Memory\\Available Bytes" as that's duplicate with the metric name, and also the "instance" label should be your hostname here.

--

Brian Brazil

www.robustperception.io

[dmi...@ymail.com]

By the name of metric, it seems that you are looking to expose Windows metrics. We have done this by writing new WMI exporter that can collect metrics from WMI (Windows, NanoServer Windows Containers, etc.) locally or remotely. Thus, we can deploy one exporter container and expose all metrics it collects within the host and its containers as Prometheus target.
Below is example we use for same counter collected by our exporter ("sonar" alias) on host ("host" alias) and another Windows container ("test" alias).

PerfOS_Memory_AvailableMBytes{Name="",alias="host",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.1",instance="10.0.0.21:8500",job="WindowsDockerHost"} 1120
PerfOS_Memory_AvailableMBytes{Name="",alias="sonar",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="localhost",instance="10.0.0.21:8500",job="WindowsDockerHost"} 1119
PerfOS_Memory_AvailableMBytes{Name="",alias="test",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.72",instance="10.0.0.21:8500",job="WindowsDockerHost"} 1120

You are welcome to use exporter out, more details here: https://goo.gl/B5yUCR

[Brian Brazil]

On 13 September 2017 at 00:22, dmitrio via Prometheus Users <promethe...@googlegroups.com> wrote:

By the name of metric, it seems that you are looking to expose Windows metrics. We have done this by writing new WMI exporter that can collect metrics from WMI (Windows, NanoServer Windows Containers, etc.) locally or remotely.  Thus, we can deploy one exporter container and expose all metrics it collects within the host and its containers as Prometheus target.

Ah, if you're after Windows then https://github.com/martinlindhe/wmi_exporter is what is recommended.

 

Below is example we use for same counter collected by our exporter ("sonar" alias) on host ("host" alias) and another Windows container ("test" alias).

PerfOS_Memory_AvailableMBytes{Name="",alias="host",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.1",instance="10.0.0.21:8500",job="WindowsDockerHost"}    1120
PerfOS_Memory_AvailableMBytes{Name="",alias="sonar",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="localhost",instance="10.0.0.21:8500",job="WindowsDockerHost"}      1119
PerfOS_Memory_AvailableMBytes{Name="",alias="test",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.72",instance="10.0.0.21:8500",job="WindowsDockerHost"}   1120

You are welcome to use exporter out, more details here: https://goo.gl/B5yUCR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2323b212-9abd-40b2-b7e7-f25a1de94247%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[dmi...@ymail.com]

Thanks, but we had to write new WMI exporter ( see link in my reply above), because "recommended" one does require code generation for metrics it does not support out of the box. 

On Wednesday, September 13, 2017 at 12:33:42 AM UTC-7, Brian Brazil wrote:

On 13 September 2017 at 00:22, dmitrio via Prometheus Users <promethe...@googlegroups.com> wrote:

By the name of metric, it seems that you are looking to expose Windows metrics. We have done this by writing new WMI exporter that can collect metrics from WMI (Windows, NanoServer Windows Containers, etc.) locally or remotely.  Thus, we can deploy one exporter container and expose all metrics it collects within the host and its containers as Prometheus target.

Ah, if you're after Windows then https://github.com/martinlindhe/wmi_exporter is what is recommended.

 

Below is example we use for same counter collected by our exporter ("sonar" alias) on host ("host" alias) and another Windows container ("test" alias).

PerfOS_Memory_AvailableMBytes{Name="",alias="host",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.1",instance="10.0.0.21:8500",job="WindowsDockerHost"}    1120
PerfOS_Memory_AvailableMBytes{Name="",alias="sonar",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="localhost",instance="10.0.0.21:8500",job="WindowsDockerHost"}      1119
PerfOS_Memory_AvailableMBytes{Name="",alias="test",collector="2E0834B277D9",exported_instance="Win32_PerfFormattedData_PerfOS_Memory",host="172.24.224.72",instance="10.0.0.21:8500",job="WindowsDockerHost"}   1120

You are welcome to use exporter out, more details here: https://goo.gl/B5yUCR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2323b212-9abd-40b2-b7e7-f25a1de94247%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io