Feature Request: SMI-S Exporter?

[RobertN]

Hi, has anyone successfully been able scrape via SMI-S (commonly used by most SAN vendors)?

This would be a great feature since most vendors support SMI-S and not always SNMP pull like on our case with HP 3Par. It does however support SNMP traps but that's pretty much useless for metrics. Most vendors also have their own monitor system just for SAN storage, in HP's case they charge $5000. I would much rather see this in Prometheus because it would be possible to get a complete picture from compute, storage and network in one Dashboard. Today we only get the information from VMware which doesn't give the complete picture.

/ Robert Nilsson

[Brian Brazil]

On 18 September 2017 at 10:16, RobertN <rob...@irob.se> wrote:

Hi, has anyone successfully been able scrape via SMI-S (commonly used by most SAN vendors)?

I'm not aware of any exporters for this, so you may wish to write one.

Brian

 

This would be a great feature since most vendors support SMI-S and not always SNMP pull like on our case with HP 3Par. It does however support SNMP traps but that's pretty much useless for metrics. Most vendors also have their own monitor system just for SAN storage, in HP's case they charge $5000. I would much rather see this in Prometheus because it would be possible to get a complete picture from compute, storage and network in one Dashboard. Today we only get the information from VMware which doesn't give the complete picture.

/ Robert Nilsson

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/244368cb-1d09-480a-87fd-190e85a81862%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[rob...@irob.se]

I would if I could but that is above my expertise as of now at least :)

There is a golang library that is able to connect to a storage device via SMI-S and check if it has 1 or more arrays.

https://github.com/tasleson/smisping

If anyone out there to use this and write an exporter it would be greatly appreciated. Must be a lot of users out there running SAN storage.

 Only the basic metrics would be enough;

Arrays

Disks

Performance metrics like, IOPS, latency, cache hit ratio, SSD hit ratio, iSCSI connections.

Regards

Robert Nilsson 

Den måndag 18 september 2017 kl. 11:20:57 UTC+2 skrev Brian Brazil:

On 18 September 2017 at 10:16, RobertN <rob...@irob.se> wrote:

Hi, has anyone successfully been able scrape via SMI-S (commonly used by most SAN vendors)?

I'm not aware of any exporters for this, so you may wish to write one.

Brian

 

This would be a great feature since most vendors support SMI-S and not always SNMP pull like on our case with HP 3Par. It does however support SNMP traps but that's pretty much useless for metrics. Most vendors also have their own monitor system just for SAN storage, in HP's case they charge $5000. I would much rather see this in Prometheus because it would be possible to get a complete picture from compute, storage and network in one Dashboard. Today we only get the information from VMware which doesn't give the complete picture.

/ Robert Nilsson

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/244368cb-1d09-480a-87fd-190e85a81862%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[dmi...@ymail.com]

Is there any WMI Mapper to expose metrics from WBEM to a host? If I am not mistaking HP had one. 

Best.

[rob...@irob.se]

Hi, I have been working on a solution from time to time on getting metrics out from the HPE 3Par Array. And the chassis supports SNMP but only for basic hardware information. The storage OS is separate and only accept queries via SSH (CLI), SMI-S or via rest API.

I then found the python 3parclient library

http://pythonhosted.org/python-3parclient/

https://github.com/hpe-storage/python-3parclient

But since I suck at scripting I don't know how to proceed.

I guess it would be possible to have the python script start a SimpleHTTPServer and host the metrics is the correct format as per any other exporter. Like the Jenkins exporter approach: https://www.robustperception.io/writing-a-jenkins-exporter-in-python/

Any ideas are welcome! 

The data output I get from running getVolumes() is:

{u'links': [{u'href': u'http://xxx.xxx.xxx.xxx:8008/api/v1/volumes',

             u'rel': u'self'},

            {u'href': u'http://xxx.xxx.xxx.xxx:8008/api/v1/volumespacedistribution',

             u'rel': u'volumeSpaceDistribution'}],

 u'members': [{u'additionalStates': [],

               u'adminSpace': {u'freeMiB': 0,

                               u'rawReservedMiB': 0,

                               u'reservedMiB': 0,

                               u'usedMiB': 0},

               u'baseId': 0,

               u'capacityEfficiency': {},

               u'copyType': 1,

               u'creationTime8601': u'2014-05-07T11:26:27+02:00',

               u'creationTimeSec': 1399454787,

               u'degradedStates': [],

               u'failedStates': [],

               u'id': 0,

               u'links': [{u'href': u'http://xxx.xxx.xxx.xxx:8008/api/v1/volumespacedistribution/admin',

                           u'rel': u'volumeSpaceDistribution'}],

               u'name': u'admin',

               u'policies': {u'caching': True,

                             u'fsvc': False,

                             u'oneHost': False,

                             u'staleSS': True,

                             u'system': True,

                             u'zeroDetect': False},

               u'provisioningType': 1,

               u'readOnly': False,

               u'sizeMiB': 10240,

               u'snapshotSpace': {u'freeMiB': 0,

                                  u'rawReservedMiB': 0,

                                  u'reservedMiB': 0,

                                  u'usedMiB': 0},

               u'ssSpcAllocLimitPct': 0,

               u'ssSpcAllocWarningPct': 0,

               u'state': 1,

               u'userSpace': {u'freeMiB': 0,

                              u'rawReservedMiB': 20480,

                              u'reservedMiB': 10240,

                              u'usedMiB': 10240},

               u'usrSpcAllocLimitPct': 0,

               u'usrSpcAllocWarningPct': 0,

               u'uuid': u'db534970-c3e3-47f1-96e5-dc8750b64359',

               u'wwn': u'60002AC000000000000000000000C6FE'},

              {u'additionalStates': [],

               u'adminSpace': {u'freeMiB': 0,

                               u'rawReservedMiB': 0,

                               u'reservedMiB': 0,

                               u'usedMiB': 0},

               u'baseId': 1,

               u'capacityEfficiency': {},

               u'copyType': 1,

               u'creationTime8601': u'2014-05-07T11:27:17+02:00',

               u'creationTimeSec': 1399454837,

               u'degradedStates': [],

               u'failedStates': [],

               u'id': 1,

[lars.pete...@gmail.com]

Hi all,

I have justed started to looking at pulling data from 3par and exposen to prometheus.

I do not think smi-s is the way to go. My experianse with smi-s its a bit to flaky.

I was thing of using the REST API as it is now days able to pull the perf data or controlsr.
Controlsr export is the fastest way i would think but its adds a dependency on the 3parcli which i do not like.

One problem we have is that we have quit many 3par. Each generate about 11mb of perf data per 5 min if we skip statpd and statld. It would be great to have those as well but that would generate a lot more data. So we about 3-4gb of data per 3par per day without statld and statpd.

In total we talk about 200000-300000 metris for all 3PARs.

I was thinking one server acting as a exporter for each site. In our biggest site have about 10 3pars. In otherwors about 33mb per 5min.

Is that too much for a scrape when we running on a 5 min intervall?

Or should we setup on server for each 3par?

Regars,

Pete

[Ben Kochie]

Prometheus assumes that scrapes are relatively small and fast, and provide the context for a single object.  This means that you want to minimize the scrape and avoid a single large collector.

I'm not very familiar with 3par devices, but I would recommend you have an exporter or proxy exporter that can let you query each one individually.

200-300k metrics for Prometheus is no problem, if the scrape is distributed over a number of devices.

But, by design, Prometheus should be distributed, so one per "site", this avoids cross-site networking issues getting in the way of data collection, and avoids one server being a SPoF.  There are external ways of aggregating data, which is a whole separate topic.

But it sounds like the perf data is generated as a stream of events, if so, those events need to be aggregated for Prometheus to ingest.  We have a terminology thing here that is a little different, because many monitoring systems make this very confusing.

We have samples, these are an individual data point and timestamps.

We have metrics, these represent a single series of samples over time that relate to one object.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6d2c4226-9ae4-4ab7-af43-0dbb52fec756%40googlegroups.com.

[lars.pete...@gmail.com]

>Prometheus assumes that scrapes are relatively small and fast, and provide the context for a single object.  This means that you want to
>minimize the scrape and avoid a single large collector.

Ok good to know. Thx,  
Can have expose multiple device on a single port with different URLs? 
Like https//:<hostname>:<port>/<3par>/metrics

>I'm not very familiar with 3par devices, but I would recommend you have an exporter or proxy exporter that can let you query each one
>individually.

Its needs to be a proxy exporter. The 3PAR via REST, SMI-S, SSH or 3PARCLI. 3PARCLI you install on the server where want to run it from.  
You cant install anything on the 3PAR it self. 

>But it sounds like the perf data is generated as a stream of events, if so, those events need to be aggregated for Prometheus to ingest.  We

>have a terminology thing here that is a little different, because many monitoring systems make this very confusing.

You can collect perf data both in bulk or a stream. Via SSH or 3PARCLI you can stream data. When you stream you can set interval down to 1 sec i think. You need one connection per stat command you run....so downside is that the 3PAR only support 96 connections at the same time. When you run out of connections you can access the 3PAR anymore so long lived connection is not something you want. To kill connections you need a serial cabled and physical access.

The 3PAR has also an internal database for the perf data call system reporter which I was planing to use. Min resolution is 5 min for the SR db so the plan was to pull the data every 5 min. You can dump all counters with the controlsr command which generates CSV files. Takes about 1 to 1 1/2 min, Or you can dump single stats command via REST i.e. you need multiple request to dump all counters. 

So the exporter would fairly simple i would say. Just pull the data every 5 min and convert it to prometheus text format.

[Ben Kochie]

On Wed, Apr 25, 2018 at 10:58 AM, <lars.pete...@gmail.com> wrote:

>Prometheus assumes that scrapes are relatively small and fast, and provide the context for a single object.  This means that you want to
>minimize the scrape and avoid a single large collector.

Ok good to know. Thx,  
Can have expose multiple device on a single port with different URLs? 
Like https//:<hostname>:<port>/<3par>/metrics

Typically, we like to install exporters 1:1 with the software they run.  For example, mysqld_exporter would live on localhost with a MySQL server.

But, this is difficult for devices that don't let you run binaries directly on them.  For these, we use a proxy method like we do for SNMP and blackbox prober.

You would run the proxy on localhost with the Prometheus server, and it would poll something like this:

http://localhost:port/3par?target=3par-host-a.example.com

There's a pattern for how this works in the blackbox_exporter:

https://github.com/prometheus/blackbox_exporter#prometheus-configuration

 

>I'm not very familiar with 3par devices, but I would recommend you have an exporter or proxy exporter that can let you query each one
>individually.

Its needs to be a proxy exporter. The 3PAR via REST, SMI-S, SSH or 3PARCLI. 3PARCLI you install on the server where want to run it from.  
You cant install anything on the 3PAR it self. 

I would probably do this via REST, building custom json->prometheus translators is reasonably easy.

There's already a "json exporter" that could be used as a starting point:

https://github.com/kawamuray/prometheus-json-exporter

It might be interesting to write something like this that supports a target parameter instead of only having one URL on the command line.

 

>But it sounds like the perf data is generated as a stream of events, if so, those events need to be aggregated for Prometheus to ingest.  We

>have a terminology thing here that is a little different, because many monitoring systems make this very confusing.
You can collect perf data both in bulk or a stream. Via SSH or 3PARCLI you can stream data. When you stream you can set interval down to 1 sec i think. You need one connection per stat command you run....so downside is that the 3PAR only support 96 connections at the same time. When you run out of connections you can access the 3PAR anymore so long lived connection is not something you want. To kill connections you need a serial cabled and physical access.

The 3PAR has also an internal database for the perf data call system reporter which I was planing to use. Min resolution is 5 min for the SR db so the plan was to pull the data every 5 min. You can dump all counters with the controlsr command which generates CSV files. Takes about 1 to 1 1/2 min, Or you can dump single stats command via REST i.e. you need multiple request to dump all counters. 

So the exporter would fairly simple i would say. Just pull the data every 5 min and convert it to prometheus text format.

The REST exporter sounds reasonably simple, but 5 minutes is generally too slow a polling interval for Prometheus targets.  We typically ingest much faster, the slowest we recommend is usually every 2 minutes.  For example, a typical prometheus client library implementation[0] can dump the status of over 4000 counters in under 50 milliseconds, and that includes network transit time.  Even the node_exporter, that's basically a proxy for parsing random files in /proc, can return in 30ms.

I guess it would take more detailed knowledge of what the 3par devices are exposing to figure out the best way to handle it.

[0]: https://gitlab.com/gitlab-org/gitaly

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/dee1d701-d596-433f-ab1d-15c7d13efb55%40googlegroups.com.