[Node Exporter] Monitoring Infiniband network traffic

[Matteo Dessalvi]

Hi all.

With the latest release of node_exporter (0.15.0) I am trying to monitor

a bunch of Infiniband network cards on our servers (we are using Mellanox 

ConnectX-3 cards at 56 Gb/sec (4X FDR)).

Since this version has integrated the pull request #579, I thought I should

be able to graph the following IB metrics:

- node_infiniband_port_data_transmitted_bytes

- node_infiniband_port_data_received_bytes

The exporter reports some value from both of them but the problem arise when 

I am trying to apply the irate/rate functions on it: the result is always zero.

For example: irate(node_infiniband_port_data_transmitted_bytes{job='my_job'}[5m])

the result on the PromQL dashboard is a flat line, independently of the time resolution 

(30 min, 1 hour, etc.).

I guess I am misunderstanding something here or most likely I am simply doing 

something wrong.

Anybody has tried to monitor IB cards with node_exporter and can give me some

hints?

Thanks in advance!

Best regards,

Matteo

[Ben Kochie]

That should be working as expected.  However I think there are some variations for different drivers.

Do you see anything for node_infiniband_legacy_data_received_bytes_total

If you query in the console, not the graph, for node_infiniband_port_data_transmitted_bytes[5m] do you see sensible data?  Does it match the data in /sys/class/infiniband?

Just to ask a stupid question, what is your node_exporter scrape interval?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9740234f-f147-4f84-8ad1-9b6bf0b678b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matteo Dessalvi]

On Mon, Nov 6, 2017 at 23:34:23 UTC+1, Ben Kochie wrote:

That should be working as expected.  However I think there are some variations for different drivers.

Do you see anything for node_infiniband_legacy_data_received_bytes_total

No, I am getting "No data". 

 

If you query in the console, not the graph, for node_infiniband_port_data_transmitted_bytes[5m] do you see sensible data?  Does it match the data in /sys/class/infiniband?

I can see numbers whenever I am querying the metrics (node_infiniband_[...]) but when I am applying the rate function the result

is always zero. 

According to https://community.mellanox.com/docs/DOC-2751, our counters should be defined under:

-  /sys/class/infiniband/mlx4_0/ports/1/counters/port_xmit_data

-  /sys/class/infiniband/mlx4_0/ports/1/counters/port_recv_data

but this is somehow weird....I am querying multiple machines and I always get the same number (no changes).......

It cannot be, something is wrong.

We are doing the IB monitoring through Ganglia using plugins developed by my colleagues (https://github.com/vpenso/ganglia-sensors)

but these python scripts are using ibstat and perfquery to get the data. I guess I am looking in the wrong place.

Just to ask a stupid question, what is your node_exporter scrape interval?

I am not defining any specific interval for the exporters, just using the 'scrape_interval' at global

level, which is set to 15 seconds.

Regards,

Matteo

On Mon, Nov 6, 2017 at 10:13 PM, Matteo Dessalvi <mdes...@gmail.com> wrote:

Hi all.

With the latest release of node_exporter (0.15.0) I am trying to monitor

a bunch of Infiniband network cards on our servers (we are using Mellanox 

ConnectX-3 cards at 56 Gb/sec (4X FDR)).

Since this version has integrated the pull request #579, I thought I should

be able to graph the following IB metrics:

- node_infiniband_port_data_transmitted_bytes

- node_infiniband_port_data_received_bytes

The exporter reports some value from both of them but the problem arise when 

I am trying to apply the irate/rate functions on it: the result is always zero.

For example: irate(node_infiniband_port_data_transmitted_bytes{job='my_job'}[5m])

the result on the PromQL dashboard is a flat line, independently of the time resolution 

(30 min, 1 hour, etc.).

I guess I am misunderstanding something here or most likely I am simply doing 

something wrong.

Anybody has tried to monitor IB cards with node_exporter and can give me some

hints?

Thanks in advance!

Best regards,

Matteo

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

I can't say for sure what's going on without looking at the raw data.

You can see that the code[0] for reading the data from proc is really simple, it walks the sysfs tree and extracts the numbers from all the files it finds.  There is no caching or any fancy magic going on here.

If you manually curl the node_exporter's metrics on a node, do you see the values changing?

It could be that the other tools are messing with the driver or stats, I can't say for sure as I'm not familiar with the IB driver stack.

[0]: https://github.com/prometheus/node_exporter/blob/master/collector/infiniband_linux.go

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b0f0c2df-af43-4cbe-b081-42a3791437e7%40googlegroups.com.

[Matteo Dessalvi]

Hi Ben.

I can confirm that there are no "node_infiniband_legacy_[...]" metrics on our exporters.

The following metrics are the only IB related ones which are exported:

node_infiniband_link_downed_total{device="mlx4_0",port="1"} 0

node_infiniband_link_error_recovery_total{device="mlx4_0",port="1"} 0

node_infiniband_port_data_received_bytes{device="mlx4_0",port="1"} 4.294967295e+09

node_infiniband_port_data_transmitted_bytes{device="mlx4_0",port="1"} 4.294967295e+09

If I try to 'scrape' those metrics directly I am always getting the same number (which varies 

from machine to machine) but I guess that explain why the rate is constantly zero (it does 

not make much sense to calculate the rate of a constant value....).

I got somehow confused by the fact that it's possible to calculate the rates 

on the following metrics:

- rate(node_network_transmit_bytes{job='my_srv1',device='ib0'}[5m]))

- rate(node_network_receive_bytes{job='my_srv1',device='ib0'}[5m]))

The problem is that those numbers over there does not match the ones

reported by our Ganglia plugins: those metrics from Prometheus

reports correctly the same trend we can see on Ganglia but they

are way lower.....roundabout 8 times lower than the numbers in

Ganglia.

It can be the fact we are using the perfquery command, which

somehow influence the data reported by the node_exporter

(perfquery can reset the counters on the card after reading it).

Regards,

Matteo

[Martin Schröder]

2017-11-07 11:47 GMT+01:00 Matteo Dessalvi <mdes...@gmail.com>:
> node_infiniband_port_data_received_bytes{device="mlx4_0",port="1"}
> 4.294967295e+09
> node_infiniband_port_data_transmitted_bytes{device="mlx4_0",port="1"}
> 4.294967295e+09

Looks like max long unsigned (2^32), so not meaningful.

Best
Martin

[Ben Kochie]

As noted in the code[0], Melanox reports the data in a funny way, exactly 4x smaller for both in and out.

[0]: https://github.com/prometheus/node_exporter/blob/master/collector/infiniband_linux.go#L151-L158

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2fa2b74e-9719-4da5-ab0d-0bd6fc12d937%40googlegroups.com.

[Matteo Dessalvi]

As I wrote before, it could be I have stumbled upon a problem on our side (I don't think

the node_exporter is involved).

So far I can confirm that:

- all the counter under /sys/class/infiniband/mlx4_0/ports/1/counters/* are basically static.

Interestingly enough, the 'port_{xmit,rcv}_data' counters reports always the same number: 4294967295

- the Ganglia plugin we are using (https://github.com/vpenso/ganglia-sensors/blob/master/lib/python_modules/infiniband.py)

is based on the 'perfquery' command, we do not iterate over /sys.

The only metrics which reports meaningful data are the following two:

- rate(node_network_transmit_bytes{job='my_srv1',device='ib0'}[5m]))

- rate(node_network_receive_bytes{job='my_srv1',device='ib0'}[5m]))

but as I wrote, the numbers did not match the ones reported in Ganglia.

Best regards,

Matteo

[Matteo Dessalvi]

After a couple of tests I believe it should work now but there are still a couple

of things which are unclear to me.

The node_infiniband_port_data_[...] continuously reports the same number: even

when I disable the gmond running our IB plugin the numbers remains unchanged.

Which means (I guess) that running perfquery did not affect the counters under

/sys/class/infiniband/mlx4_0/[...].

As stated in my previous email, the only metrics which reports meaningful data are 

the following two (note that I have switched from 'rate' to 'irate'):

- irate(node_network_transmit_bytes{job='my_srv1',device='ib0'}[5m]))

- irate(node_network_receive_bytes{job='my_srv1',device='ib0'}[5m]))

I have ran some tests with iperf and I can see that the number reported

by node_exporter are consistent with the ones reported on our Ganglia

interface.

Best regards,

Matteo

[Ben Kochie]

Yea, I don't know what is going on, it sounds like something is broken in the infiniband drivers. :(

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/67db3d13-6de2-40f5-a24b-5205a3ae6e6b%40googlegroups.com.

[Matteo Dessalvi]

On Thursday Nov 9 2017 13:13:32 UTC+1, Ben Kochie wrote:

Yea, I don't know what is going on, it sounds like something is broken in the infiniband drivers. :(

Well, not necessarily. It could be that since our IB cards (and thus our drivers) 

are roundabout three years old, the method used by node_exporter to extract 

the data is not working as expected.

Maybe with newer IB cards the node_infiniband_[...] metrics are instead exported

as expected.

In the mean time I am happy that I am getting the right numbers :-)

If anybody is using different (or newer) IB hardware it would be nice to know 

how the exporter is performing with it.

Regards,

Matteo