node_network_receive_bytes metric

[hdtv...@gmail.com]

Hello!

node_network_receive_bytes, node_network_transmit_bytes and other metrics from netdev collector have type Gauge, though they can't go down. Because rate/irate functions can't be used with gauges i have problem with alerts like:
IF rate(node_network_receive_bytes{device=~"agg[e,i]"}[1m])*8/1000000 > 500

Can you advise something?

Thanks for your help!

[Brian Brazil]

Not all types from the node exporter are correct. Those are counters and you can use rate() with them. 

--

Brian Brazil

www.robustperception.io

[hdtv...@gmail.com]

четверг, 5 января 2017 г., 22:30:49 UTC+6 пользователь Brian Brazil написал:

Yesterday i got alert with anbsolutely insane 589417.0857400887Mbit/s. And i thought it was problem with using rate on gauge type metric.

[Brian Brazil]

Do you have the raw data for that timeseries?

Usually that sort of thing is caused by your setup being incorrect, but there's always a chance it's a bug.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d3419b59-7820-41a8-8420-f45b20e821a2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[hdtv...@gmail.com]

четверг, 5 января 2017 г., 22:42:12 UTC+6 пользователь Brian Brazil написал:

> On 5 January 2017 at 16:35, <hdtv...@gmail.com> wrote:
> четверг, 5 января 2017 г., 22:30:49 UTC+6 пользователь Brian Brazil написал:
>
> > On 5 January 2017 at 16:28,  <hdtv...@gmail.com> wrote:
>
> > Hello!
>
> >
>
> >
>
> >
>
> > node_network_receive_bytes, node_network_transmit_bytes and other metrics from netdev collector have type Gauge, though they can't go down. Because rate/irate functions can't be used with gauges i have problem with alerts like:
>
> >
>
> > IF rate(node_network_receive_bytes{device=~"agg[e,i]"}[1m])*8/1000000 > 500
>
> >
>
> >
>
> >
>
> > Can you advise something?
>
> >
>
> >
>
> >
>
> > Not all types from the node exporter are correct. Those are counters and you can use rate() with them. 
>
> >
>
> > --
>
> >
>
> >
>
> > Brian Brazil
>
> > www.robustperception.io
>
>
>
> Yesterday i got alert with anbsolutely insane 589417.0857400887Mbit/s. And i thought it was problem with using rate on gauge type metric.
>
>
>
>
>
> Do you have the raw data for that timeseries?
>
>
> Usually that sort of thing is caused by your setup being incorrect, but there's always a chance it's a bug.
>
>
> Brian
>  
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/d3419b59-7820-41a8-8420-f45b20e821a2%40googlegroups.com.
>
>
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
>
> Brian Brazil
> www.robustperception.io

In the timeseries i have:
3315469095178 @1483572014.089
3315468730978 @1483572029.089

So rate() processed it as counter reset, though it wasn't.

[Ben Kochie]

In the timeseries i have:
3315469095178 @1483572014.089
3315468730978 @1483572029.089

So rate() processed it as counter reset, though it wasn't.

That's a very strange result, maybe a kernel or driver bug?  Does this happen often?  Can you give us a bit more version detail? (exporter, kernel, network card)

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/64786fb5-2012-4929-985c-55d03b74c342%40googlegroups.com.

[hdtv...@gmail.com]

node exporter version is 0.13.0-rc.2 (branch: master, revision: 42d1658d5632fe0bfb281761d5e3620cfc37990b)
kernel: Linux 4.2.0-35-generic #40~14.04.1-Ubuntu SMP Fri Mar 18 16:37:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Network cards are:
*-network:0
description: Ethernet interface
product: Ethernet Controller 10-Gigabit X540-AT2
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:04:00.0
logical name: int1
version: 01
serial: aa:aa:aa:aa:aa:aa
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=4.0.1-k duplex=full firmware=0x8000059e latency=0 link=yes multicast=yes port=twisted pair slave=yes
resources: irq:50 memory:91c00000-91dfffff memory:91e04000-91e07fff memory:92a00000-92a7ffff memory:92b00000-92bfffff memory:92c00000-92cfffff

*-network:0
description: Ethernet interface
product: Ethernet Controller 10-Gigabit X540-AT2
vendor: Intel Corporation
physical id: 0
bus info: pci@0000:05:00.0
logical name: int2
version: 01
serial: aa:aa:aa:aa:aa:aa
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical tp 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=ixgbe driverversion=4.0.1-k firmware=0x8000059e latency=0 link=no multicast=yes port=twisted pair slave=yes
resources: irq:89 memory:92200000-923fffff memory:92404000-92407fff memory:92f00000-92f7ffff memory:93000000-930fffff memory:93100000-931fffff

It was second time i got this result.

[hdtv...@gmail.com]

Checked this metric for network interfaces included in aggregated interface and there wasn't any resets. For now, i will use separate interfaces in alert conditional.