API: Network (TX,RX,CX values)

[MKA NET]

I'm grabbing TX,RX,CX values from the below endpoint.  However, I'm unsure if these values are in bits, kilobits, or megabits per second.

http://192.168.1.2:61208/api/3/network

If I presume these values are ALWAYS in bits per second, they seem to be wrong.

On the Web Interface, RX,TX are correct; however, they dynamically switch between Kbps to Mbps

I need to be certain that the API isn't dynamically/spontaneously switching TX, RX, CX values from Kbps to Mbps the way the Web interface does.  If they are, I need a way to ALWAYS return these values in bits per second or Kbits per second.  I would appreciate if someone could shed some light on this.

Thanks in advance!

[Ken Persing]

I'm just piling on here:

I just came upon the exact same problem using the restful API to pull network activity into homeassistant. The curses interface could be showing an RX of 1.2Mb/s and the API will give me a number like 480908. no amount of converting bits to bytes or dividing by 1024 can get the numbers anywhere close to matching up.

If anyone has some insights, that would be great!

Thanks,

Ken

[MKA NET]

Hi Ken, can you try my Home Assistant rest sensor and respective templates below?  It seems to be pretty close.  The only thing I'm not sure is if we have to divide by 1024 or 1000:

  - platform: rest

   name: network

   resource: http://192.168.1.2:61208/api/3/network

   value_template: N/A

   json_attributes:

     - interface_name

     - time_since_update

     - cumulative_rx

     - rx

     - cumulative_tx

     - tx

     - cumulative_cx

     - cx

     - is_up

     - speed

   verify_ssl: false

   timeout: 20

   scan_interval: 8

  - platform: template

   sensors:

     horizon_download_speed:

       value_template: '{{ ((state_attr("sensor.network", "rx") /2 | int) / 1000000) | round (2) }}'

       unit_of_measurement: "Mbps"

 - platform: template

   sensors:

     horizon_upload_speed:

       value_template: '{{ ((state_attr("sensor.network", "tx") /2 | int) / 1000000) | round (2) }}'

       unit_of_measurement: "Mbps"

 - platform: template

   sensors:

     horizon_total_speed:

       value_template: '{{ ((state_attr("sensor.network", "cx") /2 | int) / 1000000) | round (2) }}'

       unit_of_measurement: "Mbps"

[Ken Persing]

Thanks for the yaml! I was using /api/3/network/interface_name/tun0 and was having a bit of trouble with the attribute path ($.tun0[0] was not working).

Anyway, my numbers are still a pretty far off... my upload and download speeds don't vary too much and they are nowhere near matching.

why are you dividing by 2?

Thanks,

Ken

[MKA NET]

Actually, your results are exactly like mine.  I noticed that the value provided by the API was roughly twice as much as the real value last time I tested this... so, out of sheer desperation, I tried dividing the value by 2.  

There must be something we're missing. It's hard to believe that basic network bandwidth info in the Glances API is completely unusable.  I'm sure someone would have noticed this by now if it was really broken. I wasn't able to find any Github issues related to this.  I think it's time we open a Github issue for this.

BTW:  What environment are you running Glances in?  I have Glances installed on Python 3.7 (Windows 10) starting it with the below command-line:

C:\Users\MKANET\AppData\Local\Programs\Python\Python37\python.exe glances-script.py -w -0 --disable-autodiscover --disable-plugin docker --config C:\Users\MKANET\AppData\Local\Programs\Python\Python37\scripts\glances.conf -q -t 8 --cached-time 8

[MKA NET]

EDIT:  Sorry for the double-notifications, I accidentally deleted this post... adding it back.

It looks like your results are exactly like mine.  I noticed that the value provided by the API was roughly twice as much as the real value last time I tested this... so, out of sheer desperation, I tried dividing the value by 2.  

 

There must be something we're missing. It's hard to believe that basic network bandwidth info in the Glances API is completely unusable.  I'm sure someone would have noticed this by now if it was really broken. I wasn't able to find any Github issues related to this  I just opened a new issue for this on Github:

https://github.com/nicolargo/glances/issues/1704

[Ken Persing]

I ended up multiplying by 3.2 to get the numbers to match fairly closely...

I am running glances via systemd on some ubuntu20.04 (python 3.8) boxes and via SYSV init on openbsd.

no conf file, just added -w parameter.

Glances v3.1.3 with psutil v5.5.1.

There seems to be very little information specific to these data; I wonder what "time_since_update" is for and whether it needs to be included in the equation to normalise the other data points?

[Ken Persing]

hmmmmm.   https://github.com/nicolargo/glances/issues/214

[MKA NET]

Thanks for the tip to multiply by 3.2.  Maybe you wouldn't mind asking about "time_since_update" in the Github Issue I opened.  The nicolargo may offer a solution that could help us out.

[Ken Persing]

Glorioski Zero!!!

This is from the glances code: .../glances/outputs/static/js/components/plugin-network/view.html

<div class="table-row" ng-repeat="network in vm.networks track by network.interfaceName">
    <div class="table-cell text-left">{{ network.interfaceName | min_size }}</div>
    <div class="table-cell" ng-show="!vm.arguments.network_cumul && !vm.arguments.network_sum">{{ vm.arguments.byte ?
        (network.rx / network.time_since_update | bytes) : (network.rx / network.time_since_update | bits) }}
    </div>
    <div class="table-cell" ng-show="!vm.arguments.network_cumul && !vm.arguments.network_sum">{{ vm.arguments.byte ?
        (network.tx / network.time_since_update | bytes) : (network.tx / network.time_since_update | bits) }}
    </div>

    <div class="table-cell" ng-show="!vm.arguments.network_cumul && vm.arguments.network_sum"></div>
    <div class="table-cell" ng-show="!vm.arguments.network_cumul && vm.arguments.network_sum">{{ vm.arguments.byte ?
        (network.cx / network.time_since_update | bytes) : (network.cx / network.time_since_update | bits) }}
    </div>

[MKA NET]

Awesome.  Do you mind sharing your templates which use time_since_update?  Below, is what I have so far:

  - platform: template

   sensors:

     horizon_network_time_since:

       value_template: '{{ state_attr("sensor.network", "time_since_update") }}'

       unit_of_measurement: "Mbps"

  - platform: template

   sensors:

     horizon_download_speed:

       value_template: '{{ ((state_attr("sensor.network", "tx") | int) / 1000000) / state_attr("sensor.network", "time_since_update") | round (2) }}'

       unit_of_measurement: "Mbps"

[Ken Persing]

Mine was more like this:

       

value_template: '{{ ((state_attr("sensor.network", "tx") / state_attr("sensor.network", "time_since_update") | int) / 1000000) | round (2) }}'

you don't want to convert to an integer before you divide by time_since_update. :)

[MKA NET]

Thanks.  I tried using the example you have above.  I'm not able to get correct results, not even close.  Instead of getting190Mbps Rx, I'm getting 38.85Mbps (based on the templates below).  Does the code below look right to you?  How close are your results?  Maybe I'm missing something.

  - platform: rest

    name: network

    resource: http://192.168.1.2:61208/api/3/network

    value_template: N/A

    json_attributes:

      - interface_name

      - time_since_update

      - cumulative_rx

      - rx

      - cumulative_tx

      - tx

      - cumulative_cx

      - cx

      - is_up

      - speed

    verify_ssl: false

    timeout: 20

    scan_interval: 8

  - platform: template

    sensors:

      horizon_download_speed:

        value_template: '{{ ((state_attr("sensor.network", "rx") / state_attr("sensor.network", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: "Mbps"

  - platform: template

    sensors:

      horizon_upload_speed:

        value_template: '{{ ((state_attr("sensor.network", "tx") / state_attr("sensor.network", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: "Mbps"

  - platform: template

    sensors:

      horizon_total_speed:

        value_template: '{{ ((state_attr("sensor.network", "cx") / state_attr("sensor.network", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: "Mbps"

[Ken Persing]

OK...  I looked at the glances network_plugin code and the psutil docs and the output from the glances api...

Everything is in bytes.

glances takes a reading of the network interfaces from psutil, records the time and the cumulative bytes sent and received, then does it again... calculates the difference in time, and subtracts the old cumulative number from the new to come up with rx and tx.

you can see the cumulative numbers from the api match up exactly with the numbers from ifconfig, which is provided in bytes... is it converted to bits for the curses and web interfaces, but left as is for the rest api.

Try the following to get Mbps:

value_template: '{{ ((state_attr("sensor.network", "tx") / state_attr("sensor.network", "time_since_update") | int) * 8 / 1000000) | round (2) }}'

let me know if this works!

[MKA NET]

You nailed it.  Works great.  We just needed to convert megabits/sec to megabytes/sec.  Hopefully, this will help others trying to do the same thing in the future.

[MKA NET]

Hey Ken, not sure if you're still following this thread.  I'm curious, do you know if Glances API "diskio" requires anything special for calculating write_bytes and read_bytes?  Currently below is what I'm using.. converting it to MB/sec.  However, my results are completely wrong.

  - platform: rest

   name: diskio_0

   resource: http://192.168.1.2:61208/api/3/diskio

   value_template: '{{ ( value_json[0] ) }}'

   json_attributes:

     - time_since_update

     - disk_name

     - read_count

     - write_count

     - read_bytes

     - write_bytes

  - platform: template

   sensors:

     horizon_c_drive_read_bytes:

       value_template: '{{ ((state_attr("sensor.diskio_0", "read_bytes") | int) / 1000000) | round (2) }}'

       unit_of_measurement: 'MB/s'

 - platform: template

   sensors:

     horizon_c_drive_write_bytes:

       value_template: '{{ ((state_attr("sensor.diskio_0", "write_bytes") | int) / 1000000) | round (2) }}'

       unit_of_measurement: 'MB/s'

 - platform: template

   sensors:

     horizon_e_drive_read_bytes:

       value_template: '{{ ((state_attr("sensor.diskio_1", "read_bytes") | int) / 1000000) | round (2) }}'

       unit_of_measurement: 'MB/s'

 - platform: template

   sensors:

     horizon_e_drive_write_bytes:

       value_template: '{{ ((state_attr("sensor.diskio_1", "write_bytes") | int) / 1000000) | round (2) }}'

       unit_of_measurement: 'MB/s'

[MKA NET]

I tried looking through the code for the diskio plugin, I didn't see anything that I could figure out.  I even tried dividing by time_since_update.  My results are all over the place.

https://github.com/nicolargo/glances/blob/develop/glances/outputs/static/js/components/plugin-diskio/view.html

https://github.com/nicolargo/glances/blob/develop/glances/plugins/glances_diskio.py

[kenneth...@gmail.com]

Sorry, I didn't see this message. I haven't played around with disk i/o, but looking at the documentation, I see the following:

Those files you reference look like the right ones... check out the very bottom of glances_diskio.py:

# Bitrate
txps = self.auto_unit(
int(i['read_bytes'] // i['time_since_update']))
rxps = self.auto_unit(
int(i['write_bytes'] // i['time_since_update']))
msg = '{:>7}'.format(txps)
ret.append(self.curse_add_line(msg,
self.get_views(item=i[self.get_key()],
key='read_bytes',
option='decoration')))
msg = '{:>7}'.format(rxps)
ret.append(self.curse_add_line(msg,
self.get_views(item=i[self.get_key()],
key='write_bytes',
option='decoration')))
return ret

so the author is taking read_bytes and dividing by time _since_update.  I am pulling diskio stats from my hypervisor, so I haven't tried implementing any of this. I'm a couple weeks late on reading this thread, so let me know if you've figured it out. If not,  I'll try to add sth to hassio that will track glances diskio and see how it works.

[MKA NET]

Hi Ken, thanks for replying.  Yes, I realized that the read_bytes and write_bytes are divided by time_since_update in the same fashion as network TX, RX, CX based on the code.  I also confrmed this by asking the Glances developer.  He said that any API component that involves time_since_update should divide by time_since_update.

I tried posting my Home Assistant templates; however, I'm not able to keep the original originial formatting.  I kept read/write bytes in MB/sec as opposed to Mb/sec since disk throughputs are typically measured in MB/sec.  I'm guessing the calculated values are correct.  I haven't looked at the calculated values that closely yet to know to be certain.  I'm just taking the developer's word on it.

  - platform: template

    sensors:

      horizon_c_drive_read_bytes:

        value_template: '{{ ((state_attr("sensor.diskio_0", "read_bytes") / state_attr("sensor.diskio_0", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: 'MB/s'

  - platform: template

    sensors:

      horizon_c_drive_write_bytes:

        value_template: '{{ ((state_attr("sensor.diskio_0", "write_bytes") / state_attr("sensor.diskio_0", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: 'MB/s'

  - platform: template

    sensors:

      horizon_e_drive_read_bytes:

        value_template: '{{ ((state_attr("sensor.diskio_1", "read_bytes") / state_attr("sensor.diskio_1", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: 'MB/s'

  - platform: template

    sensors:

      horizon_e_drive_write_bytes:

        value_template: '{{ ((state_attr("sensor.diskio_1", "write_bytes") / state_attr("sensor.diskio_1", "time_since_update") | int) / 1000000) | round (2) }}'

        unit_of_measurement: 'MB/s'

[MKA NET]

Sorry, I messed up my previous response while editing.. hopefully you can figure out what I was trying to say.

[aade...@gmail.com]

Hi Ken and Michael,

I just came across this thread and I ran into similar problem and would love some help with it.  Could you please shed light on the unit of "time_since_update". Also how do I get single values from the API result of a network interface. Say I want to retrieve the value of "rx", how do I do that using the api call? Currently I run it as follows: curl http://localhost:61208/api/3/network/interface_name/ens2 and I get the following result:
{"ens2": [{"time_since_update": 788.5855550765991, "cx": 4410274, "is_up": true, "key": "interface_name", "speed": 68718428160, "tx": 3076676, "cumulative_rx": 1940213370, "rx": 1333598, "cumulative_cx": 6341115829, "cumulative_tx": 4400902459, "interface_name": "ens2"}]}

If I try running it like this: curl http://localhost:61208/api/3/network/interface_name/ens2/0 or [0], it returns an error to me.