possible to get the traffic rate that is going out on a interface (ex: eno1)

[Deeraj V]

Hello Team,

Is it possible to get the traffic rate that is going out on a interface (ex: eno1) towards a particular destination like network or ip?

Does Prometheus support any metrics ?

If yes, can anybody help me out as I am stuck with requirements?

Thanks

DV 

[Julius Volz]

Hi Deeraj,

While the Node Exporter exposes transmit / receive metrics for each network interface, this information would not tell you where the traffic is going. Putting destinations like public IPs into Prometheus label values would quickly create unboundedly large cardinality (huge numbers of time series) that would blow up your Prometheus server. Putting network IDs in there (like AS numbers) probably would still be problematic, but somewhat less so, depending on your use case. However, you'd anyways need to build some kind of exporter that tells you that kind of level of detail in your metrics first. Some exporters like https://github.com/neptune-networks/flow-exporter actually capture Netflow data and report source and destination AS identities, maybe that can be interesting to take a look at.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9222c6ae-7768-4c6e-86e0-8ec522647fc6n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Brian Candler]

There are a variety of tools for this:

- tools which capture packets on the interface and classify them into their own format: e.g. packetbeat, SiLK + flowviewer, pmacct, NTOPng [some features of the latter are commercial only]

- tools which capture packets on the interface and turn them into standard Netflow/IPFIX records: e.g. softflowd, goflow

    => you then also need something which stores and queries these flow records, e.g. elastiflow, nfdump/nfsen

- IDS tools which perform deeper packet analysis, e.g. zeek (formerly bro)

- or you can just capture raw packets on the wire using tcpdump/wireshark and analyse them later, e.g. with packetq

Most of these don't work directly with prometheus, because of the high cardinality issue that Julius highlighted.  You will need a proper log storage backend like elasticsearch or loki.

However, I have managed to get one of them to work in a prometheus-exporter type of way, and that's pmacct.  This could be useful if you can pre-define the ranges of addresses that you want to expose, so that you have limited cardinality.

To make this work I used exporter_exporter to run a small bit of python code (included below) to query pmacct's own in-RAM counter store.  pmacct is configured to aggregate non-local source and destination addresses. With this configuration, the results you get look like this:

# TYPE flow_bytes untyped
flow_bytes{net_src="::",ip_dst="2001:db8:1234:8fa::2",aggregate="inbound"} 7695
flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.33",aggregate="inbound"} 566
flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.1",aggregate="inbound"} 581964
flow_bytes{net_src="::",ip_dst="2001:db8:1234:800::1",aggregate="inbound"} 10883
flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::32",aggregate="inbound"} 1300
flow_bytes{net_src="::",ip_dst="2001:db8:1234:8f9::4",aggregate="inbound"} 2138
flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.234",aggregate="inbound"} 530
flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::58",aggregate="inbound"} 912
flow_bytes{net_src="0.0.0.0",ip_dst="10.12.255.47",aggregate="inbound"} 84
flow_bytes{net_src="0.0.0.0",ip_dst="192.0.2.233",aggregate="inbound"} 12084
flow_bytes{net_src="::",ip_dst="2001:db8:1234:8ff::33",aggregate="inbound"} 107517
flow_bytes{net_src="::",ip_dst="2001:db8:1234:800:e809:5c7e:b33c:4f48",aggregate="inbound"} 1796
flow_bytes{ip_src="10.12.255.31",net_dst="0.0.0.0",aggregate="outbound"} 1981
flow_bytes{ip_src="2001:db8:1234:8ff::58",net_dst="::",aggregate="outbound"} 904
flow_bytes{ip_src="10.12.255.56",net_dst="0.0.0.0",aggregate="outbound"} 52
flow_bytes{ip_src="2001:db8:1234:8fa::2",net_dst="::",aggregate="outbound"} 43842
flow_bytes{ip_src="2001:db8:1234:800::1",net_dst="::",aggregate="outbound"} 63675
flow_bytes{ip_src="10.12.255.254",net_dst="0.0.0.0",aggregate="outbound"} 209
flow_bytes{ip_src="2001:db8:1234:8ff::33",net_dst="::",aggregate="outbound"} 18578
flow_bytes{ip_src="2001:db8:1234:800:e809:5c7e:b33c:4f48",net_dst="::",aggregate="outbound"} 1709
flow_bytes{ip_src="10.12.0.100",net_dst="0.0.0.0",aggregate="outbound"} 4250
flow_bytes{ip_src="10.12.255.33",net_dst="0.0.0.0",aggregate="outbound"} 2887
flow_bytes{ip_src="192.0.2.234",net_dst="0.0.0.0",aggregate="outbound"} 566
flow_bytes{ip_src="10.12.255.1",net_dst="0.0.0.0",aggregate="outbound"} 84
flow_bytes{ip_src="2001:db8:1234:8f9::4",net_dst="::",aggregate="outbound"} 2140
flow_bytes{ip_src="2001:db8:1234:8ff::32",net_dst="::",aggregate="outbound"} 1298
flow_bytes{ip_src="10.12.255.47",net_dst="0.0.0.0",aggregate="outbound"} 581964

That is: for "inbound" traffic, the individual destination addresses are shown, and the source is always 0.0.0.0 or :: (i.e. "the Intenet"), aggregated together.  For "outbound" traffic, the destination is always 0.0.0.0 or ::, and individual source IP addresses are shown. It lets you answer questions like "which devices on my local network are sending (or receiving) the most traffic?", and the cardinality is limited by the number of active local IPs you have.  But it won't answer questions like "where on the Internet is my outbound traffic going?"; that's too high cardinality for Prometheus.

The configs to achieve this are below.  Note: I was using nfacctd to receive Netflow records generated by my router.  If you want to sniff packets going through a Linux server interface, then you'll need to run pmacctd instead of nfacctd.  More info here. 

# On pmacct server

## /etc/pmacct/nfacctd.conf

nfacctd_port: 2055
plugins: memory[inbound], memory[outbound]

imt_path[inbound]: /tmp/inbound.pipe
aggregate_filter[inbound]: dst net 10.0.0.0/8 or dst net 192.0.2.232/30 or dst net 2001:db8:1234::/56
aggregate[inbound]: dst_host
imt_mem_pools_number[inbound]: 64
imt_mem_pools_size[inbound]: 65536

imt_path[outbound]: /tmp/outbound.pipe
aggregate_filter[outbound]: src net 10.0.0.0/8 or src net 192.0.2.232/30 or src net 2001:db8:1234::/56
aggregate[outbound]: src_host
imt_mem_pools_number[outbound]: 64
imt_mem_pools_size[outbound]: 65536

## /etc/systemd/system/nfacct.service

[Unit]
Description=nfacctd
Documentation=https://github.com/pmacct/pmacct/wiki
After=network-online.target

[Service]
User=nobody
Group=nogroup
ExecStart=/usr/sbin/nfacctd -f /etc/pmacct/nfacctd.conf
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

## /etc/prometheus/expexp.yaml

modules:
  pmacct:
    method: exec
    timeout: 5s
    exec:
      command: /usr/local/bin/pmacct.py

## /etc/systemd/system/exporter_exporter.service

[Unit]
Description=Prometheus exporter proxy
Documentation=https://github.com/QubitProducts/exporter_exporter
After=network-online.target

[Service]
User=nobody
Group=nogroup
ExecStart=/usr/local/bin/exporter_exporter -config.file /etc/prometheus/expexp.yaml
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

## /usr/local/bin/pmacct.py

#!/usr/bin/python3

import json
import subprocess

LABELS={}   # add any static labels here, eg hostname

def export(metric, labels, value):
    lstr = ",".join(("%s=\"%s\"" % (k,v) for k,v in labels.items()))
    print("%s{%s} %d" % (metric, lstr, value))

for aggregate in ["inbound", "outbound"]:
    res = subprocess.run(["pmacct", "-s", "-p", "/tmp/%s.pipe" % aggregate, "-O", "json"],
           stdout=subprocess.PIPE)
    if res.returncode:
        print(res.stdout)
        res.check_returncode()
    for line in res.stdout.splitlines():
        data = json.loads(line)
        b = data.pop("bytes")
        p = data.pop("packets")
        data.update(LABELS)
        data["aggregate"] = aggregate
        export("flow_bytes", data, b)
        export("flow_packets", data, p)

## Test

curl localhost:9999/proxy?module=pmacct

# On prometheus server

  - job_name: pmacct
    scrape_interval: 1m
    metrics_path: /proxy
    params:
      module: [pmacct]
    static_configs:
      - targets:
        - pmacct.example.net:9999

[Deeraj V]

Thanks a lot juliu...@ @ Brian Candler for valuable answer and time. Let me go through 

Thanks once again.

Thanks

DV