Wrong Measurement of sFlow with Prometheus

[ABCD]

Hi. I appreciate your S/W and this channel to talk with it.

I am using an OvS constructing a SDN with a SDN Controller.

I am trying to measure bytes between hosts using sFlow.

Here is the simple topology:

This explains how my SDN works.

It's a bit simple.

The server contains SDN controller, Prometheus, sFlow-RT with exporter.

I started iPerf3, sending 10Mbps for 60 seconds.

It was totally 71.6MB by iPerf3 and wireshark saids it corresponds to 75,530,368B (72.0MB). They are slightly different. (I guess the packet headers caused this)

Anyway, I checked it with sFlow-RT and Prometheus, it said

96,518,344 (If it is octet, it corresponds to 96.5MB)

So I just wonder why is it too much different with the accurate traffic.

I configured sFlow in OvS with

Sample rates 128, polling 20

I don't know why this happen so can you let me know if there is sth I am missing?

[Peter Phaal]

You don't mention the Prometheus scrape interval for sFlow-RT flow metrics, or the flow definition. The value of t (default 15 seconds for the sFlow-RT prometheus app) should be at least as large as the scrape interval, see Flow Smoothing.

In addition, you have a sampling rate of 128. A 60 second iperf test at 10Mbit/s would be expected to generate around 50,000 1500 byte packets, which sampled at 1:128, would yield approximately 400 samples. This means you estimate from sFlow should be within +/- 8.5 % of the correct value, see Packet Sampling Basics. The relative error in each second, or each 15 second, scrape interval would be considerably higher. If you are trying to detect 10Mbps per second flows reliably for SDN applications, then the recommended sampling rate is 1 in 10, see Large flow detection.

It also looks like you queried Prometheus using a 29m (29 minute interval size)?

In any case, the real-time flow rate calculated by sFlow-RT is primarily intended for rapid detection and control of large flows (DDoS detection, load balancing, etc). If you want to get accurate flow totals, then you are better off configuring sFlow-RT to log flow records, since the log entries will contain the total volume of the flow, not the instantaneous rate, see Handle flow records.

[Peter Phaal]

Please don't drop the sflow-rt group when replying.

I would suggest that you use the browse-flows app to troubleshoot, https://github.com/sflow-rt/browse-flows. Set up a flow with keys: ipsource,ipdestination and value: bps. Run your iperf3 test and verify that the traffic rate is being reported correctly during the 60 second test.

One possible source of error might be how bytes are counted. The "bytes" value for sFlow-RT starts at the beginning of the layer 2 header. An iperf UDP test might only be counting UDP payload bytes. Change the value in browse-flows to "ipbytes" to track IP payload bytes or "udpbytes" to track UDP payload bytes. Generally it is a good idea to experiment with flow definitions in browse-flows as a first step before building the Prometheus scrape task.

Another check you can do is to install browse-metrics, https://github.com/sflow-rt/browse-metrics, and verify that the ifinoctets counter associated with the switch port connecting to the iperf3 traffic generator reports the expected value. If there is no other network activity on the OVS instance then the sflow-test application will automatically do this cross check for you, https://github.com/sflow-rt/sflow-test

On Tue, Mar 18, 2025 at 11:20 PM ABCD <pjhpj...@gmail.com> wrote:

Yeah, I missed my prometheus.yml config.

After I read your reply, I tried to figure out what's wrong.

So I configured prometheus as your response.

In my prometheus.yml:

global:

  scrape_interval: 10s

  evaluation_interval: 15s

#...

  - job_name: "sflow-rt-exporter"

    metrics_path: /app/prometheus/scripts/export.js/flows/ALL/txt

    static_configs:

      - targets: ["localhost:8008"]

    params:

      metric: ["sflow_bytes"]

      key:

        [

          "null:agent:unknown",

          "null:ipsource:unknown",

          "null:ipdestination:unknown",

          "null:tcpsourceport:unknown",

          "null:tcpdestinationport:unknown",

          "null:udpsourceport:unknown",

          "null:udpdestinationport:unknown",

          "null:ipprotocol:unknown",

          "null:ethernetprotocol:unknown",

        ]

      label:

        [

          "agent",

          "srcIP",

          "dstIP",

          "tcpSrcPort",

          "tcpDstPort",

          "udpSrcPort",

          "udpDstPort",

          "ipProtocolType",

          "ethernetType",

        ]  

      value: ['bytes']

      scale: ['8']

      aggMode: ['sum']

      minValue: ['1000']

      maxFlows: ['1000000']

      t: ['10']

I synchronized t with my scrape interval which is 10s.

I configured aggMode as "sum" but I guess this is a default value for it.

increase the sampling rate from 128 to 10 (1 packet over 10)

admin@PICOS-OVS:~$ sudo ovs-vsctl list sflow
_uuid               : 760a84d8-25ac-4a14-a795-e95efd1bb722
agent               : "eth0"
external_ids        : {}
header              : 128
polling             : 20
sampling            : 10
targets             : ["172.16.10.149:6343"]

2 options I changed but still meet the same problem as below:

Prometheus: 83371516 = 79.509274MB

iPerf3: 71.6MB

Actually after iperf3 test, the bytes keep getting bigger even it is ended 30 seconds ago. It lasts a 2~5 minutes. now it became 83705343

I guess this happened because of duplicated counter sum or something, I don't know.

Maybe I need the third option you suggested that Hanling flow records, right?

2025년 3월 18일 화요일 오전 1시 25분 17초 UTC+9에 peter...@gmail.com님이 작성:

[ABCD]

Now I got who is confusing me.

But first, let me tell the result of what did you say.

First, I append "group" to my promtheus.yml. but it doesn't affect my prometheus wrong measurement.

Here is the prometheus.yml snippet

--------------------------------------------------------------------------------

  - job_name: "sflow-rt-exporter"
      ...
      t: ['10']
      group: ['this']

--------------------------------------------------------------------------------

Secondly, I installed apps: browse-flows, browse-metrics, sflow-test and tested it.

this is browse-flows, key: ipsource (and ipdestination, too), value: bps.

As I mentioned at my reply, the graph lasts a few even the iperf3 ended up already.

the "bps" resulted well, but prometheus is still wrong.

Third, test with ipbytes or udpbytes/tcpbytes.

this is "ipbytes"

ip bytes shows the same graph form while tcpbytes, udpbytes doesn't work.

So I tried to configure prometheus.yml value to "ipbytes" so that I can measure the exact bytes.

--------------------------------------------------------------------------------

  - job_name: "sflow-rt-exporter"
      ...

      value: ['ipbytes']

      ...
      t: ['10']
      group: ['this']

--------------------------------------------------------------------------------

But it still exceeded the exact bytes(715MB).

Lastly, I tried to test it with browse-metrics and sflow-test (cross-check)

Still vague, the prometheus says wrong number.

But finally I figured out what makes this happening.

I mean, prometheus result keep increasing even the iperf test ended up.

It's because of the "t" parameter which means "smoothing level or time".

It makes the graph's end round and increases the counts number, I guess.

So I tested it with various "t" values (in prometheus.yml) and its result says:

t = 500 (maximum), the increment lasts 48minutes,

t = 10, the increment lasts 3 minutes, prometheus measurement: 888.131557MB

t = 1, the increment lasts 70~75 seconds, prometheus measurement: 802.036606MB

t = 0.8, the increment lasts 70~75 seconds, prometheus measurement: 812.605983MB (it increased)

t = 0.5, the increment lasts 70~75 seconds, prometheus measurement: varies. 1,207.39316MB, 1,018.61167MB, 852.924301MB, 1,110.31862MB

I concluded that this problem is related with "t" value.

It is the level of smoothing. (I already read the article)

But still don't know how to fix it.

And since my network users have a variety range of traffic, it is unable to find the best "t" value.

I guess I should write a program which utilize the flow record bytes.

Is my guess correct?

2025년 3월 20일 목요일 오전 1시 20분 9초 UTC+9에 peter...@gmail.com님이 작성:

[Peter Phaal]

Thanks for the detailed reply. Yes, the t value is critical to plotting time series graphs. The correct value for t depends on your application. For example, a small value of t makes the measurements more responsive so that you can detect and respond to a DDoS attack quickly. On the other hand, when logging data to Prometheus and plotting Grafana charts, a higher t value ensures that the charts are smoother and more accurate. In this situation I would generally recommend a t value that is double the prometheus scrape interval, i.e. for a scrape interval of 10 seconds you would set t=20.

A dashboard is useful for addressing operational concerns, e.g. is the network congested? who is generating the traffic? etc. However, this isn't the best method of generating accounting data. If you want accurate daily / weekly totals then you are better off setting log:true in the flow definitions and then summing the resulting flow records, either by tailing the flow log using the REST API, or by writing a javascript sFlow-RT application with setFlowHandler() to process the logs.

[ABCD]

Since I am getting through a grind, can you advise for me?

To be honest, I am not familiar with javascript, RESTful API.

I tried with this code:

sflow-rt/my-app/scripts/account.js

--------------------------------------------------------------------------------

// author:

// version: 0.1

// date: 3/21/2025

// description: sFlow-RT account flow record bytes

var maxFlows = 100;

var minValue = 1000;

var t = 2;

setFlow("myFlow", {

  keys: "ipsource, ipdestination, tcpsourceport, tcpdestinationport, udpsourceport, udpdestinationport, ipprotocol, ethernetprotocol",

  value: "bytes",

  filter: null,

  t: t,

  n: 20,

  log: true,

});

setFlowHandler(function (flow) {

  logInfo(JSON.stringify(flow));

});

function deleteUserFlows(now) {

  clearFlow("myFlow");

}

setIntervalHandler(function (now) {

  deleteUserFlows(now);

});

setHttpHandler(function (req) {

  if (!req.path || req.path.length === 0) {

    return { message: "sFlow-RT API is running", request: req };

  }

  switch (req.path[0]) {

    case "flows":

      return { message: "Flows API is active", request: req };

    default:

      throw "not_found";

  }

});

--------------------------------------------------------------------------------

But it doesn't have any effects. I don't know why.

Let me guess,

1. If I write an application with sflow-rt, I am wondering whether the app "prometheus" get the accurate total numbers.

Because it seems the data which "prometheus" app exports is the same as I want to do. (log for acurate total bytes)

I think it would be useless if I left a log (setFlow(name, {log: true,})) because it might be same with prometheus values.

That means, the problem I reported on this page is just limited to my switch.

2. How about just write "sFlow prometheus exporter" not via sFlow-RT?

If this can be effective, I will start to write a Go Code which exports prometheus data from sFlow samples directly.

2-2. What should packet I grab if I want to account the accruate bytes among huge and various data?

Counter sample? or Flow Sample?

As I know, the Counter sample is preferred but it can be a serious problem if the packet is malformed.

2025년 3월 21일 금요일 오전 4시 14분 41초 UTC+9에 peter...@gmail.com님이 작성:

[Peter Phaal]

Prometheus isn't suitable for the high cardinality data you are collecting with the flow attributes in your script: "ipsource, ipdestination, tcpsourceport, tcpdestinationport, udpsourceport, udpdestinationport, ipprotocol, ethernetprotocol". This detail will overwhelm the database, making it large and slow to query. Also, tcpsourceport and udpsourceport are typically mutually exclusive (unless you are monitoring tunneled traffic) and so you won't see any flow records generated. The sFlow-RT prometheus app is best used to reduce the cardinality of the data in order to populate high level dashboards, for example, sFlow-RT Countries and Networks, rolls up flow data by source/destination country code and ASN.

If your goal is traffic accounting then typically you only need ipsource address as the key and perhaps a filter to exclude traffic to local destination ip addresses. If you want bidirectional total then you would create a second flow definition for ipdestination. The flowHandler can then keep the sum of bytes by IP address, something like total[flow.flowKeys] += flow.value. You could then modify the httpHandler to allow access to the total[] values.

The value returned by the sFlow-RT prometheus app is not the same as the total of the flow records. The prometheus app uses the sFlow-RT activeFlows() query to get the instantaneous value of the bytes/second rate computed by sFlow-RT for each flow. A flow record contains the sum of bytes observed since the last flow record logged for the flow.

The counter data you showed in your screen capture is not malformed, it's just a poorly worded warning indicating that the sFlow 1004 record type (see sFlow OpenFlow Structures) is not understood by the packet analyzer. Note: This structure is understood by sFlow-RT and is useful for SDN applications, see Hybrid OpenFlow ECMP testbed for an example. Counter data is often used for accounting by service providers, but has the disadvantage that you cannot separate traffic classes (local vs non-local traffic, etc).

[ABCD]

I really appreciate you.

Tho the problem still remains, but it was really helpful for me. I really tried hard to aggregate bytes from aggregation. Now I got that the cause of this happening is my switch exports wrong sflow packets. I tested to aggregate with a tiny sflow-rt app and simply added the values from another OvS (in software form). The bytes are accurately corresponding to my iperf3 test. So the answer is my HW machine which is operated by NOS prints wrong values. This is not a sFlow, sFlow-RT and prometheus problem.

Now I am trying to monitor flows on SDN Controller platform. This is a grind too, but no more than dealing with a stupid switch.

Even though I couldn't get satisfying result from this dialogue, it was pretty helpful for me to know much more about this amazing application. Shortly, a study for me. Thanks again to you.

2025년 3월 25일 화요일 오전 1시 24분 21초 UTC+9에 peter...@gmail.com님이 작성: