Dashboard: abnormal display of quantity

98 views

Skip to first unread message

Dmitry

unread,

Apr 28, 2020, 10:38:05 AM4/28/20

to Prometheus Users

How to achieve optimal display of load values using SNMP Interface Throughput dashboard? I tried changing the format, but the display became abnormal.

level=info ts=2020-04-27T14:32:53.014Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.2, branch=HEAD, revision=18254838fbe25dcc732c950ae05f78ed4db1292c)"

level=info ts=2020-04-27T14:32:53.014Z caller=main.go:334 build_context="(go=go1.13.10, user=root@9cb154c268a2, date=20200420-08:27:08)"

level=info ts=2020-04-27T14:32:53.014Z caller=main.go:335 host_details="(Linux 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 e860ad394954 (none))"

level=info ts=2020-04-27T14:32:53.014Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"

level=info ts=2020-04-27T14:32:53.014Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2020-04-27T14:32:53.015Z caller=main.go:667 msg="Starting TSDB ..."

level=info ts=2020-04-27T14:32:53.015Z caller=web.go:515 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2020-04-27T14:32:53.020Z caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"

level=info ts=2020-04-27T14:32:53.024Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3

level=info ts=2020-04-27T14:32:53.024Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3

level=info ts=2020-04-27T14:32:53.025Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3

level=info ts=2020-04-27T14:32:53.025Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3

level=info ts=2020-04-27T14:32:53.025Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=4.83243ms

level=info ts=2020-04-27T14:32:53.027Z caller=main.go:683 fs_type=EXT4_SUPER_MAGIC

level=info ts=2020-04-27T14:32:53.027Z caller=main.go:684 msg="TSDB started"

level=info ts=2020-04-27T14:32:53.027Z caller=main.go:788 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T14:32:53.027Z caller=main.go:816 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T14:32:53.027Z caller=main.go:635 msg="Server is ready to receive web requests."

level=warn ts=2020-04-27T16:14:30.660Z caller=main.go:512 msg="Received SIGTERM, exiting gracefully..."

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:535 msg="Stopping scrape discovery manager..."

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:549 msg="Stopping notify discovery manager..."

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:571 msg="Stopping scrape manager..."

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:531 msg="Scrape discovery manager stopped"

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:545 msg="Notify discovery manager stopped"

level=info ts=2020-04-27T16:14:30.660Z caller=manager.go:875 component="rule manager" msg="Stopping rule manager..."

level=info ts=2020-04-27T16:14:30.660Z caller=manager.go:885 component="rule manager" msg="Rule manager stopped"

level=info ts=2020-04-27T16:14:30.660Z caller=main.go:565 msg="Scrape manager stopped"

level=info ts=2020-04-27T16:14:30.661Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."

level=info ts=2020-04-27T16:14:30.661Z caller=main.go:738 msg="Notifier manager stopped"

level=info ts=2020-04-27T16:14:30.661Z caller=main.go:750 msg="See you next time!"

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:298 msg="no time or size retention was set so using the default time retention" duration=15d

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.2, branch=HEAD, revision=18254838fbe25dcc732c950ae05f78ed4db1292c)"

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:334 build_context="(go=go1.13.10, user=root@9cb154c268a2, date=20200420-08:27:08)"

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:335 host_details="(Linux 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 e860ad394954 (none))"

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"

level=info ts=2020-04-27T16:14:31.228Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2020-04-27T16:14:31.230Z caller=main.go:667 msg="Starting TSDB ..."

level=info ts=2020-04-27T16:14:31.230Z caller=web.go:515 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2020-04-27T16:14:31.233Z caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"

level=info ts=2020-04-27T16:14:31.237Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4

level=info ts=2020-04-27T16:14:31.237Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4

level=info ts=2020-04-27T16:14:31.239Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4

level=info ts=2020-04-27T16:14:31.253Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4

level=info ts=2020-04-27T16:14:31.253Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4

level=info ts=2020-04-27T16:14:31.253Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=19.478768ms

level=info ts=2020-04-27T16:14:31.254Z caller=main.go:683 fs_type=EXT4_SUPER_MAGIC

level=info ts=2020-04-27T16:14:31.254Z caller=main.go:684 msg="TSDB started"

level=info ts=2020-04-27T16:14:31.254Z caller=main.go:788 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T16:14:31.255Z caller=main.go:816 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T16:14:31.255Z caller=main.go:635 msg="Server is ready to receive web requests."

level=warn ts=2020-04-27T16:16:22.658Z caller=main.go:512 msg="Received SIGTERM, exiting gracefully..."

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:535 msg="Stopping scrape discovery manager..."

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:549 msg="Stopping notify discovery manager..."

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:571 msg="Stopping scrape manager..."

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:531 msg="Scrape discovery manager stopped"

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:545 msg="Notify discovery manager stopped"

level=info ts=2020-04-27T16:16:22.658Z caller=manager.go:875 component="rule manager" msg="Stopping rule manager..."

level=info ts=2020-04-27T16:16:22.658Z caller=manager.go:885 component="rule manager" msg="Rule manager stopped"

level=info ts=2020-04-27T16:16:22.658Z caller=main.go:565 msg="Scrape manager stopped"

level=info ts=2020-04-27T16:16:22.660Z caller=notifier.go:598 component=notifier msg="Stopping notification manager..."

level=info ts=2020-04-27T16:16:22.660Z caller=main.go:738 msg="Notifier manager stopped"

level=info ts=2020-04-27T16:16:22.660Z caller=main.go:750 msg="See you next time!"

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:298 msg="no time or size retention was set so using the default time retention" duration=15d

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:333 msg="Starting Prometheus" version="(version=2.17.2, branch=HEAD, revision=18254838fbe25dcc732c950ae05f78ed4db1292c)"

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:334 build_context="(go=go1.13.10, user=root@9cb154c268a2, date=20200420-08:27:08)"

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:335 host_details="(Linux 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64 e860ad394954 (none))"

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:336 fd_limits="(soft=1048576, hard=1048576)"

level=info ts=2020-04-27T16:16:23.193Z caller=main.go:337 vm_limits="(soft=unlimited, hard=unlimited)"

level=info ts=2020-04-27T16:16:23.194Z caller=web.go:515 component=web msg="Start listening for connections" address=0.0.0.0:9090

level=info ts=2020-04-27T16:16:23.194Z caller=main.go:667 msg="Starting TSDB ..."

level=info ts=2020-04-27T16:16:23.197Z caller=head.go:575 component=tsdb msg="replaying WAL, this may take awhile"

level=info ts=2020-04-27T16:16:23.199Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5

level=info ts=2020-04-27T16:16:23.199Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5

level=info ts=2020-04-27T16:16:23.200Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5

level=info ts=2020-04-27T16:16:23.212Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5

level=info ts=2020-04-27T16:16:23.213Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5

level=info ts=2020-04-27T16:16:23.213Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5

level=info ts=2020-04-27T16:16:23.213Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=15.738232ms

level=info ts=2020-04-27T16:16:23.214Z caller=main.go:683 fs_type=EXT4_SUPER_MAGIC

level=info ts=2020-04-27T16:16:23.214Z caller=main.go:684 msg="TSDB started"

level=info ts=2020-04-27T16:16:23.214Z caller=main.go:788 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T16:16:23.215Z caller=main.go:816 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml

level=info ts=2020-04-27T16:16:23.215Z caller=main.go:635 msg="Server is ready to receive web requests."

level=info ts=2020-04-27T17:18:52.160Z caller=compact.go:495 component=tsdb msg="write block" mint=1587997130993 maxt=1588003200000 ulid=01E6YAC44G09S22842CA5WJDH0 duration=112.594089ms

level=info ts=2020-04-27T17:18:52.171Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.858503ms

level=info ts=2020-04-27T17:18:52.183Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=0 last=1 duration=11.792669ms

level=info ts=2020-04-27T19:00:01.151Z caller=compact.go:495 component=tsdb msg="write block" mint=1588003200000 maxt=1588010400000 ulid=01E6YG5AWQS6FM3S2S7P3RWZP8 duration=104.306442ms

level=info ts=2020-04-27T19:00:01.159Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.317661ms

level=info ts=2020-04-27T19:00:01.170Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=2 last=3 duration=10.733994ms

level=info ts=2020-04-27T21:00:01.108Z caller=compact.go:495 component=tsdb msg="write block" mint=1588010400000 maxt=1588017600000 ulid=01E6YQ124JRMQ54J1ZXVW8Z7RQ duration=65.409989ms

level=info ts=2020-04-27T21:00:01.115Z caller=head.go:662 component=tsdb msg="head GC completed" duration=878.557µs

level=info ts=2020-04-27T23:00:01.102Z caller=compact.go:495 component=tsdb msg="write block" mint=1588017600000 maxt=1588024800000 ulid=01E6YXWSCM7JTY8NQDJZ77AM5J duration=58.759606ms

level=info ts=2020-04-27T23:00:01.106Z caller=head.go:662 component=tsdb msg="head GC completed" duration=576.681µs

level=info ts=2020-04-27T23:00:01.120Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=4 last=5 duration=14.580929ms

level=info ts=2020-04-27T23:00:01.184Z caller=compact.go:441 component=tsdb msg="compact blocks" count=2 mint=1587997130993 maxt=1588010400000 ulid=01E6YXWSF4PTPSV0GTNEQSSSWF sources="[01E6YAC44G09S22842CA5WJDH0 01E6YG5AWQS6FM3S2S7P3RWZP8]" duration=60.119548ms

level=info ts=2020-04-28T01:00:01.115Z caller=compact.go:495 component=tsdb msg="write block" mint=1588024800000 maxt=1588032000000 ulid=01E6Z4RGMPF2102942T86T0248 duration=69.202234ms

level=info ts=2020-04-28T01:00:01.123Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.485947ms

level=info ts=2020-04-28T03:00:01.148Z caller=compact.go:495 component=tsdb msg="write block" mint=1588032000000 maxt=1588039200000 ulid=01E6ZBM7WM9QNK23HVSXQQ01MW duration=103.792783ms

level=info ts=2020-04-28T03:00:01.154Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.165593ms

level=info ts=2020-04-28T03:00:01.205Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=6 last=7 duration=50.417287ms

level=info ts=2020-04-28T03:00:01.284Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1588010400000 maxt=1588032000000 ulid=01E6ZBM81SHMHYEM9RZSPWNMHC sources="[01E6YQ124JRMQ54J1ZXVW8Z7RQ 01E6YXWSCM7JTY8NQDJZ77AM5J 01E6Z4RGMPF2102942T86T0248]" duration=75.481504ms

level=info ts=2020-04-28T05:00:01.111Z caller=compact.go:495 component=tsdb msg="write block" mint=1588039200000 maxt=1588046400000 ulid=01E6ZJFZ4M9S6QPW4G9BRBS6VR duration=66.1857ms

level=info ts=2020-04-28T05:00:01.117Z caller=head.go:662 component=tsdb msg="head GC completed" duration=679.72µs

level=info ts=2020-04-28T07:00:01.117Z caller=compact.go:495 component=tsdb msg="write block" mint=1588046400000 maxt=1588053600000 ulid=01E6ZSBPCP2KMTMT1FNKV4BE7Q duration=71.150364ms

level=info ts=2020-04-28T07:00:01.125Z caller=head.go:662 component=tsdb msg="head GC completed" duration=677.02µs

level=info ts=2020-04-28T07:00:01.174Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=8 last=9 duration=49.042165ms

level=info ts=2020-04-28T09:00:01.109Z caller=compact.go:495 component=tsdb msg="write block" mint=1588053600000 maxt=1588060800000 ulid=01E7007DMP82ERD9NJZSGK8PWF duration=63.24805ms

level=info ts=2020-04-28T09:00:01.113Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.143854ms

level=info ts=2020-04-28T09:00:01.201Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1588032000000 maxt=1588053600000 ulid=01E7007DPWDYSY30F1A3NFNACM sources="[01E6ZBM7WM9QNK23HVSXQQ01MW 01E6ZJFZ4M9S6QPW4G9BRBS6VR 01E6ZSBPCP2KMTMT1FNKV4BE7Q]" duration=84.579036ms

level=info ts=2020-04-28T11:00:01.114Z caller=compact.go:495 component=tsdb msg="write block" mint=1588060800000 maxt=1588068000000 ulid=01E70734WPYXWC1NZJ2Z2Z3KM8 duration=68.547898ms

level=info ts=2020-04-28T11:00:01.121Z caller=head.go:662 component=tsdb msg="head GC completed" duration=1.103804ms

level=info ts=2020-04-28T11:00:01.171Z caller=head.go:732 component=tsdb msg="WAL checkpoint complete" first=10 last=11 duration=50.198202ms

level=info ts=2020-04-28T11:00:01.265Z caller=compact.go:441 component=tsdb msg="compact blocks" count=3 mint=1587997130993 maxt=1588053600000 ulid=01E707350QSGBTTJ9JXCQV5FPY sources="[01E6YXWSF4PTPSV0GTNEQSSSWF 01E6ZBM81SHMHYEM9RZSPWNMHC 01E7007DPWDYSY30F1A3NFNACM]" duration=90.278861ms

level=info ts=2020-04-28T13:00:01.110Z caller=compact.go:495 component=tsdb msg="write block" mint=1588068000000 maxt=1588075200000 ulid=01E70DYW4Q6YKP3Z6BW1VR5NFE duration=63.118489ms

level=info ts=2020-04-28T13:00:01.114Z caller=head.go:662 component=tsdb msg="head GC completed" duration=655.53µs

1.GIF

2.GIF

3.GIF

Brian Candler

unread,

Apr 28, 2020, 1:39:26 PM4/28/20

to Prometheus Users

1. You'll need to say what you mean by "abnormal". Those graphs just look like perfectly normal graphs to me, except the values on 2.GIF are very large. The third image is so tiny it makes no sense.

2. You'll need to show what PromQL queries you are sending.

3. I am guessing this is Grafana you're showing? Grafana isn't part of prometheus, so you'd be better off first formulating your queries in the prometheus web interface (normally on port 9090). When you have queries that show what you expect, then migrate them to Grafana.

Dmitry

unread,

Apr 28, 2020, 2:35:18 PM4/28/20

to Prometheus Users

Sorry, I’ll supplement the information. These are the metrics I get from my device. I did not copy everything, since it is the octets that are taken and visualized in the grafana dashboard. I wanted to change the scale from kbps to mbps. So the chart would be displayed in megabits. And if I edit the dashboard json and change the values to megabits, then they become very large, as in my second screenshot, which is not true. There can be no such values.
I'm using the snmp_exporter and dashboard.

HELP ifHCInOctets The total number of octets received on the interface, including framing characters - 1.3.6.1.2.1.31.1.1.1.6
# TYPE ifHCInOctets counter
ifHCInOctets{ifAlias="",ifDescr="ether1",ifIndex="3",ifName="ether1"} 1.147762052e+09
# HELP ifHCOutOctets The total number of octets transmitted out of the interface, including framing characters - 1.3.6.1.2.1.31.1.1.1.10
# TYPE ifHCOutOctets counter
ifHCOutOctets{ifAlias="",ifDescr="ether1",ifIndex="3",ifName="ether1"} 3.90196102e+08
# HELP ifInOctets The total number of octets received on the interface, including framing characters - 1.3.6.1.2.1.2.2.1.10
# TYPE ifInOctets counter
ifInOctets{ifAlias="",ifDescr="ether1",ifIndex="3",ifName="ether1"} 1.147762052e+09
# HELP ifOutOctets The total number of octets transmitted out of the interface, including framing characters - 1.3.6.1.2.1.2.2.1.16
# TYPE ifOutOctets counter
ifOutOctets{ifAlias="",ifDescr="ether1",ifIndex="3",ifName="ether1"} 3.90196102e+08

Here i try to change format to megabits.
"yaxes": [
            {
              "format": "Bps",
              "label": null,
              "logBase": 1,
              "max": null,
              "min": null,
              "show": true

4.GIF

Brian Candler

unread,

Apr 28, 2020, 3:21:02 PM4/28/20

to Prometheus Users

You still haven't shown your query. I will guess it's something like:

irate(ifHCInOctets[5m])

If you view this in PromQL browser, you should get a value in bytes per second. If you want bits per second, then:

irate(ifHCInOctets[5m])*8

To visualize this correctly in Grafana, all you have to do is to go to the Visualization button and select Data Rate > bytes/sec (or bits/sec)

Grafana will automatically scale the axis. If it's 2 bits per second, it will display 2 bps. if it's 2000000, it will display 2 Mbps.

Reply all

Reply to author

Forward

0 new messages