Smokeping_prober CPU usage optimization possible?

[Alexander Wilke]

Hello,

I am running smokeping_prober from one VM to Monitor around 500 destinations.

Around 30 devices are monitored with 0.2s Intervall and Others with 1.65s Intervall.

Prometheus scrapes every 5s.

So there are roughly 600 icmp ipv4 24byte pings per Seconds.

CPU usage jumps between 700-1200% using "top"

What Else except reducing Interval or Host Count could Help to reduce CPU usage?

Is the UDP Socket "better" or any other optimization which could be relevant for that Type of Traffic? Running on RHEL8

Someone with similar CPU usage and this amount of pings per Seconds? Maybe Others Ping 6.000 Destination every 10s?

[Ben Kochie]

Best thing you can do is capture some pprof data. That will show you what it's spending the time on.

:9374/debug/pprof/heap 

:9374/debug/pprof/profile?seconds=30

You can post the results to https://pprof.me/ for sharing.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d803c1a2-64ee-48d1-8513-b864856f53c8n%40googlegroups.com.

[Alexander Wilke]

Hello,

sorry for the delay. here are the results. to be honest - I do not understand anything of it.

Smokeping_Prober Heap:

https://pprof.me/a1e7400d32859dbc217e2182398485df/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes&dashboard_items=icicle

 

smokeping_prober profile30s

https://pprof.me/340674b335e114e4b0df6b4582f0644e/?profileType=profile%3Asamples%3Acount%3Acpu%3Ananoseconds%3Adelta

[Alexander Wilke]

Hello,

any Chance to investigate the Reports and any suggestions?

[Ben Kochie]

Looking at the CPU profile, I'm seeing almost all the time spent in the Go runtime. Mostly the ICMP packet receiving code and garbage collection. I'm not sure there's a lot we can optimize here as it's core Go code for ICMP packet handling.

Can you also post me a graph of a few metrics queries?

rate(process_cpu_seconds_total{job="smokeping_prober"}[30s])

rate(go_gc_duration_seconds_count{job="smokeping_prober"}[5m])

rate(go_gc_duration_seconds_sum{job="smokeping_prober"}[5m])

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/62aeaacb-9fd6-4d64-8bec-d7171f592766n%40googlegroups.com.

[Alexander Wilke]

Hello,

I attached a few screenshots showing the results and graphs for 1h and 6h.

In addition I added a screenshot from node_exporter metrics to give you an overview of the system itself.

On the same system there is prometheus, grafana, snmp_exporter (200-800% CPU), smokeping prober, node_exporter, blackbox_exporter.

The main CPU consumers are snmp_exporter and smokeping.

[Ben Kochie]

Interesting, thanks for the data. It does seem like the process is spending a lot of time doing GC like I thought.

One trick you could try is to increase the memory allocated to the prober, which would reduce the time spent on GC.

The default setting is is GOGC=100.

You could try increasing this by setting the environment variable, GOGC.

Try something like GOGC=200 or GOGC=300.

This will make the process use more memory, but it should reduce the CPU time spent.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/045a6fb0-b82c-4425-8bae-e2c99403b2adn%40googlegroups.com.

[Alexander Wilke]

Hello Ben,

I googled a little bit and found this:

https://github.com/prometheus/prometheus/issues/2665#issuecomment-342149607

As far as I understand this variable is not working anymore or not used anymore !?
I tried in a test environment:

export GOGC=200

And the restarted (not reload) prometheus and in the UI --> "Runime & Build Information" the GOGC is still empty.

1.) Is this environment variable set correctly?

2.) Is the variable still working?

3.) If it is still working can I apply it only to smokeping_prober but not other services like prometheus? Sounds like higher GOGC has tradeoffs for queries in the prometheus tsdb ?

[Ben Kochie]

The variable change is for the smokeping prober, not prometheus.

I don't know how you run your services.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bc0953fd-9d80-48ce-8263-d9f5882db6d5n%40googlegroups.com.