snmp-exporter Upgrade 0.19 to 0.20 nil pointer exception, if device is not reachable

279 views
Skip to first unread message

ohey...@gmail.com

unread,
Aug 31, 2021, 6:49:31 AM8/31/21
to Prometheus Users
Hi,

I just did an upgrade from prom/snmp-exporter:v0.19.0 to prom/snmp-exporter:v0.20.0  and now I get a stack trace for devices, which are not reachable. Any idea why? First time tests with 0.20.0 release. This didn't happen with the 0.19.0 release.

Regards,

Olaf

```
level=info ts=2021-08-31T10:42:37.654Z caller=collector.go:224 module=ipn target=10.17.125.48 msg="Error scraping target" err="error getting target 10.17.125.48: recover: runtime error: invalid memory address or nil pointer dereference\nStack:goroutine 177 [running]:\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc0003195c8)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:326 +0xa5\npanic(0x9e4880, 0xe89790)\n\t/usr/local/go/src/runtime/panic.go:969 +0x1b9\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0000da900, 0xc0001e82a0, 0xc000bd0101, 0x0, 0xb34ee0, 0xc000c0a020)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:372 +0x20e\ngithub.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0000da900, 0xc000c70b60, 0x2, 0x2, 0x0, 0x0, 0xc000c3c1b0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/gosnmp.go:363 +0x159\nmain.ScrapeTarget(0xb40660, 0xc00043a200, 0xc00034201c, 0xc, 0xc0001912c0, 0xb34f40, 0xc0003b9e00, 0x0, 0x0, 0x0, ...)\n\t/app/collector.go:133 +0x5be\nmain.collector.Collect(0xb40660, 0xc00043a200, 0xc00034201c, 0xc, 0xc0001912c0, 0xb34f40, 0xc0003b9e00, 0xc0000bf140)\n\t/app/collector.go:222 +0xbc\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:446 +0x1a2\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:457 +0x5ce\n\ngoroutine 1 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f6b76821eb8, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000bdc018, 0x72, 0x0, 0x0, 0xa85b68)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc000bdc000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc\nnet.(*netFD).accept(0xc000bdc000, 0xc5c19e3f0727fe93, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_unix.go:172 +0x45\nnet.(*TCPListener).accept(0xc000bc8300, 0x612e0785, 0xc00050fad0, 0x48e3a6)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 +0x32\nnet.(*TCPListener).Accept(0xc000bc8300, 0xc00050fb20, 0x18, 0xc000000180, 0x7f8dec)\n\t/usr/local/go/src/net/tcpsock.go:261 +0x65\nnet/http.(*Server).Serve(0xc000bda000, 0xb3ee20, 0xc000bc8300, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2937 +0x266\ngithub.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc000bc8300, 0xc000bda000, 0x0, 0x0, 0xb34f40, 0xc0002343c0, 0x0, 0xe0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:192 +0x1b0\ngithub.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc000bda000, 0x0, 0x0, 0xb34f40, 0xc0002343c0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:184 +0xfd\nmain.main()\n\t/app/main.go:248 +0xb71\n\ngoroutine 32 [syscall, 1 minutes]:\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:147 +0x9d\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x25\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:150 +0x45\n\ngoroutine 49 [select, 1 minutes]:\nmain.main.func1(0xc0000be1e0, 0xb34f40, 0xc0002343c0)\n\t/app/main.go:179 +0xe5\ncreated by main.main\n\t/app/main.go:177 +0x7ce\n\ngoroutine 53 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0009b01e0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc00080e460, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nmain.handler(0xb3f0a0, 0xc0003500e0, 0xc000571a00, 0xb34f40, 0xc0002343c0)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/app/main.go:201 +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc000bc82a0, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/usr/local/go/src/net/http/server.go:2417 +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000bda000, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)\n\t/usr/local/go/src/net/http/server.go:2843 +0xa3\nnet/http.(*conn).serve(0xc000230500, 0xb40660, 0xc000bd2180)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 262 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f6b76821208, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000b6e318, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000b6e300, 0xc0002a8000, 0xffff, 0xffff, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5\nnet.(*netFD).Read(0xc000b6e300, 0xc0002a8000, 0xffff, 0xffff, 0x9bd360, 0x38, 0x40)\n\t/usr/local/go/src/net/fd_posix.go:55 +0x4f\nnet.(*conn).Read(0xc0004c0010, 0xc0002a8000, 0xffff, 0xffff, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:182 +0x8e\ngithub.com/gosnmp/gosnmp.(*GoSNMP).receive(0xc000bbe000, 0xc000c600c0, 0x38, 0x40, 0x38, 0x0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:1149 +0x58\ngithub.com/gosnmp/gosnmp.(*GoSNMP).sendOneRequest(0xc000bbe000, 0xc000a00620, 0xc0001f9501, 0x40e2b8, 0xe0, 0xa703c0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:223 +0x437\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send(0xc000bbe000, 0xc000a00620, 0xc00090e101, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:349 +0xf1\ngithub.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc000bbe000, 0xc0004937c0, 0x2, 0x2, 0x0, 0x0, 0xc00058e3c0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/gosnmp.go:363 +0x159\nmain.ScrapeTarget(0xb40660, 0xc0007de000, 0xc00034205c, 0xd, 0xc000190870, 0xb34f40, 0xc00058e090, 0x0, 0x0, 0x0, ...)\n\t/app/collector.go:133 +0x5be\nmain.collector.Collect(0xb40660, 0xc0007de000, 0xc00034205c, 0xd, 0xc000190870, 0xb34f40, 0xc00058e090, 0xc0006940c0)\n\t/app/collector.go:222 +0xbc\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:446 +0x1a2\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:457 +0x5ce\n\ngoroutine 37 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00062e050, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc000b79650, 0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nmain.handler(0xb3f0a0, 0xc000620000, 0xc000570200, 0xb34f40, 0xc0002343c0)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/app/main.go:201 +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc000bc82a0, 0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/usr/local/go/src/net/http/server.go:2417 +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000bda000, 0xb3f0a0, 0xc000620000, 0xc000570200)\n\t/usr/local/go/src/net/http/server.go:2843 +0xa3\nnet/http.(*conn).serve(0xc0000d81e0, 0xb40660, 0xc000860000)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 211 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f6b76821c00, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc00081e298, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc00081e280, 0xc000bde000, 0"
level=debug ts=2021-08-31T10:42:37.655Z caller=main.go:113 module=ipn target=10.17.125.48 msg="Finished scrape" duration_seconds=40.02079067
```

Brian Candler

unread,
Aug 31, 2021, 7:53:49 AM8/31/21
to Prometheus Users
I can't reproduce that panic:

# curl 'localhost:9116/snmp?module=if_mib&target=1.1.1.1&debug=true'
An error has occurred while serving metrics:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error getting target 1.1.1.1: request timeout (after 3 retries)

# journalctl -eu snmp_exporter
... (no panic)
Aug 31 11:48:11 prometheus snmp_exporter[207]: level=info ts=2021-08-31T11:48:11.386Z caller=collector.go:224 module=if_mib target=1.1.1.1 msg="Error scraping target" err="error getting target 1.1.1.1: request timeout (after 3 retries)"

I am running the official release binary (snmp_exporter-0.20.0.linux-amd64.tar.gz) under Ubuntu 20.04.2 LTS x86_64

Can you say:
- which platform are you running on?
- where did your snmp_exporter binary come from?  Are you using a release binary, or did you compile it yourself?  If you compiled it yourself, what version of go did you use?
- what does this curl command show?
# curl -Ss localhost:9116/metrics | grep build_info

In my case I get:
# HELP snmp_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which snmp_exporter was built.
# TYPE snmp_exporter_build_info gauge
snmp_exporter_build_info{branch="HEAD",goversion="go1.15.8",revision="c33572b6c8c8e43a479fde0f9fa8ac86e15598bc",version="0.20.0"} 1

ohey...@gmail.com

unread,
Aug 31, 2021, 8:58:44 AM8/31/21
to Prometheus Users
Hi Brian, 

I am using the docker image from https://hub.docker.com/r/prom/snmp-exporter/tags?page=1&ordering=last_updated, which is prom/snmp-exporter:v0.20.0  linux/amd64.
# TYPE snmp_exporter_build_info gauge
snmp_exporter_build_info{branch="HEAD",goversion="go1.15.8",revision="c33572b6c8c8e43a479fde0f9fa8ac86e15598bc",version="0.20.0"} 1


I just noticed that with 0.19.0 I get a onliner exception:

snmp-exporter v0.19.0 docker image:

level=info ts=2021-08-31T12:24:40.270Z caller=collector.go:224 module=ipn target=10.17.125.14 msg="Error scraping target" err="error getting target 10.17.125.14: recover: runtime error: invalid memory address or nil pointer dereference"
level=debug ts=2021-08-31T12:24:40.271Z caller=main.go:110 module=ipn target=10.17.125.14 msg="Finished scrape" duration_seconds=40.003079323

Doing an snmpwalk I get the timeout: 

snmpwalk -v3 -l authPriv -u bla -a sha -A blabla -x aes -X blablabla 10.17.125.14 1
Timeout: No Response from 10.17.125.14

Regards,
Olaf

Brian Candler

unread,
Aug 31, 2021, 9:01:17 AM8/31/21
to Prometheus Users
Can you reproduce the panic using "curl" to send a request, as per my example?  Maybe it's something about the way prometheus is talking to snmp_exporter.

Here is your backtrace, reformatted so that it's readable.  It looks to me like the pointer *GoSNMP is null.

error getting target 10.17.125.48: recover: runtime error: invalid memory address or nil pointer dereference
Stack:goroutine 177 [running]:
panic(0x9e4880, 0xe89790)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0000da900, 0xc0001e82a0, 0xc000bd0101, 0x0, 0xb34ee0, 0xc000c0a020)
github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0000da900, 0xc000c70b60, 0x2, 0x2, 0x0, 0x0, 0xc000c3c1b0)
main.ScrapeTarget(0xb40660, 0xc00043a200, 0xc00034201c, 0xc, 0xc0001912c0, 0xb34f40, 0xc0003b9e00, 0x0, 0x0, 0x0, ...)
/app/collector.go:133 +0x5be
main.collector.Collect(0xb40660, 0xc00043a200, 0xc00034201c, 0xc, 0xc0001912c0, 0xb34f40, 0xc0003b9e00, 0xc0000bf140)
/app/collector.go:222 +0xbc

goroutine 1 [IO wait]:
internal/poll.runtime_pollWait(0x7f6b76821eb8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000bdc018, 0x72, 0x0, 0x0, 0xa85b68)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000bdc000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000bdc000, 0xc5c19e3f0727fe93, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000bc8300, 0x612e0785, 0xc00050fad0, 0x48e3a6)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000bc8300, 0xc00050fb20, 0x18, 0xc000000180, 0x7f8dec)
/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc000bda000, 0xb3ee20, 0xc000bc8300, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2937 +0x266
github.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc000bc8300, 0xc000bda000, 0x0, 0x0, 0xb34f40, 0xc0002343c0, 0x0, 0xe0)
main.main()
/app/main.go:248 +0xb71

goroutine 32 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 49 [select, 1 minutes]:
main.main.func1(0xc0000be1e0, 0xb34f40, 0xc0002343c0)
/app/main.go:179 +0xe5
created by main.main
/app/main.go:177 +0x7ce

goroutine 53 [select]:
net/http.HandlerFunc.ServeHTTP(0xc00080e460, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)
/usr/local/go/src/net/http/server.go:2042 +0x44
main.handler(0xb3f0a0, 0xc0003500e0, 0xc000571a00, 0xb34f40, 0xc0002343c0)
/app/main.go:110 +0x6d7
main.main.func2(0xb3f0a0, 0xc0003500e0, 0xc000571a00)
/app/main.go:201 +0x51
net/http.HandlerFunc.ServeHTTP(0xc000bc82a0, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000bda000, 0xb3f0a0, 0xc0003500e0, 0xc000571a00)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000230500, 0xb40660, 0xc000bd2180)
/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 262 [IO wait]:
internal/poll.runtime_pollWait(0x7f6b76821208, 0x72, 0xb36220)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000b6e318, 0x72, 0xb36200, 0xe444f0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000b6e300, 0xc0002a8000, 0xffff, 0xffff, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000b6e300, 0xc0002a8000, 0xffff, 0xffff, 0x9bd360, 0x38, 0x40)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0004c0010, 0xc0002a8000, 0xffff, 0xffff, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
github.com/gosnmp/gosnmp.(*GoSNMP).receive(0xc000bbe000, 0xc000c600c0, 0x38, 0x40, 0x38, 0x0)
github.com/gosnmp/gosnmp.(*GoSNMP).sendOneRequest(0xc000bbe000, 0xc000a00620, 0xc0001f9501, 0x40e2b8, 0xe0, 0xa703c0)
github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc000bbe000, 0xc000a00620, 0xc00090e101, 0x0, 0x0, 0x0)
github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc000bbe000, 0xc0004937c0, 0x2, 0x2, 0x0, 0x0, 0xc00058e3c0)
main.ScrapeTarget(0xb40660, 0xc0007de000, 0xc00034205c, 0xd, 0xc000190870, 0xb34f40, 0xc00058e090, 0x0, 0x0, 0x0, ...)
/app/collector.go:133 +0x5be
main.collector.Collect(0xb40660, 0xc0007de000, 0xc00034205c, 0xd, 0xc000190870, 0xb34f40, 0xc00058e090, 0xc0006940c0)
/app/collector.go:222 +0xbc

goroutine 37 [select]:
net/http.HandlerFunc.ServeHTTP(0xc000b79650, 0xb3f0a0, 0xc000620000, 0xc000570200)
/usr/local/go/src/net/http/server.go:2042 +0x44
main.handler(0xb3f0a0, 0xc000620000, 0xc000570200, 0xb34f40, 0xc0002343c0)
/app/main.go:110 +0x6d7
main.main.func2(0xb3f0a0, 0xc000620000, 0xc000570200)
/app/main.go:201 +0x51
net/http.HandlerFunc.ServeHTTP(0xc000bc82a0, 0xb3f0a0, 0xc000620000, 0xc000570200)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc000620000, 0xc000570200)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000bda000, 0xb3f0a0, 0xc000620000, 0xc000570200)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0000d81e0, 0xb40660, 0xc000860000)
/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 211 [IO wait]:
internal/poll.runtime_pollWait(0x7f6b76821c00, 0x72, 0xb36220)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc00081e298, 0x72, 0xb36200, 0xe444f0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc00081e280, 0xc000bde000, 0

Brian Candler

unread,
Aug 31, 2021, 9:12:01 AM8/31/21
to Prometheus Users
Ah OK, so you're using SNMPv3.  However I still can't reproduce it with

if_mib3:
  <<: *if_mib
  version: 3
  auth:
    security_level: authPriv
    username: bla
    auth_protocol: SHA
    password: blabla
    priv_protocol: AES
    priv_password: blablabla

# curl 'localhost:9116/snmp?module=if_mib3&target=1.1.1.1&debug=true'

I suggest you try reproducing locally, with curl to your docker container and also curl to a local instance of snmp_exporter.

If it reproduces in the first case but not the second then it's most likely a problem with the docker packaging, and you should take it up with whoever maintains the docker container.

If you can't reproduce it with curl, even on the docker container, then it could be something about how prometheus is talking to snmp_exporter.  You might need to look at tcpdump to see how it's scraping it, to find a way to reproduce with curl.

One other thing: are you using the out-of-the-box snmp.yml, or one that you've generated yourself?

ohey...@gmail.com

unread,
Aug 31, 2021, 10:25:30 AM8/31/21
to Prometheus Users
Here is the curl command forwarded via kubectl port-forward to the snmp-exporter  kubernetes container
If I call the 1.1.1.1 which not existing, I get:

curl 'localhost:9116/snmp?module=ipn&target=1.1.1.1&debug=true'
An error has occurred while serving metrics:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error getting target 1.1.1.1: request timeout (after 3 retries)

If I call the existing ip, I get the exception
curl 'localhost:9116/snmp?module=ipn&target=10.17.125.14&debug=true' 

looks like the response breaks it and not a timeout. Would be interesting to see the real response, what's missing or breaking it. Other devices of our module ipn are working fine, btw. these are Cisco N9k devices.

I am using a configuration which is generated with a generator docker image.

An error has occurred while serving metrics:

error collecting metric Desc{fqName: "snmp_error", help: "Error scraping target", constLabels: {}, variableLabels: []}: error getting target 10.17.125.14: recover: runtime error: invalid memory address or nil pointer dereference
Stack:goroutine 507 [running]:
panic(0x9e4880, 0xe89790)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0003e6360, 0xc001982460, 0xc000a9a001, 0x0, 0xb34ee0, 0xc000b6fe40)
github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0003e6360, 0xc000c33060, 0x2, 0x2, 0x0, 0x0, 0xc00083bad0)
main.ScrapeTarget(0xb40660, 0xc000458000, 0xc0005b801c, 0xc, 0xc0001520f0, 0xb34f40, 0xc00083b7a0, 0x0, 0x0, 0x0, ...)
/app/collector.go:133 +0x5be
main.collector.Collect(0xb40660, 0xc000458000, 0xc0005b801c, 0xc, 0xc0001520f0, 0xb34f40, 0xc00083b7a0, 0xc0012fe2a0)
internal/poll.runtime_pollWait(0x7efcd8f14eb8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000d0c018, 0x72, 0x0, 0x0, 0xa85b68)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000d0c000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000d0c000, 0x3631c3285197d274, 0x0, 0x0)
/usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000b74300, 0x612e317d, 0xc000da3ad0, 0x48e3a6)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000b74300, 0xc000da3b20, 0x18, 0xc000000180, 0x7f8dec)
/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc000d0a000, 0xb3ee20, 0xc000b74300, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2937 +0x266
github.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc000b74300, 0xc000d0a000, 0x0, 0x0, 0xb34f40, 0xc000299e30, 0x0, 0xe0)
main.main()
/app/main.go:248 +0xb71

goroutine 8 [syscall, 3 minutes]:
os/signal.signal_recv(0x0)
/usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 9 [select, 3 minutes]:
main.main.func1(0xc0002801e0, 0xb34f40, 0xc000299e30)
/app/main.go:179 +0xe5
created by main.main
/app/main.go:177 +0x7ce

goroutine 13 [select]:
net/http.HandlerFunc.ServeHTTP(0xc0001062a0, 0xb3f0a0, 0xc000234e00, 0xc000898300)
/usr/local/go/src/net/http/server.go:2042 +0x44
main.handler(0xb3f0a0, 0xc000234e00, 0xc000898300, 0xb34f40, 0xc000299e30)
/app/main.go:110 +0x6d7
main.main.func2(0xb3f0a0, 0xc000234e00, 0xc000898300)
/app/main.go:201 +0x51
net/http.HandlerFunc.ServeHTTP(0xc000b742a0, 0xb3f0a0, 0xc000234e00, 0xc000898300)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc000234e00, 0xc000898300)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc000d0a000, 0xb3f0a0, 0xc000234e00, 0xc000898300)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000192000, 0xb40660, 0xc000b7e180)
/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 561 [IO wait]:
internal/poll.runtime_pollWait(0x7efcd8f145a8, 0x72, 0xb36220)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000566398, 0x72, 0xb36200, 0xe444f0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000566380, 0xc0001e80d1, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000566380, 0xc0001e80d1, 0x1, 0x1, 0x0, 0x0, 0xb3ac20)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005b2020, 0xc0001e80d1, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
net/http.(*connReader).backgroundRead(0xc0001e80c0)
/usr/local/go/src/net/http/server.go:690 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:686 +0xd5

goroutine 554 [IO wait]:
internal/poll.runtime_pollWait(0x7efcd8f14208, 0x72, 0xb36220)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000566298, 0x72, 0xb36200, 0xe444f0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000566280, 0xc000c25781, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000566280, 0xc000c25781, 0x1, 0x1, 0x0, 0x0, 0xb3ac20)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000c2e008, 0xc000c25781, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
net/http.(*connReader).backgroundRead(0xc000c25770)
/usr/local/go/src/net/http/server.go:690 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:686 +0xd5

goroutine 46 [IO wait]:
internal/poll.runtime_pollWait(0x7efcd8f14c00, 0x72, 0xb36220)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000566118, 0x72, 0xb36200, 0xe444f0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000566100, 0xc000d82000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000566100, 0xc000d82000, 0x1000, 0x1000, 0x59e3d3, 0xe8c0e0, 0xa02260)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000568008, 0xc000d82000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
net/http.(*connReader).Read(0xc000560360, 0xc000d82000, 0x1000, 0x1000, 0x59ffb4, 0x80, 0x203000)
/usr/local/go/src/net/http/server.go:798 +0x1ad
bufio.(*Reader).fill(0xc00007c2a0)
/usr/local/go/src/bufio/bufio.go:101 +0x105
bufio.(*Reader).ReadSlice(0xc00007c2a0, 0xa, 0x28, 0xc000713988, 0x40d950, 0xc000467100, 0x100)
/usr/local/go/src/bufio/bufio.go:360 +0x3d
bufio.(*Reader).ReadLine(0xc00007c2a0, 0xc000467100, 0x479294, 0xc000090c00, 0x0, 0x40e2b8, 0x30)
/usr/local/go/src/bufio/bufio.go:389 +0x34
net/textproto.(*Reader).readLineSlice(0xc0008d9500, 0xc000467100, 0x4d788d, 0xc000566100, 0x467500, 0xc000582d80)
/usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
/usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc00007c2a0, 0x0, 0xc000467100, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:1012 +0xaa
net/http.(*conn).readRequest(0xc0000e43c0, 0xb40660, 0xc00057a080, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:984 +0x19a
net/http.(*conn).serve(0xc0000e43c0, 0xb40660, 0xc00057a080)
/usr/local/go/src/net/http/server.go:1851 +0x705
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/serv


Calling the same broken device from my laptop with the darwin snmp_exporter v0.20.0, fresh download from your github release page and my config, I get:

snmp_exporter-0.20.0.darwin-amd64$ ./snmp_exporter --config.file="snmp.yml"
level=info ts=2021-08-31T14:08:04.742Z caller=main.go:152 msg="Starting snmp_exporter" version="(version=0.20.0, branch=HEAD, revision=c33572b6c8c8e43a479fde0f9fa8ac86e15598bc)"
level=info ts=2021-08-31T14:08:04.742Z caller=main.go:153 build_context="(go=go1.15.8, user=root@eebd39e6960e, date=20210212-11:35:55)"
level=info ts=2021-08-31T14:08:04.792Z caller=main.go:246 msg="Listening on address" address=:9116
level=info ts=2021-08-31T14:08:04.792Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
level=info ts=2021-08-31T14:10:45.612Z caller=collector.go:224 module=ipn target=10.17.125.14 msg="Error scraping target" err="error getting target 10.17.125.14: recover: runtime error: invalid memory address or nil pointer dereference\nStack:goroutine 45 [running]:\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc0001295c8)
panic(0x15e6140, 0x1a90af0)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0000b0120, 0xc00039e0e0, 0xc00052c701, 0x0, 0x1737940, 0xc000d16330)
github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0000b0120, 0xc000dceec0, 0x2, 0x2, 0x0, 0x0, 0xc0001caab0)
main.ScrapeTarget(0x1743200, 0xc0009180c0, 0xc000624091, 0xc, 0xc0002254a0, 0x17379a0, 0xc000699ef0, 0x0, 0x0, 0x0, ...)
/app/collector.go:133 +0x5be
main.collector.Collect(0x1743200, 0xc0009180c0, 0xc000624091, 0xc, 0xc0002254a0, 0x17379a0, 0xc000699ef0, 0xc0002004e0)
goroutine 1 [IO wait, 2 minutes]:
internal/poll.runtime_pollWait(0x3724ed0, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000dfb198, 0x72, 0x0, 0x0, 0x1686b4b)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Accept(0xc000dfb180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc000dfb180, 0xfa50add9a8b61eed, 0xb001a8b61eed, 0x100000001)
usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc000e0fac0, 0x612e3820, 0xc000cc5ad0, 0x108d3a6)
/usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc000e0fac0, 0xc000cc5b20, 0x18, 0xc000000180, 0x13fa24c)
/usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc00069a1c0, 0x17419c0, 0xc000e0fac0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2937 +0x266
github.com/prometheus/exporter-toolkit/web.Serve(0x17419c0, 0xc000e0fac0, 0xc00069a1c0, 0x0, 0x0, 0x17379a0, 0xc0002025d0, 0x0, 0xe0)
main.main()
/app/main.go:248 +0xb71

goroutine 49 [syscall, 2 minutes]:
os/signal.signal_recv(0x0)\
/usr/local/go/src/runtime/sigqueue.go:144 +0x9d
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
/usr/local/go/src/os/signal/signal.go:150 +0x45
goroutine 50 [select, 2 minutes]:
main.main.func1(0xc000200240, 0x17379a0, 0xc0002025d0)
/app/main.go:179 +0xe5
created by main.main
/app/main.go:177 +0x7ce

ngoroutine 54 [select]:
net/http.HandlerFunc.ServeHTTP(0xc00054e690, 0x1741c40, 0xc00039e000, 0xc0000e2000)
/usr/local/go/src/net/http/server.go:2042 +0x44
main.handler(0x1741c40, 0xc00039e000, 0xc0000e2000, 0x17379a0, 0xc0002025d0)
/app/main.go:110 +0x6d7
main.main.func2(0x1741c40, 0xc00039e000, 0xc0000e2000)
/app/main.go:201 +0x51
net/http.HandlerFunc.ServeHTTP(0xc000e0fa60, 0x1741c40, 0xc00039e000, 0xc0000e2000)
/usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0x1aa6da0, 0x1741c40, 0xc00039e000, 0xc0000e2000)
/usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc00069a1c0, 0x1741c40, 0xc00039e000, 0xc0000e2000)
/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000288aa0, 0x1743200, 0xc000dd3980)
usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2969 +0x36c
goroutine 43 [IO wait]:
internal/poll.runtime_pollWait(0x3724de8, 0x72, 0x1738d80)
/usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000dfb218, 0x72, 0x1738d00, 0x1a4b790, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000dfb200, 0xc000e18c41, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5
net.(*netFD).Read(0xc000dfb200, 0xc000e18c41, 0x1, 0x1, 0xc000dd3a58, 0xc000051f68, 0x107671c)
/usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc000dc78c8, 0xc000e18c41, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:182 +0x8e
net/http.(*connReader).backgroundRead(0xc000e18c30)
/usr/local/go/src/net/http/server.go:690 +0x58
created by net/http.(*connReader).startBackgroundRead
/usr/local/go/src/net/http/server.go:686 +0xd5

goroutine 46 [semacquire]:
sync.runtime_Semacquire(0xc0000207b8)
/usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0000207b0)
/usr/local/go/src/sync/waitgroup.go:130 +0x65
\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"

ohey...@gmail.com

unread,
Sep 1, 2021, 6:22:26 AM9/1/21
to Prometheus Users

TCP dump on the snmp-exporter shows that there is no response to the call itself:

1 0.000000 10.246.204.141 10.17.125.14 SNMP 106 get-request
2 0.001040 10.17.125.14 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0
3 0.001256 10.246.204.141 10.17.125.14 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
4 0.002108 10.17.125.14 10.246.204.141 SNMP 175 report 1.3.6.1.6.3.15.1.1.2.0
5 0.002253 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
6 10.002399 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
7 20.002477 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
8 30.002632 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
9 60.000442 10.246.204.141 10.17.125.14 SNMP 106 get-request
10 60.001468 10.17.125.14 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0
11 60.001603 10.246.204.141 10.17.125.14 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
12 60.002489 10.17.125.14 10.246.204.141 SNMP 175 report 1.3.6.1.6.3.15.1.1.2.0
13 60.002688 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
14 70.002817 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
15 80.002944 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
16 90.002988 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
17 179.999713 10.246.204.141 10.17.125.14 SNMP 106 get-request
18 180.000850 10.17.125.14 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0
19 180.001000 10.246.204.141 10.17.125.14 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
20 180.001864 10.17.125.14 10.246.204.141 SNMP 175 report 1.3.6.1.6.3.15.1.1.2.0
21 180.001942 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
22 190.002179 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
23 200.002143 10.246.204.141 10.17.125.14 SNMP 197 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0


Only the snmp error counter is delivered back all the time, but the no response produces the nil pointer exception right?

Regards,

Olaf

Brian Candler

unread,
Sep 1, 2021, 7:40:03 AM9/1/21
to Prometheus Users
I'm not clear on what you're saying.  Originally you said this applied to devices "which are not reachable".  Are you now saying that the device is reachable and responding to some SNMP queries but not others?  Is the problematic device 10.17.125.14 ? (previously you showed 10.17.125.48)

I have certainly seen cases where devices can return malformed SNMP responses in certain situations, e.g. truncated packets, which the SNMP manager (i.e. client) ignores; it just resends the original request until it gives up and times out.  In fact, I have a device like that on my network right now.  But snmp_exporter isn't giving a panic in that case; just msg="Error scraping target" err="error walking target x.x.x.x: request timeout (after 3 retries)"

Suggestions:

* See if you can reproduce the problem with "snmpbulkwalk" (not "snmpwalk") from net-snmp tools
* See if the problem goes away if you set a lower value for max_repetitions in snmp.yml (the default is 25); the corresponding snmpbulkwalk option is -Cr <N> and defaults to 10.

Also, that looks like a very strange version of tcpdump you're using.  Try using the standard Linux tcpdump, on either the docker host or inside the snmp_exporter container:
    tcpdump -i eth0 -nn -s0 -X host 10.17.125.14
When given the full packet to decode (-s0), tcpdump can sometimes highlight problems with the ASN formatting, e.g. truncated packets.

It would also be good to run tcpdump on the target device, or as close to it as possible, and see if you are getting IP fragmentation or other oddities.

I agree that snmp_exporter ideally shouldn't log a panic.  Fixing that would require you to capture the situation which causes it to panic for you in a way that's reproducible for a developer.  But it's starting to look like the real problem is that the device itself is giving an invalid SNMP response, in which case, snmp_exporter wouldn't be able to handle it anyway; ISTM that catching the panic is a reasonable way to handle the problem.

ohey...@gmail.com

unread,
Sep 1, 2021, 8:30:22 AM9/1/21
to Prometheus Users
Sorry,  I mixed it up with the ips. I am not aware of what is happening, Device itself is reachable, but the response from snmpwalk (Timeout: No Response from 10.17.125.48) or snmp_exporter (nil pointer exception).

I am using the tcpdump on the underlying Kubernetes nodes, which is:
tcpdump --help
tcpdump version 4.9.3
libpcap version 1.9.1 (with TPACKET_V3)
OpenSSL 1.1.1k  25 Mar 2021

I am dumping with: "tcpdump -i bond2 host 10.17.125.48 -s 65535 -w dumpnode1" and afterwards I am loading the file into Wireshark and decode it the snmp protocol and the EngineID from the dump and mysnmp v3 settings for this device:

I get exactly the same results with that ip:

4 89.998427 10.246.204.141 10.17.125.48 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
5 89.999266 10.17.125.48 10.246.204.141 SNMP 176 report 1.3.6.1.6.3.15.1.1.2.0
6 89.999342 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
7 99.999515 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
8 109.999648 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
9 119.999752 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
10 149.997236 10.246.204.141 10.17.125.48 SNMP 106 get-request
11 149.998367 10.17.125.48 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0
12 149.998500 10.246.204.141 10.17.125.48 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
13 149.999344 10.17.125.48 10.246.204.141 SNMP 176 report 1.3.6.1.6.3.15.1.1.2.0
14 149.999498 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
15 159.999630 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
16 169.999816 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
17 179.999849 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
18 329.997044 10.246.204.141 10.17.125.48 SNMP 106 get-request
19 329.998176 10.17.125.48 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0
20 329.998309 10.246.204.141 10.17.125.48 SNMP 195 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
21 329.999266 10.17.125.48 10.246.204.141 SNMP 176 report 1.3.6.1.6.3.15.1.1.2.0
22 329.999339 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
23 339.999467 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
24 349.999865 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
25 359.999957 10.246.204.141 10.17.125.48 SNMP 198 get-request 1.3.6.1.2.1.1.1.0 1.3.6.1.2.1.1.3.0
26 449.996962 10.246.204.141 10.17.125.48 SNMP 106 get-request
27 449.998087 10.17.125.48 10.246.204.141 SNMP 147 report 1.3.6.1.6.3.15.1.1.4.0


Using your tcpdump command and dumping on the node, where the request comes from:

12:27:57.634370 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=r U="" E= C="" GetRequest(14)
0x0000:  4500 005c 7940 4000 3f11 638c 0af6 cc8d  E..\y@@.?.c.....
0x0010:  0a11 7d30 a6b0 00a1 0048 5f1e 303e 0201  ..}0.....H_.0>..
0x0020:  0330 1102 043e ae50 5402 0300 ffff 0401  .0...>.PT.......
0x0030:  0402 0103 0410 300e 0400 0201 0002 0100  ......0.........
0x0040:  0400 0400 0400 3014 0400 0400 a00e 0204  ......0.........
0x0050:  3eae 5054 0201 0002 0100 3000            >.PT......0.
12:27:57.635510 IP 10.17.125.48.161 > 10.246.204.141.42672:  F= U="" E=_80_00_00_09_03_00_a3_8e_8c_3e_06 C="" Report(33)  .1.3.6.1.6.3.15.1.1.4.0=425050
0x0000:  4500 0085 d547 0000 3811 4e5c 0a11 7d30  E....G..8.N\..}0
0x0010:  0af6 cc8d 00a1 a6b0 0071 1fd9 3067 0201  .........q..0g..
0x0020:  0330 1102 043e ae50 5402 0300 ffe3 0401  .0...>.PT.......
0x0030:  0002 0103 041b 3019 040b 8000 0009 0300  ......0.........
0x0040:  a38e 8c3e 0602 0100 0201 0004 0004 0004  ...>............
0x0050:  0030 3204 0b80 0000 0903 00a3 8e8c 3e06  .02...........>.
0x0060:  0400 a821 0204 3eae 5054 0201 0002 0100  ...!..>.PT......
0x0070:  3013 3011 060a 2b06 0106 030f 0101 0400  0.0...+.........
0x0080:  4103 067c 5a                             A..|Z
12:27:57.635651 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=apr U="cc-snmpcollector" [!scoped PDU]7f_90_40_01_da_68_03_d3_41_8c_8c_0f_6d_33_73_6d_73_4d_bf_67_b8_bf_0b_2c_ad_83_4f_b0_05_03_0a_fb_cc_2c_cb_b9_33_c0_2f_55_ac_41_b1_a5_ee_45_d6_11_9b_6f_ea_ac_d0_0b_85_5f_ac_fd_9c_52_07
0x0000:  4500 00b5 7941 4000 3f11 6332 0af6 cc8d  E...yA@.?.c2....
0x0010:  0a11 7d30 a6b0 00a1 00a1 5f77 3081 9602  ..}0......_w0...
0x0020:  0103 3011 0204 3eae 5055 0203 00ff ff04  ..0...>.PU......
0x0030:  0107 0201 0304 3f30 3d04 0b80 0000 0903  ......?0=.......
0x0040:  00a3 8e8c 3e06 0201 0002 0100 0410 6363  ....>.........cc
0x0050:  2d73 6e6d 7063 6f6c 6c65 6374 6f72 040c  -snmpcollector..
0x0060:  5607 e59f adf6 957c c840 1b52 0408 2838  V......|.@.R..(8
0x0070:  8d8a 7de2 066b 043d 7f90 4001 da68 03d3  ..}..k.=..@..h..
0x0080:  418c 8c0f 6d33 736d 734d bf67 b8bf 0b2c  A...m3smsM.g...,
0x0090:  ad83 4fb0 0503 0afb cc2c cbb9 33c0 2f55  ..O......,..3./U
0x00a0:  ac41 b1a5 ee45 d611 9b6f eaac d00b 855f  .A...E...o....._
0x00b0:  acfd 9c52 07                             ...R.
12:27:57.636499 IP 10.17.125.48.161 > 10.246.204.141.42672:  F=a U="cc-snmpcollector" E=_80_00_00_09_03_00_a3_8e_8c_3e_06 C="" Report(30)  .1.3.6.1.6.3.15.1.1.2.0=424432
0x0000:  4500 00a2 d548 0000 3811 4e3e 0a11 7d30  E....H..8.N>..}0
0x0010:  0af6 cc8d 00a1 a6b0 008e 3f02 3081 8302  ..........?.0...
0x0020:  0103 3011 0204 3eae 5055 0203 00ff e304  ..0...>.PU......
0x0030:  0101 0201 0304 3a30 3804 0b80 0000 0903  ......:08.......
0x0040:  00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04  ....>.........n.
0x0050:  1063 632d 736e 6d70 636f 6c6c 6563 746f  .cc-snmpcollecto
0x0060:  7204 0c6b 68a6 3d05 efde e33c 0dd5 7904  r..kh.=....<..y.
0x0070:  0030 2f04 0b80 0000 0903 00a3 8e8c 3e06  .0/...........>.
0x0080:  0400 a81e 0201 0002 0100 0201 0030 1330  .............0.0
0x0090:  1106 0a2b 0601 0603 0f01 0102 0041 0306  ...+.........A..
0x00a0:  79f0                                     y.
12:27:57.636578 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=apr U="cc-snmpcollector" [!scoped PDU]63_dc_d8_88_bc_35_f9_0c_bf_80_e0_80_ce_ef_03_3f_5c_d7_18_93_46_e4_6c_06_49_09_ba_45_73_ca_1e_e2_cc_c8_96_bf_51_7b_3e_7e_65_86_f7_8c_b1_d9_91_fd_4c_72_70_c8_c0_c1_4f_bb_02_13_8e_30_bd
0x0000:  4500 00b8 7942 4000 3f11 632e 0af6 cc8d  E...yB@.?.c.....
0x0010:  0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902  ..}0......_z0...
0x0020:  0103 3011 0204 3eae 5056 0203 00ff ff04  ..0...>.PV......
0x0030:  0107 0201 0304 4230 4004 0b80 0000 0903  ......B0@.......
0x0040:  00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04  ....>.........n.
0x0050:  1063 632d 736e 6d70 636f 6c6c 6563 746f  .cc-snmpcollecto
0x0060:  7204 0ce9 99ca f365 84a6 df1e aa13 9e04  r......e........
0x0070:  0828 388d 8a7d e206 6c04 3d63 dcd8 88bc  .(8..}..l.=c....
0x0080:  35f9 0cbf 80e0 80ce ef03 3f5c d718 9346  5.........?\...F
0x0090:  e46c 0649 09ba 4573 ca1e e2cc c896 bf51  .l.I..Es.......Q
0x00a0:  7b3e 7e65 86f7 8cb1 d991 fd4c 7270 c8c0  {>~e.......Lrp..
0x00b0:  c14f bb02 138e 30bd                      .O....0.
12:28:07.636786 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=apr U="cc-snmpcollector" [!scoped PDU]6e_47_0c_78_2a_e7_10_19_fb_6e_d8_3d_5e_1b_0d_19_89_56_d7_24_46_69_05_31_e1_4c_b8_95_1e_6f_29_08_89_e6_cc_d5_81_42_9a_ce_c8_01_1d_03_65_af_64_de_1d_c1_b6_2e_7b_67_7b_bd_9a_8a_f3_2d_bf
0x0000:  4500 00b8 8e8f 4000 3f11 4de1 0af6 cc8d  E.....@.?.M.....
0x0010:  0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902  ..}0......_z0...
0x0020:  0103 3011 0204 3eae 5057 0203 00ff ff04  ..0...>.PW......
0x0030:  0107 0201 0304 4230 4004 0b80 0000 0903  ......B0@.......
0x0040:  00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04  ....>.........n.
0x0050:  1063 632d 736e 6d70 636f 6c6c 6563 746f  .cc-snmpcollecto
0x0060:  7204 0cd9 42c1 c699 f51d a0db 4d2f bb04  r...B.......M/..
0x0070:  0828 388d 8a7d e206 6d04 3d6e 470c 782a  .(8..}..m.=nG.x*
0x0080:  e710 19fb 6ed8 3d5e 1b0d 1989 56d7 2446  ....n.=^....V.$F
0x0090:  6905 31e1 4cb8 951e 6f29 0889 e6cc d581  i.1.L...o)......
0x00a0:  429a cec8 011d 0365 af64 de1d c1b6 2e7b  B......e.d.....{
0x00b0:  677b bd9a 8af3 2dbf                      g{....-.
12:28:17.636916 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=apr U="cc-snmpcollector" [!scoped PDU]6b_2e_28_d6_6e_16_14_fe_19_2f_7c_8a_50_df_55_6e_c6_ed_75_45_b3_06_32_b7_9f_61_7c_53_52_d0_db_c4_61_3c_f0_42_24_af_63_9b_99_21_40_b3_ac_4c_c2_72_c6_a6_46_6a_84_7c_d2_67_77_70_bc_b4_1a
0x0000:  4500 00b8 a855 4000 3f11 341b 0af6 cc8d  E....U@.?.4.....
0x0010:  0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902  ..}0......_z0...
0x0020:  0103 3011 0204 3eae 5058 0203 00ff ff04  ..0...>.PX......
0x0030:  0107 0201 0304 4230 4004 0b80 0000 0903  ......B0@.......
0x0040:  00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04  ....>.........n.
0x0050:  1063 632d 736e 6d70 636f 6c6c 6563 746f  .cc-snmpcollecto
0x0060:  7204 0cad 6828 91ec bdb9 f9d2 3dae 1704  r...h(......=...
0x0070:  0828 388d 8a7d e206 6e04 3d6b 2e28 d66e  .(8..}..n.=k.(.n
0x0080:  1614 fe19 2f7c 8a50 df55 6ec6 ed75 45b3  ..../|.P.Un..uE.
0x0090:  0632 b79f 617c 5352 d0db c461 3cf0 4224  .2..a|SR...a<.B$
0x00a0:  af63 9b99 2140 b3ac 4cc2 72c6 a646 6a84  .c..!@..L.r..Fj.
0x00b0:  7cd2 6777 70bc b41a                      |.gwp...
12:28:27.637047 IP 10.246.204.141.42672 > 10.17.125.48.161:  F=apr U="cc-snmpcollector" [!scoped PDU]0b_01_37_24_7b_7a_5c_5e_03_5c_e4_ff_79_c5_6f_41_e1_72_06_e8_a0_c0_e9_d5_0c_5b_5a_7e_1c_f7_63_92_f8_3f_92_94_8f_48_0f_6c_69_8c_b8_60_4a_a5_7b_72_ad_9d_32_2e_18_5d_07_42_3e_ec_bd_05_0e
0x0000:  4500 00b8 ac5d 4000 3f11 3013 0af6 cc8d  E....]@.?.0.....
0x0010:  0a11 7d30 a6b0 00a1 00a4 5f7a 3081 9902  ..}0......_z0...
0x0020:  0103 3011 0204 3eae 5059 0203 00ff ff04  ..0...>.PY......
0x0030:  0107 0201 0304 4230 4004 0b80 0000 0903  ......B0@.......
0x0040:  00a3 8e8c 3e06 0201 0b02 0401 7f09 6e04  ....>.........n.
0x0050:  1063 632d 736e 6d70 636f 6c6c 6563 746f  .cc-snmpcollecto
0x0060:  7204 0cbf 0ae9 dc09 fbf7 aedc 2c76 2b04  r...........,v+.
0x0070:  0828 388d 8a7d e206 6f04 3d0b 0137 247b  .(8..}..o.=..7${
0x0080:  7a5c 5e03 5ce4 ff79 c56f 41e1 7206 e8a0  z\^.\..y.oA.r...
0x0090:  c0e9 d50c 5b5a 7e1c f763 92f8 3f92 948f  ....[Z~..c..?...
0x00a0:  480f 6c69 8cb8 604a a57b 72ad 9d32 2e18  H.li..`J.{r..2..
0x00b0:  5d07 423e ecbd 050e                      ].B>....


which is the whole communication between the device and the node during the call:

level=debug ts=2021-09-01T12:27:57.634Z caller=main.go:102 module=ipn target=10.17.125.48 msg="Starting scrape"
level=debug ts=2021-09-01T12:27:57.634Z caller=collector.go:131 module=ipn target=10.17.125.48 msg="Getting OIDs" oids=2
level=info ts=2021-09-01T12:28:37.638Z caller=collector.go:224 module=ipn target=10.17.125.48 msg="Error scraping target" err="error getting target 10.17.125.48: recover: runtime error: invalid memory address or nil pointer dereference\nStack:goroutine 305377 [running]:\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000fc35c8)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:326 +0xa5\npanic(0x9e4880, 0xe89790)\n\t/usr/local/go/src/runtime/panic.go:969 +0x1b9\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send(0xc00014a240, 0xc0006f2460, 0xc000c21c01, 0x0, 0xb34ee0, 0xc00114a370)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:372 +0x20e\ngithub.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc00014a240, 0xc000bd1c00, 0x2, 0x2, 0x0, 0x0, 0xc000227b60)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/gosnmp.go:363 +0x159\nmain.ScrapeTarget(0xb40660, 0xc00042c280, 0xc0005c86dc, 0xc, 0xc000189a40, 0xb34f40, 0xc000227890, 0x0, 0x0, 0x0, ...)\n\t/app/collector.go:133 +0x5be\nmain.collector.Collect(0xb40660, 0xc00042c280, 0xc0005c86dc, 0xc, 0xc000189a40, 0xb34f40, 0xc000227890, 0xc0015d69c0)\n\t/app/collector.go:222 +0xbc\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:446 +0x1a2\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:457 +0x5ce\n\ngoroutine 1 [IO wait, 14 minutes]:\ninternal/poll.runtime_pollWait(0x7f92c58a8eb8, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000c94018, 0x72, 0x0, 0x0, 0xa85b68)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc000c94000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc\nnet.(*netFD).accept(0xc000c94000, 0x2deef00cbedce7d5, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_unix.go:172 +0x45\nnet.(*TCPListener).accept(0xc0007fc300, 0x612f6e8c, 0xc00060fad0, 0x48e3a6)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 +0x32\nnet.(*TCPListener).Accept(0xc0007fc300, 0xc00060fb20, 0x18, 0xc000000180, 0x7f8dec)\n\t/usr/local/go/src/net/tcpsock.go:261 +0x65\nnet/http.(*Server).Serve(0xc000c92000, 0xb3ee20, 0xc0007fc300, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2937 +0x266\ngithub.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc0007fc300, 0xc000c92000, 0x0, 0x0, 0xb34f40, 0xc0000b0c00, 0x0, 0xe0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:192 +0x1b0\ngithub.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc000c92000, 0x0, 0x0, 0xb34f40, 0xc0000b0c00, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:184 +0xfd\nmain.main()\n\t/app/main.go:248 +0xb71\n\ngoroutine 30 [syscall, 1369 minutes]:\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:147 +0x9d\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x25\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:150 +0x45\n\ngoroutine 31 [select, 1369 minutes]:\nmain.main.func1(0xc00007c300, 0xb34f40, 0xc0000b0c00)\n\t/app/main.go:179 +0xe5\ncreated by main.main\n\t/app/main.go:177 +0x7ce\n\ngoroutine 153 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000c210e0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc0001f6a10, 0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nmain.handler(0xb3f0a0, 0xc000c920e0, 0xc0002ee100, 0xb34f40, 0xc0000b0c00)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/app/main.go:201 +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc0007fc2a0, 0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2417 +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000c92000, 0xb3f0a0, 0xc000c920e0, 0xc0002ee100)\n\t/usr/local/go/src/net/http/server.go:2843 +0xa3\nnet/http.(*conn).serve(0xc0000df900, 0xb40660, 0xc00028e540)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 38 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc0005703c0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc0001c61c0, 0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nmain.handler(0xb3f0a0, 0xc0006f2ee0, 0xc009152000, 0xb34f40, 0xc0000b0c00)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/app/main.go:201 +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc0007fc2a0, 0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2417 +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc000c92000, 0xb3f0a0, 0xc0006f2ee0, 0xc009152000)\n\t/usr/local/go/src/net/http/server.go:2843 +0xa3\nnet/http.(*conn).serve(0xc00011e3c0, 0xb40660, 0xc0002ea080)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 305373 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f92c58a82f0, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000c94518, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c94500, 0xc000a8f0f1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5\nnet.(*netFD).Read(0xc000c94500, 0xc000a8f0f1, 0x1, 0x1, 0xc0009b1c18, 0xc000401768, 0x4774bc)\n\t/usr/local/go/src/net/fd_posix.go:55 +0x4f\nnet.(*conn).Read(0xc00024c0e0, 0xc000a8f0f1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:182 +0x8e\nnet/http.(*connReader).backgroundRead(0xc000a8f0e0)\n\t/usr/local/go/src/net/http/server.go:690 +0x58\ncreated by net/http.(*connReader).startBackgroundRead\n\t/usr/local/go/src/net/http/server.go:686 +0xd5\n\ngoroutine 305437 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f92c5595c00, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000c05618, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c05600, 0xc000bc2521, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5\nnet.(*netFD).Read(0xc000c05600, 0xc000bc2521, 0x1, 0x1, 0xea4f80, 0xea4f90, 0xc000405778)\n\t/usr/local/go/src/net/fd_posix.go:55 +0x4f\nnet.(*conn).Read(0xc000288060, 0xc000bc2521, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:182 +0x8e\nnet/http.(*connReader).backgroundRead(0xc000bc2510)\n\t/usr/local/go/src/net/http/server.go:690 +0x58\ncreated by net/http.(*connReader).startBackgroundRead\n\t/usr/local/go/src/net/http/server.go:686 +0xd5\n\ngoroutine 57 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000f40690, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc000c93340, 0xc0067af700)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc000829260, 0xb3f0a0, 0xc000c93340, 0xc0067af700)\n\t/usr/local/go/src/net/http/server.go:2042 +0x4"
level=debug ts=2021-09-01T12:28:37.638Z caller=main.go:113 module=ipn target=10.17.125.48 msg="Finished scrape" duration_seconds=40.004547915

Ben Kochie

unread,
Sep 1, 2021, 8:41:19 AM9/1/21
to ohey...@gmail.com, Prometheus Users
Can you try with the latest code? There may have been some fixes to the Go SNMP library that are not released yet.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/04d59fc6-bd55-4a2c-b824-bc766dbea6a2n%40googlegroups.com.

ohey...@gmail.com

unread,
Sep 1, 2021, 9:43:24 AM9/1/21
to Prometheus Users
yes - you are right, building from main solved the exception:

level=info ts=2021-09-01T13:34:46.251Z caller=main.go:148 msg="Starting snmp_exporter" version="(version=, branch=, revision=)"
level=info ts=2021-09-01T13:34:46.251Z caller=main.go:149 build_context="(go=go1.16.6, user=, date=)"
level=info ts=2021-09-01T13:34:46.304Z caller=main.go:244 msg="Listening on address" address=:9116
level=info ts=2021-09-01T13:34:46.304Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
level=debug ts=2021-09-01T13:39:17.422Z caller=main.go:98 module=ipn target=10.17.125.48 msg="Starting scrape"
level=debug ts=2021-09-01T13:39:17.422Z caller=collector.go:176 module=ipn target=10.17.125.48 msg="Getting OIDs" oids=2
level=info ts=2021-09-01T13:39:57.494Z caller=collector.go:273 module=ipn target=10.17.125.48 msg="Error scraping target" err="error getting target 10.17.125.48: not in time window"
level=debug ts=2021-09-01T13:39:57.494Z caller=main.go:109 module=ipn target=10.17.125.48 msg="Finished scrape" duration_seconds=40.071955962

Thanks a lot and I will wait for the next release to roll it out.

Regards

Olaf

Brian Candler

unread,
Sep 1, 2021, 10:07:16 AM9/1/21
to Prometheus Users
OK great.  Interesting tcpdump output; I've no idea about the "[!scoped PDU]", or why it doesn't show as "GetBulk" or similar.
Reply all
Reply to author
Forward
0 new messages