prometheus snmp-exporter stack trace: invalid memory address

[ohey...@gmail.com]

Hi,

we are running on 0.20.0 and scraping devices, which are not configured correctly (probably acl issue on switch side), produce these errors all the time. This device for example is a Cisco N7k.

```

level=info ts=2022-03-18T08:29:09.347Z caller=collector.go:224 module=pxdlrouternxos target=10.14.25.10 msg="Error scraping target" err="error getting target 10.14.25.10: recover: runtime error: invalid memory address or nil pointer dereference\nStack:goroutine 358625 [running]:\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000e155c8)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:326 +0xa5\npanic(0x9e4880, 0xe89790)\n\t/usr/local/go/src/runtime/panic.go:969 +0x1b9\ngithub.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0004f6c60, 0xc0004c42a0, 0xc0000de601, 0x0, 0xb34ee0, 0xc000b609c0)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:372 +0x20e\ngithub.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0004f6c60, 0xc00012e300, 0x6, 0x6, 0x0, 0x0, 0xc000522b10)\n\t/go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/gosnmp.go:363 +0x159\nmain.ScrapeTarget(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, 0xc000161770, 0xb34f40, 0xc000522840, 0x0, 0x0, 0x0, ...)\n\t/app/collector.go:133 +0x5be\nmain.collector.Collect(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, 0xc000161770, 0xb34f40, 0xc000522840, 0xc000281020)\n\t/app/collector.go:222 +0xbc\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:446 +0x1a2\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:457 +0x5ce\n\ngoroutine 1 [IO wait, 216 minutes]:\ninternal/poll.runtime_pollWait(0x7f0f18245708, 0x72, 0x0)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc0004c6018, 0x72, 0x0, 0x0, 0xa85b68)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc0004c6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc\nnet.(*netFD).accept(0xc0004c6000, 0x1eadb908595c5b95, 0x0, 0x0)\n\t/usr/local/go/src/net/fd_unix.go:172 +0x45\nnet.(*TCPListener).accept(0xc0004a8300, 0x6234102b, 0xc00052dad0, 0x48e3a6)\n\t/usr/local/go/src/net/tcpsock_posix.go:139 +0x32\nnet.(*TCPListener).Accept(0xc0004a8300, 0xc00052db20, 0x18, 0xc000000180, 0x7f8dec)\n\t/usr/local/go/src/net/tcpsock.go:261 +0x65\nnet/http.(*Server).Serve(0xc0004c4000, 0xb3ee20, 0xc0004a8300, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:2937 +0x266\ngithub.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc0004a8300, 0xc0004c4000, 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0xe0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:192 +0x1b0\ngithub.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc0004c4000, 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:184 +0xfd\nmain.main()\n\t/app/main.go:248 +0xb71\n\ngoroutine 45 [syscall, 3871 minutes]:\nos/signal.signal_recv(0x0)\n\t/usr/local/go/src/runtime/sigqueue.go:147 +0x9d\nos/signal.loop()\n\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x25\ncreated by os/signal.Notify.func1.1\n\t/usr/local/go/src/os/signal/signal.go:150 +0x45\n\ngoroutine 46 [select, 3871 minutes]:\nmain.main.func1(0xc00012e360, 0xb34f40, 0xc000286420)\n\t/app/main.go:179 +0xe5\ncreated by main.main\n\t/app/main.go:177 +0x7ce\n\ngoroutine 50 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f0f18245620, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc0004c6098, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5\nnet.(*netFD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, 0x92a41b, 0xc00052f7f8, 0x7f2b36)\n\t/usr/local/go/src/net/fd_posix.go:55 +0x4f\nnet.(*conn).Read(0xc0004ae050, 0xc0004d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:182 +0x8e\nnet/http.(*connReader).Read(0xc0004ac750, 0xc0004d8000, 0x1000, 0x1000, 0x10012864e, 0x0, 0x7f0f18281db8)\n\t/usr/local/go/src/net/http/server.go:798 +0x1ad\nbufio.(*Reader).fill(0xc0002806c0)\n\t/usr/local/go/src/bufio/bufio.go:101 +0x105\nbufio.(*Reader).ReadSlice(0xc0002806c0, 0xc00025460a, 0x7f0f18281db8, 0xc00052f988, 0x40d950, 0xc00022a100, 0x100)\n\t/usr/local/go/src/bufio/bufio.go:360 +0x3d\nbufio.(*Reader).ReadLine(0xc0002806c0, 0xc00022a100, 0x479294, 0xea09c0, 0x0, 0xa5b4c0, 0xc000730a80)\n\t/usr/local/go/src/bufio/bufio.go:389 +0x34\nnet/textproto.(*Reader).readLineSlice(0xc000730a80, 0xc00022a100, 0x4d788d, 0xc0004c6080, 0x467500, 0xc001bc3e00)\n\t/usr/local/go/src/net/textproto/reader.go:58 +0x6c\nnet/textproto.(*Reader).ReadLine(...)\n\t/usr/local/go/src/net/textproto/reader.go:39\nnet/http.readRequest(0xc0002806c0, 0x0, 0xc00022a100, 0x0, 0x0)\n\t/usr/local/go/src/net/http/request.go:1012 +0xaa\nnet/http.(*conn).readRequest(0xc00024c0a0, 0xb40660, 0xc0004b4180, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/http/server.go:984 +0x19a\nnet/http.(*conn).serve(0xc00024c0a0, 0xb40660, 0xc0004b4180)\n\t/usr/local/go/src/net/http/server.go:1851 +0x705\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 358626 [semacquire]:\nsync.runtime_Semacquire(0xc000880838)\n\t/usr/local/go/src/runtime/sema.go:56 +0x45\nsync.(*WaitGroup).Wait(0xc000880830)\n\t/usr/local/go/src/sync/waitgroup.go:130 +0x65\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc000880830, 0xc000281020, 0xc000281080)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:463 +0x2b\ncreated by github.com/prometheus/client_golang/prometheus.(*Registry).Gather\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:462 +0x60d\n\ngoroutine 201493 [select]:\ngithub.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00041a2d0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc\ngithub.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99\nnet/http.HandlerFunc.ServeHTTP(0xc0004edc70, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nmain.handler(0xb3f0a0, 0xc0004c41c0, 0xc00022a000, 0xb34f40, 0xc000286420)\n\t/app/main.go:110 +0x6d7\nmain.main.func2(0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/app/main.go:201 +0x51\nnet/http.HandlerFunc.ServeHTTP(0xc0004a82a0, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/usr/local/go/src/net/http/server.go:2042 +0x44\nnet/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/usr/local/go/src/net/http/server.go:2417 +0x1ad\nnet/http.serverHandler.ServeHTTP(0xc0004c4000, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)\n\t/usr/local/go/src/net/http/server.go:2843 +0xa3\nnet/http.(*conn).serve(0xc00024c640, 0xb40660, 0xc0000ca100)\n\t/usr/local/go/src/net/http/server.go:1925 +0x8ad\ncreated by net/http.(*Server).Serve\n\t/usr/local/go/src/net/http/server.go:2969 +0x36c\n\ngoroutine 358543 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f0f18245368, 0x72, 0xb36220)\n\t/usr/local/go/src/runtime/netpoll.go:222 +0x55\ninternal/poll.(*pollDesc).wait(0xc000290298, 0x72, 0xb36200, 0xe444f0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/local/go/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5\nnet.(*netFD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0xc0000ca1d8, 0xc000856f68, 0x4774bc)\n\t/usr/local/go/src/net/fd_posix.go:55 +0x4f\nnet.(*conn).Read(0xc00000e040, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/local/go/src/net/net.go:182 +0x8e\nnet/http.(*connReader).backgroundRead(0xc000ecf590)\n\t/usr/local/go/src/net/http/server.go:690 +0x58\ncreated by net/http.(*connReader).startBackgroundRead\n\t/usr/local/go/src/net/http/server.go:686 +0xd5\n\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"

```

Any idea, why this errors out? Other devices, which cannot be scraped are not producing these stack traces.

Thanks a lot and regards,

Olaf

[Brian Candler]

Can you share the content of your "pxdlrouternxos" module?

Does "all the time" mean on every scrape?

Meanwhile: here is your stack trace reformatted.

error getting target 10.14.25.10: recover: runtime error: invalid memory address or nil pointer dereference

Stack:goroutine 358625 [running]:
github.com/gosnmp/gosnmp.(*GoSNMP).send.func1(0xc000e155c8)
        /go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:326 +0xa5
panic(0x9e4880, 0xe89790)
        /usr/local/go/src/runtime/panic.go:969 +0x1b9

github.com/gosnmp/gosnmp.(*GoSNMP).send(0xc0004f6c60, 0xc0004c42a0, 0xc0000de601, 0x0, 0xb34ee0, 0xc000b609c0)

        /go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/marshal.go:372 +0x20e

github.com/gosnmp/gosnmp.(*GoSNMP).Get(0xc0004f6c60, 0xc00012e300, 0x6, 0x6, 0x0, 0x0, 0xc000522b10)

        /go/pkg/mod/github.com/gosnmp/gos...@v1.29.0/gosnmp.go:363 +0x159

main.ScrapeTarget(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, 0xc000161770, 0xb34f40, 0xc000522840, 0x0, 0x0, 0x0, ...)

        /app/collector.go:133 +0x5be

main.collector.Collect(0xb40660, 0xc0004b41c0, 0xc00045ca67, 0xe, 0xc000161770, 0xb34f40, 0xc000522840, 0xc000281020)

        /app/collector.go:222 +0xbc
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:446 +0x1a2
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:457 +0x5ce

goroutine 1 [IO wait, 216 minutes]:

internal/poll.runtime_pollWait(0x7f0f18245708, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0004c6018, 0x72, 0x0, 0x0, 0xa85b68)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92

internal/poll.(*FD).Accept(0xc0004c6000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)

        /usr/local/go/src/internal/poll/fd_unix.go:394 +0x1fc
net.(*netFD).accept(0xc0004c6000, 0x1eadb908595c5b95, 0x0, 0x0)
        /usr/local/go/src/net/fd_unix.go:172 +0x45
net.(*TCPListener).accept(0xc0004a8300, 0x6234102b, 0xc00052dad0, 0x48e3a6)
        /usr/local/go/src/net/tcpsock_posix.go:139 +0x32
net.(*TCPListener).Accept(0xc0004a8300, 0xc00052db20, 0x18, 0xc000000180, 0x7f8dec)
        /usr/local/go/src/net/tcpsock.go:261 +0x65
net/http.(*Server).Serve(0xc0004c4000, 0xb3ee20, 0xc0004a8300, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2937 +0x266

github.com/prometheus/exporter-toolkit/web.Serve(0xb3ee20, 0xc0004a8300, 0xc0004c4000, 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0xe0)

        /go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:192 +0x1b0

github.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc0004c4000, 0x0, 0x0, 0xb34f40, 0xc000286420, 0x0, 0x0)

        /go/pkg/mod/github.com/prometheus/exporter...@v0.5.1/web/tls_config.go:184 +0xfd
main.main()
        /app/main.go:248 +0xb71

goroutine 45 [syscall, 3871 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:147 +0x9d
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x25
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:150 +0x45

goroutine 46 [select, 3871 minutes]:
main.main.func1(0xc00012e360, 0xb34f40, 0xc000286420)
        /app/main.go:179 +0xe5
created by main.main
        /app/main.go:177 +0x7ce

goroutine 50 [IO wait]:
internal/poll.runtime_pollWait(0x7f0f18245620, 0x72, 0xb36220)
        /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc0004c6098, 0x72, 0xb36200, 0xe444f0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92

internal/poll.(*FD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

        /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5

net.(*netFD).Read(0xc0004c6080, 0xc0004d8000, 0x1000, 0x1000, 0x92a41b, 0xc00052f7f8, 0x7f2b36)

        /usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0004ae050, 0xc0004d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:182 +0x8e

net/http.(*connReader).Read(0xc0004ac750, 0xc0004d8000, 0x1000, 0x1000, 0x10012864e, 0x0, 0x7f0f18281db8)

        /usr/local/go/src/net/http/server.go:798 +0x1ad
bufio.(*Reader).fill(0xc0002806c0)
        /usr/local/go/src/bufio/bufio.go:101 +0x105

bufio.(*Reader).ReadSlice(0xc0002806c0, 0xc00025460a, 0x7f0f18281db8, 0xc00052f988, 0x40d950, 0xc00022a100, 0x100)

        /usr/local/go/src/bufio/bufio.go:360 +0x3d

bufio.(*Reader).ReadLine(0xc0002806c0, 0xc00022a100, 0x479294, 0xea09c0, 0x0, 0xa5b4c0, 0xc000730a80)

        /usr/local/go/src/bufio/bufio.go:389 +0x34

net/textproto.(*Reader).readLineSlice(0xc000730a80, 0xc00022a100, 0x4d788d, 0xc0004c6080, 0x467500, 0xc001bc3e00)

        /usr/local/go/src/net/textproto/reader.go:58 +0x6c
net/textproto.(*Reader).ReadLine(...)
        /usr/local/go/src/net/textproto/reader.go:39
net/http.readRequest(0xc0002806c0, 0x0, 0xc00022a100, 0x0, 0x0)
        /usr/local/go/src/net/http/request.go:1012 +0xaa

net/http.(*conn).readRequest(0xc00024c0a0, 0xb40660, 0xc0004b4180, 0x0, 0x0, 0x0)

        /usr/local/go/src/net/http/server.go:984 +0x19a
net/http.(*conn).serve(0xc00024c0a0, 0xb40660, 0xc0004b4180)
        /usr/local/go/src/net/http/server.go:1851 +0x705
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 358626 [semacquire]:
sync.runtime_Semacquire(0xc000880838)
        /usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc000880830)
        /usr/local/go/src/sync/waitgroup.go:130 +0x65
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc000880830, 0xc000281020, 0xc000281080)
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:463 +0x2b
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:462 +0x60d

goroutine 201493 [select]:
github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00041a2d0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/registry.go:513 +0xbbc
github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /go/pkg/mod/github.com/prometheus/client...@v1.9.0/prometheus/promhttp/http.go:126 +0x99
net/http.HandlerFunc.ServeHTTP(0xc0004edc70, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /usr/local/go/src/net/http/server.go:2042 +0x44
main.handler(0xb3f0a0, 0xc0004c41c0, 0xc00022a000, 0xb34f40, 0xc000286420)
        /app/main.go:110 +0x6d7
main.main.func2(0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /app/main.go:201 +0x51
net/http.HandlerFunc.ServeHTTP(0xc0004a82a0, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /usr/local/go/src/net/http/server.go:2042 +0x44
net/http.(*ServeMux).ServeHTTP(0xea0040, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /usr/local/go/src/net/http/server.go:2417 +0x1ad
net/http.serverHandler.ServeHTTP(0xc0004c4000, 0xb3f0a0, 0xc0004c41c0, 0xc00022a000)
        /usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc00024c640, 0xb40660, 0xc0000ca100)
        /usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2969 +0x36c

goroutine 358543 [IO wait]:
internal/poll.runtime_pollWait(0x7f0f18245368, 0x72, 0xb36220)
        /usr/local/go/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000290298, 0x72, 0xb36200, 0xe444f0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92

internal/poll.(*FD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, 0x0)

        /usr/local/go/src/internal/poll/fd_unix.go:159 +0x1a5

net.(*netFD).Read(0xc000290280, 0xc000ecf5a1, 0x1, 0x1, 0xc0000ca1d8, 0xc000856f68, 0x4774bc)

        /usr/local/go/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc00000e040, 0xc000ecf5a1, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:182 +0x8e
net/http.(*connReader).backgroundRead(0xc000ecf590)
        /usr/local/go/src/net/http/server.go:690 +0x58
created by net/http.(*connReader).startBackgroundRead
        /usr/local/go/src/net/http/server.go:686 +0xd5

[ohey...@gmail.com]

Hi Brian,

thanks a lot for the formatting. The config is here: pxdlrouternxos config . ~1750 lines. The smp v3 settings are added during helm run via a configmap.

Regards

Olaf

[ohey...@gmail.com]

and here is the generator config: pxdlrouternxos generator config

[Brian Candler]

Does "all the time" mean on every scrape, or just intermittently?

I've just tested this on a different type of device, with snmpv2 and snmp_exporter 0.20.0. I don't get any errors, although I do get non-standard metric names, e.g. snmp_pxdlrouternxos_ifHCInOctets (if it were me, I'd just call it ifHCInOctets)

But otherwise, without a reproducing test case, I'm afraid I can't help you.

What I could suggest is that you do a binary chop. Divide the "walk" section into two; test with the first half and then with the second half. If one half causes a crash and the other doesn't, then repeat the chop on the bad half, until you can identify which MIB is causing the issue.

[Ben Kochie]

I wonder if this is an instance of https://github.com/gosnmp/gosnmp/pull/354.

This is fixed in the Go SNMP library, but we haven't cut a new snmp_exporter build in a while, so it's not in the release.

If you could try a fresh build from main to see if it's still a bug.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7956644c-a2ce-4773-b9e4-b7a197bdcfdan%40googlegroups.com.

[ohey...@gmail.com]

This snmp_exporter binary was built against master and with go 18.1 and gosnmp v1.34.0.

Looks very promising. It's running for 3h now and no trace at all. I also like the new error messages, which make much more sense, instead of the old timeout message, because everything was a timeout in the old exporter.

   ts=2022-03-22T12:26:40.683Z caller=collector.go:274 level=info module=pxgeneric target=10.11.25.10 msg="Error scraping target" err="error getting target 10.11.25.10: unknown username"
  ts=2022-03-22T12:33:09.370Z caller=collector.go:274 level=info module=pxdlrouternxos target=10.11.25.14 msg="Error scraping target" err="error getting target 10.11.25.14: not in time window"

  Thanks a lot and regards,

Olaf

[ohey...@gmail.com]

no stack traces anymore. It's running now for a week.

again, thanks a lot for the hint.

Regards

Olaf

[Ben Kochie]

Great, I'll see about cutting a release soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f001b5c-a319-4ccb-a390-57e94bf09233n%40googlegroups.com.