snmpd crash after "response to proxy request illegal. We're screwed"
Tomasz Chmielewski
Is there anything I can do to debug it further?
There was a similar case back in 2005, with no fix though:
This is what happens if I run snmpd in the foreground and I hit this bug:
*** glibc detected *** /usr/sbin/snmpd: double free or corruption (!prev): 0x0000000000899ad0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x39d227230f]
/lib64/libc.so.6(cfree+0x4b)[0x39d227276b]
/usr/lib64/libnetsnmp.so.25[0x7f36c1413886]
/usr/lib64/libnetsnmp.so.25(_sess_read+0x6c5)[0x7f36c1414f45]
/usr/lib64/libnetsnmp.so.25(snmp_sess_read2+0x9)[0x7f36c1415839]
/usr/lib64/libnetsnmp.so.25(snmp_read2+0x23)[0x7f36c14158f3]
/usr/sbin/snmpd(main+0x1d4d)[0x404c8d]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x39d221d994]
/usr/sbin/snmpd[0x402d29]
======= Memory map: ========
00400000-00406000 r-xp 00000000 08:01 4298203 /usr/sbin/snmpd
00606000-00607000 rw-p 00006000 08:01 4298203 /usr/sbin/snmpd
00607000-008da000 rw-p 00000000 00:00 0 [heap]
39d1e00000-39d1e1c000 r-xp 00000000 08:01 8216955 /lib64/ld-2.5.so
39d201b000-39d201c000 r--p 0001b000 08:01 8216955 /lib64/ld-2.5.so
39d201c000-39d201d000 rw-p 0001c000 08:01 8216955 /lib64/ld-2.5.so
39d2200000-39d234e000 r-xp 00000000 08:01 8216957 /lib64/libc-2.5.so
39d234e000-39d254d000 ---p 0014e000 08:01 8216957 /lib64/libc-2.5.so
39d254d000-39d2551000 r--p 0014d000 08:01 8216957 /lib64/libc-2.5.so
39d2551000-39d2552000 rw-p 00151000 08:01 8216957 /lib64/libc-2.5.so
39d2552000-39d2557000 rw-p 00000000 00:00 0
39d2600000-39d2608000 r-xp 00000000 08:01 8216747 /lib64/libwrap.so.0.7.6
39d2608000-39d2807000 ---p 00008000 08:01 8216747 /lib64/libwrap.so.0.7.6
39d2807000-39d2809000 rw-p 00007000 08:01 8216747 /lib64/libwrap.so.0.7.6
39d2a00000-39d2a82000 r-xp 00000000 08:01 8216966 /lib64/libm-2.5.so
39d2a82000-39d2c81000 ---p 00082000 08:01 8216966 /lib64/libm-2.5.so
39d2c81000-39d2c82000 r--p 00081000 08:01 8216966 /lib64/libm-2.5.so
39d2c82000-39d2c83000 rw-p 00082000 08:01 8216966 /lib64/libm-2.5.so
39d2e00000-39d2e0a000 r-xp 00000000 08:01 4296039 /usr/lib64/libsysfs.so.2.0.0
39d2e0a000-39d3009000 ---p 0000a000 08:01 4296039 /usr/lib64/libsysfs.so.2.0.0
39d3009000-39d300a000 rw-p 00009000 08:01 4296039 /usr/lib64/libsysfs.so.2.0.0
39d3200000-39d3216000 r-xp 00000000 08:01 8217003 /lib64/libpthread-2.5.so
39d3216000-39d3415000 ---p 00016000 08:01 8217003 /lib64/libpthread-2.5.so
39d3415000-39d3416000 r--p 00015000 08:01 8217003 /lib64/libpthread-2.5.so
39d3416000-39d3417000 rw-p 00016000 08:01 8217003 /lib64/libpthread-2.5.so
39d3417000-39d341b000 rw-p 00000000 00:00 0
39d3600000-39d363b000 r-xp 00000000 08:01 8217008 /lib64/libsepol.so.1
39d363b000-39d383b000 ---p 0003b000 08:01 8217008 /lib64/libsepol.so.1
39d383b000-39d383c000 rw-p 0003b000 08:01 8217008 /lib64/libsepol.so.1
39d383c000-39d3846000 rw-p 00000000 00:00 0
39d3a00000-39d3a15000 r-xp 00000000 08:01 8217070 /lib64/libselinux.so.1
39d3a15000-39d3c15000 ---p 00015000 08:01 8217070 /lib64/libselinux.so.1
39d3c15000-39d3c17000 rw-p 00015000 08:01 8217070 /lib64/libselinux.so.1
39d3c17000-39d3c18000 rw-p 00000000 00:00 0
39d3e00000-39d3e07000 r-xp 00000000 08:01 8217004 /lib64/librt-2.5.so
39d3e07000-39d4007000 ---p 00007000 08:01 8217004 /lib64/librt-2.5.so
39d4007000-39d4008000 r--p 00007000 08:01 8217004 /lib64/librt-2.5.so
39d4008000-39d4009000 rw-p 00008000 08:01 8217004 /lib64/librt-2.5.so
39d4200000-39d4215000 r-xp 00000000 08:01 8216971 /lib64/libnsl-2.5.so
39d4215000-39d4414000 ---p 00015000 08:01 8216971 /lib64/libnsl-2.5.so
39d4414000-39d4415000 r--p 00014000 08:01 8216971 /lib64/libnsl-2.5.so
39d4415000-39d4416000 rw-p 00015000 08:01 8216971 /lib64/libnsl-2.5.so
39d4416000-39d4418000 rw-p 00000000 00:00 0
39d4600000-39d4609000 r-xp 00000000 08:01 8216989 /lib64/libcrypt-2.5.so
39d4609000-39d4808000 ---p 00009000 08:01 8216989 /lib64/libcrypt-2.5.so
39d4808000-39d4809000 r--p 00008000 08:01 8216989 /lib64/libcrypt-2.5.so
39d4809000-39d480a000 rw-p 00009000 08:01 8216989 /lib64/libcrypt-2.5.so
39d480a000-39d4838000 rw-p 00000000 00:00 0
39d4a00000-39d4b2d000 r-xp 00000000 08:01 8217072 /lib64/libcrypto.so.0.9.8e
39d4b2d000-39d4d2c000 ---p 0012d000 08:01 8217072 /lib64/libcrypto.so.0.9.8e
39d4d2c000-39d4d4d000 rw-p 0012c000 08:01 8217072 /lib64/libcrypto.so.0.9.8e
39d4d4d000-39d4d51000 rw-p 00000000 00:00 0
39d4e00000-39d4f2c000 r-xp 00000000 08:01 4420539 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
39d4f2c000-39d512b000 ---p 0012c000 08:01 4420539 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
39d512b000-39d5134000 rw-p 0012b000 08:01 4420539 /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
39d5134000-39d5136000 rw-p 00000000 00:00 0
39d5200000-39d5211000 r-xp 00000000 08:01 8216976 /lib64/libresolv-2.5.so
39d5211000-39d5411000 ---p 00011000 08:01 8216976 /lib64/libresolv-2.5.so
39d5411000-39d5412000 r--p 00011000 08:01 8216976 /lib64/libresolv-2.5.so
39d5412000-39d5413000 rw-p 00012000 08:01 8216976 /lib64/libresolv-2.5.so
39d5413000-39d5415000 rw-p 00000000 00:00 0
39d5600000-39d570f000 r-xp 00000000 08:01 4302501 /usr/lib64/librpmdb-4.4.soAborted
--
Tomasz Chmielewski
http://wpkg.org
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Net-snmp-users mailing list
Net-snm...@lists.sourceforge.net
Please see the following page to unsubscribe or change other options:
https://lists.sourceforge.net/lists/listinfo/net-snmp-users
Dave Shield
> snmpd (5.6.1) crashes for me quite frequently after logging "response to proxy request illegal. We're screwed".
>
> Is there anything I can do to debug it further?
What proxy entries do you have in the snmpd.conf file?
What queries are you sending to the agent?
> There was a similar case back in 2005, with no fix though:
>
> http://groups.google.com/group/mailing.unix.net-snmp-users/browse_thread/thread/2454a5a796fb5814/1e7ebe0455051c7a
>From a quick scan of that thread, it looks as if the underlying problem
was a bug in squid, which failed to handle GETNEXT requests for two or
more varbinds correctly.
Are you also working with squid - or do you have different proxied
applications?
Dave
Tomasz Chmielewski
> On 29 March 2011 11:47, Tomasz Chmielewski<man...@wpkg.org> wrote:
>> snmpd (5.6.1) crashes for me quite frequently after logging "response to proxy request illegal. We're screwed".
>>
>> Is there anything I can do to debug it further?
>
> What proxy entries do you have in the snmpd.conf file?
proxy -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10
> What queries are you sending to the agent?
Well, not so sure here.
I checked with tcpdump or strace, and snmpd sends this:
0\201\220\2\1\1\4\6public\240\201\202\2\4\0327\373\313\2\1\0\2\1\0000t0\16\6\n+\6\1\4\1\202\211w\n\t\5\0000\16\6\n+\6\1\4\1\202\211w\n\6\5\0000\16\6\n+\6\1\4\1\202\211w\n\2\5\0000\20\6\f+\6\1\4\1\202\211w\n\r\1\3\5\0000\20\6\f+\6\1\4\1\202\211w\n\r\1\2\5\0000\16\6\n+\6\1\4\1\202\211w\n\6\5\0000\16\6\n+\6\1\4\1\202\211w\n\1\5\0
and gets exactly that:
0+\2\1\1\4\6public\242\36\2\4\0327\373\313\2\1\0\2\1\0000\0200\16\6\n+\6\1\4\1\202\211w\n\t\201\0
After which it crashes.
Probably gets something inappropriate... but still, it shouldn't just
crash IMO.
>> There was a similar case back in 2005, with no fix though:
>>
>> http://groups.google.com/group/mailing.unix.net-snmp-users/browse_thread/thread/2454a5a796fb5814/1e7ebe0455051c7a
>
> From a quick scan of that thread, it looks as if the underlying problem
> was a bug in squid, which failed to handle GETNEXT requests for two or
> more varbinds correctly.
> Are you also working with squid - or do you have different proxied
> applications?
It's a different proxy application.
--
Tomasz Chmielewski
http://wpkg.org
------------------------------------------------------------------------------
Dave Shield
>> What queries are you sending to the agent?
>
> Well, not so sure here.
What is the client application that's sending the original request?
snmpget? snmpwalk? something else?
> I checked with tcpdump or strace, and snmpd sends this:
Hmmm... that's not particularly easy to work with.
Could you try running the agent using the option '-d'
and send the request again. That should show several
raw packets dumps:
- the incoming request from the original client application
- the request forwarded to the proxied agent
- the response from the proxied agent
(and in principle, the response sent back to the client application).
Dave
Tomasz Chmielewski
> On 29 March 2011 17:36, Tomasz Chmielewski<man...@wpkg.org> wrote:
>>> What queries are you sending to the agent?
>>
>> Well, not so sure here.
>
> What is the client application that's sending the original request?
> snmpget? snmpwalk? something else?
snmpd talks to some "nonstandard"/custom application which gets the information about the proxy.
Perhaps it responds with something illegal... Not sure if snmpd should crash though.
>> I checked with tcpdump or strace, and snmpd sends this:
>
> Hmmm... that's not particularly easy to work with.
> Could you try running the agent using the option '-d'
> and send the request again. That should show several
> raw packets dumps:
>
> - the incoming request from the original client application
> - the request forwarded to the proxied agent
> - the response from the proxied agent
>
> (and in principle, the response sent back to the client application).
>From /etc/snmp/snmpd.conf:
proxy -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10
Below, two outputs ending with crash:
1)
could not open /proc/net/if_inet6
error on subcontainer 'ia_addr' insert (-1)
cannot open /proc/net/snmp6 ...
NET-SNMP version 5.6.1
[smux_accept] accepted fd 13 from 127.0.0.1:58421
accepted smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
error on subcontainer 'ia_addr' insert (-1)
error on subcontainer 'ia_addr' insert (-1)
Wrong netlink message type 3
error on subcontainer 'ia_addr' insert (-1)
error on subcontainer 'ia_addr' insert (-1)
Wrong netlink message type 3
error on subcontainer 'ia_addr' insert (-1)
error on subcontainer 'ia_addr' insert (-1)
Wrong netlink message type 3
error on subcontainer 'ia_addr' insert (-1)
Received 36 byte packet from UDP: [a.a.a.a]:51304->[b.b.b.b]:161
0000: 30 22 02 01 01 04 06 70 75 62 6C 69 63 A1 15 02 0".....public...
0016: 04 6E 46 5C EF 02 01 00 02 01 00 30 07 30 05 06 .nF\.......0.0..
0032: 01 2B 05 00 .+..
Connection from UDP: [a.a.a.a]:51304->[b.b.b.b]:161
Received SNMP packet(s) from UDP: [a.a.a.a]:51304->[b.b.b.b]:161
GETNEXT message
-- SNMPv2-SMI::org
Sending 117 bytes to UDP: [a.a.a.a]:51304->[b.b.b.b]:161
0000: 30 73 02 01 01 04 06 70 75 62 6C 69 63 A2 66 02 0s.....public.f.
0016: 04 6E 46 5C EF 02 01 00 02 01 00 30 58 30 56 06 .nF\.......0X0V.
0032: 08 2B 06 01 02 01 01 01 00 04 4A 4C 69 6E 75 78 .+........JLinux
0048: 20 54 52 55 45 2D 50 4F 43 2D 53 43 36 31 30 30 TRUE-POC-SC6100
0064: 20 32 2E 36 2E 33 35 2E 35 20 23 31 20 53 4D 50 2.6.35.5 #1 SMP
0080: 20 54 75 65 20 53 65 70 20 32 38 20 30 30 3A 33 Tue Sep 28 00:3
0096: 33 3A 33 36 20 4E 5A 44 54 20 32 30 31 30 20 78 3:36 NZDT 2010 x
0112: 38 36 5F 36 34 86_64
Received 196 byte packet from UDP: [a.a.a.a]:58756->[b.b.b.b]:161
0000: 30 81 C1 02 01 01 04 06 70 75 62 6C 69 63 A0 81 0.......public..
0016: B3 02 04 6E 46 5C FB 02 01 00 02 01 00 30 81 A4 ...nF\.......0..
0032: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 09 05 00 0...+......w....
0048: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 06 05 00 0...+......w....
0064: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 02 05 00 0...+......w....
0080: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 05 00 05 00 0...+.....e.....
0096: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 06 00 05 00 0...+.....e.....
0112: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 0E 00 05 00 0...+.....e.....
0128: 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 03 0...+......w....
0144: 05 00 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D ..0...+......w..
0160: 01 02 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 ....0...+......w
0176: 0A 06 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 ....0...+......w
0192: 0A 01 05 00 ....
Connection from UDP: [a.a.a.a]:58756->[b.b.b.b]:161
Received SNMP packet(s) from UDP: [a.a.a.a]:58756->[b.b.b.b]:161
GET message
-- SNMPv2-SMI::enterprises.34039.10.9
-- SNMPv2-SMI::enterprises.34039.10.6
-- SNMPv2-SMI::enterprises.34039.10.2
-- UCD-SNMP-MIB::memTotalReal.0
-- UCD-SNMP-MIB::memAvailReal.0
-- UCD-SNMP-MIB::memBuffer.0
-- SNMPv2-SMI::enterprises.34039.10.13.1.3
-- SNMPv2-SMI::enterprises.34039.10.13.1.2
-- SNMPv2-SMI::enterprises.34039.10.6
-- SNMPv2-SMI::enterprises.34039.10.1
Sending 147 bytes to UDP: [127.0.0.1]:1611->[0.0.0.0]:0
0000: 30 81 90 02 01 01 04 06 70 75 62 6C 69 63 A0 81 0.......public..
0016: 82 02 04 49 42 E3 A5 02 01 00 02 01 00 30 74 30 ...IB........0t0
0032: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 09 05 00 30 ...+......w....0
0048: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 06 05 00 30 ...+......w....0
0064: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 02 05 00 30 ...+......w....0
0080: 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 03 05 ...+......w.....
0096: 00 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 .0...+......w...
0112: 02 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
0128: 06 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
0144: 01 05 00 ...
Received 45 byte packet from UDP: [127.0.0.1]:1611->[0.0.0.0]:29711
0000: 30 2B 02 01 01 04 06 70 75 62 6C 69 63 A2 1E 02 0+.....public...
0016: 04 49 42 E3 A5 02 01 00 02 01 00 30 10 30 0E 06 .IB........0.0..
0032: 0A 2B 06 01 04 01 82 89 77 0A 09 81 00 .+......w....
response to proxy request illegal. We're screwed.
2)
could not open /proc/net/if_inet6
error on subcontainer 'ia_addr' insert (-1)
cannot open /proc/net/snmp6 ...
NET-SNMP version 5.6.1
[smux_accept] accepted fd 13 from 127.0.0.1:32158
accepted smux peer: oid SNMPv2-SMI::enterprises.674.10892.1, descr Systems Management SNMP MIB Plug-in Manager
error on subcontainer 'ia_addr' insert (-1)
error on subcontainer 'ia_addr' insert (-1)
Wrong netlink message type 3
error on subcontainer 'ia_addr' insert (-1)
error on subcontainer 'ia_addr' insert (-1)
Wrong netlink message type 3
error on subcontainer 'ia_addr' insert (-1)
Received 36 byte packet from UDP: [a.a.a.a]:53133->[b.b.b.b]:161
0000: 30 22 02 01 01 04 06 70 75 62 6C 69 63 A1 15 02 0".....public...
0016: 04 45 5D ED E7 02 01 00 02 01 00 30 07 30 05 06 .E]........0.0..
0032: 01 2B 05 00 .+..
Connection from UDP: [a.a.a.a]:53133->[b.b.b.b]:161
Received SNMP packet(s) from UDP: [a.a.a.a]:53133->[b.b.b.b]:161
GETNEXT message
-- SNMPv2-SMI::org
Sending 117 bytes to UDP: [a.a.a.a]:53133->[b.b.b.b]:161
0000: 30 73 02 01 01 04 06 70 75 62 6C 69 63 A2 66 02 0s.....public.f.
0016: 04 45 5D ED E7 02 01 00 02 01 00 30 58 30 56 06 .E]........0X0V.
0032: 08 2B 06 01 02 01 01 01 00 04 4A 4C 69 6E 75 78 .+........JLinux
0048: 20 54 52 55 45 2D 50 4F 43 2D 53 43 36 31 30 30 TRUE-POC-SC6100
0064: 20 32 2E 36 2E 33 35 2E 35 20 23 31 20 53 4D 50 2.6.35.5 #1 SMP
0080: 20 54 75 65 20 53 65 70 20 32 38 20 30 30 3A 33 Tue Sep 28 00:3
0096: 33 3A 33 36 20 4E 5A 44 54 20 32 30 31 30 20 78 3:36 NZDT 2010 x
0112: 38 36 5F 36 34 86_64
Received 196 byte packet from UDP: [a.a.a.a]:57903->[b.b.b.b]:161
0000: 30 81 C1 02 01 01 04 06 70 75 62 6C 69 63 A0 81 0.......public..
0016: B3 02 04 45 5D ED E9 02 01 00 02 01 00 30 81 A4 ...E]........0..
0032: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 09 05 00 0...+......w....
0048: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 06 05 00 0...+......w....
0064: 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A 02 05 00 0...+......w....
0080: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 05 00 05 00 0...+.....e.....
0096: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 06 00 05 00 0...+.....e.....
0112: 30 0E 06 0A 2B 06 01 04 01 8F 65 04 0E 00 05 00 0...+.....e.....
0128: 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 03 0...+......w....
0144: 05 00 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D ..0...+......w..
0160: 01 02 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 ....0...+......w
0176: 0A 06 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 ....0...+......w
0192: 0A 01 05 00 ....
Connection from UDP: [a.a.a.a]:57903->[b.b.b.b]:161
Received SNMP packet(s) from UDP: [a.a.a.a]:57903->[b.b.b.b]:161
GET message
-- SNMPv2-SMI::enterprises.34039.10.9
-- SNMPv2-SMI::enterprises.34039.10.6
-- SNMPv2-SMI::enterprises.34039.10.2
-- UCD-SNMP-MIB::memTotalReal.0
-- UCD-SNMP-MIB::memAvailReal.0
-- UCD-SNMP-MIB::memBuffer.0
-- SNMPv2-SMI::enterprises.34039.10.13.1.3
-- SNMPv2-SMI::enterprises.34039.10.13.1.2
-- SNMPv2-SMI::enterprises.34039.10.6
-- SNMPv2-SMI::enterprises.34039.10.1
Sending 147 bytes to UDP: [127.0.0.1]:1611->[0.0.0.0]:0
0000: 30 81 90 02 01 01 04 06 70 75 62 6C 69 63 A0 81 0.......public..
0016: 82 02 04 0E 7B 78 00 02 01 00 02 01 00 30 74 30 ....{x.......0t0
0032: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 09 05 00 30 ...+......w....0
0048: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 06 05 00 30 ...+......w....0
0064: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 02 05 00 30 ...+......w....0
0080: 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 03 05 ...+......w.....
0096: 00 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 .0...+......w...
0112: 02 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
0128: 06 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
0144: 01 05 00 ...
Received 45 byte packet from UDP: [127.0.0.1]:1611->[0.0.0.0]:47463
0000: 30 2B 02 01 01 04 06 70 75 62 6C 69 63 A2 1E 02 0+.....public...
0016: 04 0E 7B 78 00 02 01 00 02 01 00 30 10 30 0E 06 ..{x.......0.0..
0032: 0A 2B 06 01 04 01 82 89 77 0A 09 81 00 .+......w....
response to proxy request illegal. We're screwed.
--
Tomasz Chmielewski
http://wpkg.org
------------------------------------------------------------------------------
Dave Shield
> Below, two outputs ending with crash:
> Sending 147 bytes to UDP: [127.0.0.1]:1611->[0.0.0.0]:0
> 0000: 30 81 90 02 01 01 04 06 70 75 62 6C 69 63 A0 81 0.......public..
> 0016: 82 02 04 49 42 E3 A5 02 01 00 02 01 00 30 74 30 ...IB........0t0
> 0032: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 09 05 00 30 ...+......w....0
> 0048: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 06 05 00 30 ...+......w....0
> 0064: 0E 06 0A 2B 06 01 04 01 82 89 77 0A 02 05 00 30 ...+......w....0
> 0080: 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 03 05 ...+......w.....
> 0096: 00 30 10 06 0C 2B 06 01 04 01 82 89 77 0A 0D 01 .0...+......w...
> 0112: 02 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
> 0128: 06 05 00 30 0E 06 0A 2B 06 01 04 01 82 89 77 0A ...0...+......w.
> 0144: 01 05 00 ...
>
>
> Received 45 byte packet from UDP: [127.0.0.1]:1611->[0.0.0.0]:29711
> 0000: 30 2B 02 01 01 04 06 70 75 62 6C 69 63 A2 1E 02 0+.....public...
> 0016: 04 49 42 E3 A5 02 01 00 02 01 00 30 10 30 0E 06 .IB........0.0..
> 0032: 0A 2B 06 01 04 01 82 89 77 0A 09 81 00 .+......w....
OK - that's definitely useful.
It looks as if the response that comes back is bogus.
What seems to be happening is that the proxied agent (running on port
1161) is asked for several varbinds, but only returns a value (actually
the exception noSuchInstance) for the first one.
The Net-SNMP agent is expecting to get a value (or exception) for
*each* of the requested varbinds, and gets confused when the second
and subsequent varbinds simply aren't there.
You're quite correct - the agent shouldn't crash. But there's definitely
something wrong with the proxied application.
Let's just check exactly what's happening, by taking the Net-SNMP agent
out of the equation.
Can you please try the following requests, and let us know what response
you get:
- snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
- snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
.1.3.6.1.4.1.34039.10.9
- snmpgetnext -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.8
- snmpgetnext -v 2c -c public localhost:1161
.1.3.6.1.4.1.34039.10.8 .1.3.6.1.4.1.34039.10.1
- snmpgetnext -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10
- snmpwalk -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10
Thanks
Dave
Tomasz Chmielewski
> Let's just check exactly what's happening, by taking the Net-SNMP agent
> out of the equation.
> Can you please try the following requests, and let us know what response
> you get:
>
> - snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
>
> - snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
> .1.3.6.1.4.1.34039.10.9
>
> - snmpgetnext -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.8
>
> - snmpgetnext -v 2c -c public localhost:1161
> .1.3.6.1.4.1.34039.10.8 .1.3.6.1.4.1.34039.10.1
>
> - snmpgetnext -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10
>
> - snmpwalk -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10
It's like below; I get a timeout on the last one:
# snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9
SNMPv2-SMI::enterprises.34039.10.9 = No Such Instance currently exists at this OID
# snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9 .1.3.6.1.4.1.34039.10.9
SNMPv2-SMI::enterprises.34039.10.9 = No Such Instance currently exists at this OID
# snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.8
SNMPv2-SMI::enterprises.34039.10.8 = No more variables left in this MIB View (It is past the end of the MIB tree)
# snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.8 .1.3.6.1.4.1.34039.10.1
SNMPv2-SMI::enterprises.34039.10.8 = No more variables left in this MIB View (It is past the end of the MIB tree)
SNMPv2-SMI::enterprises.34039.10.1.1.1.0 = STRING: "Version: v2.1.6-53-g8ef652f"
# snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10
SNMPv2-SMI::enterprises.34039.10.1.1.1.0 = STRING: "Version: v2.1.6-53-g8ef652f"
# snmpwalk -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10
SNMPv2-SMI::enterprises.34039.10.1.1.1.0 = STRING: "Version: v2.1.6-53-g8ef652f"
SNMPv2-SMI::enterprises.34039.10.1.1.2.0 = STRING: "T000000FNPW62S90104C4G"
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.1.0 = STRING: "1030.85"
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.2.0 = STRING: "1748.2"
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.3.0 = Gauge32: 10388
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.4.0 = Gauge32: 0
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.5.0 = Gauge32: 7733
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.6.0 = Gauge32: 2483
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.7.0 = Gauge32: 168
SNMPv2-SMI::enterprises.34039.10.1.2.1.1.8.0 = Gauge32: 3
SNMPv2-SMI::enterprises.34039.10.1.2.1.2.1.0 = STRING: "10.332675167"
SNMPv2-SMI::enterprises.34039.10.1.2.1.2.2.0 = STRING: "319.585084852"
SNMPv2-SMI::enterprises.34039.10.1.2.1.2.3.0 = STRING: "174.598732032"
SNMPv2-SMI::enterprises.34039.10.1.2.1.2.4.0 = STRING: "7.80559765778"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.1.0 = STRING: "1583.96"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.2.0 = STRING: "328.0"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.3.0 = STRING: "2175.0"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.4.0 = STRING: "47.0"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.5.0 = STRING: "32.0"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.6.0 = STRING: "19.0"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.1.1 = STRING: "0 ~ 1K"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.1.2 = STRING: "118896"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.1.3 = STRING: "12346096"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.2.1 = STRING: "1K ~ 3K"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.2.2 = STRING: "34803541"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.2.3 = STRING: "113526944"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.3.1 = STRING: "3K ~ 10K"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.3.2 = STRING: "9445537"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.3.3 = STRING: "37340344"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.4.1 = STRING: "10K ~ 32K"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.4.2 = STRING: "19697860"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.4.3 = STRING: "57778674"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.5.1 = STRING: "32K ~ 100K"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.5.2 = STRING: "14400419"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.5.3 = STRING: "28564609"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.6.1 = STRING: "100K ~ 1M"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.6.2 = STRING: "5886262"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.6.3 = STRING: "10136818"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.7.1 = STRING: "1M ~ 3M"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.7.2 = STRING: "2528318"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.7.3 = STRING: "4335186"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.8.1 = STRING: "3M ~ 10M"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.8.2 = STRING: "125959"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.8.3 = STRING: "244518"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.9.1 = STRING: "10M ~ 32M"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.9.2 = STRING: "45348"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.9.3 = STRING: "119979"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.10.1 = STRING: "32M ~ 100M"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.10.2 = STRING: "11675"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.10.3 = STRING: "48239"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.11.1 = STRING: "100M ~ 1G"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.11.2 = STRING: "3716"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.11.3 = STRING: "18591"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.12.1 = STRING: ">1G"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.12.2 = STRING: "670"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.7.12.3 = STRING: "4421"
SNMPv2-SMI::enterprises.34039.10.1.2.1.3.8.0 = STRING: "152.787"
SNMPv2-SMI::enterprises.34039.10.1.2.2.1.1.0 = STRING: "1565.839"
SNMPv2-SMI::enterprises.34039.10.1.2.2.1.2.0 = STRING: "1761.0"
Timeout: No Response from localhost:1611
--
Tomasz Chmielewski
http://wpkg.org
Dave Shield
>> - snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
>> .1.3.6.1.4.1.34039.10.9
My apologies - that should have been
snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
.1.3.6.1.4.1.34039.10.2
> # snmpwalk -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10
> SNMPv2-SMI::enterprises.34039.10.1.1.1.0 = STRING: "Version: v2.1.6-53-g8ef652f"
:
> SNMPv2-SMI::enterprises.34039.10.1.2.2.1.2.0 = STRING: "1761.0"
> Timeout: No Response from localhost:1611
Hmmm...
What happens with:
- snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.2
- snmpgetnext -v 2c -c public localhost:1611
.1.3.6.1.4.1.34039.10.1.9999
?
Dave
Tomasz Chmielewski
> On 30 March 2011 10:03, Tomasz Chmielewski<man...@wpkg.org> wrote:
>>> - snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
>>> .1.3.6.1.4.1.34039.10.9
>
> My apologies - that should have been
>
> snmpget -v 2c -c public localhost:1161 .1.3.6.1.4.1.34039.10.9
> .1.3.6.1.4.1.34039.10.2
We were getting:
# snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9 .1.3.6.1.4.1.34039.10.2
SNMPv2-SMI::enterprises.34039.10.9 = No Such Instance currently exists at this OID
But - thanks for your help - I think we found a bug in the application and it now replies:
# snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9 .1.3.6.1.4.1.34039.10.2
SNMPv2-SMI::enterprises.34039.10.9 = INTEGER: 0
SNMPv2-SMI::enterprises.34039.10.2 = No Such Instance currently exists at this OID
snmpd does not crash anymore. Could it be it was the cause for snmpd crashing?
Other outputs you asked for (before, after the "fix" the output is the same):
# snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.2
SNMPv2-SMI::enterprises.34039.10.2 = No more variables left in this MIB View (It is past the end of the MIB tree)
# snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.1.9999
SNMPv2-SMI::enterprises.34039.10.1.9999 = No more variables left in this MIB View (It is past the end of the MIB tree)
--
Tomasz Chmielewski
http://wpkg.org
------------------------------------------------------------------------------
Dave Shield
> But - thanks for your help - I think we found a bug in the application and it now replies:
>
> # snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9 .1.3.6.1.4.1.34039.10.2
> SNMPv2-SMI::enterprises.34039.10.9 = INTEGER: 0
> SNMPv2-SMI::enterprises.34039.10.2 = No Such Instance currently exists at this OID
>
>
> snmpd does not crash anymore. Could it be it was the cause for snmpd crashing?
Yes - an incomplete response might well result in the Net-SNMP agent crashing.
I'll look to see if I can spot how to protect against this.
But I can see another problem.
The response above shows that the proxied agent is now returning a value
for .1.3.6.1.4.1.34039.10.9
This means that a GETNEXT request for any OID prior to this,
ought to return the same varbind (or something earlier).
However:
> Other outputs you asked for (before, after the "fix" the output is the same):
>
> # snmpgetnext -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.2
> SNMPv2-SMI::enterprises.34039.10.2 = No more variables left in this MIB View
This is reporting that there isn't anything after ....10.2
These two responses are inconsistent.
Either GET on 10.9 should return noSuchInstance
or GETNEXT on 10.2 should return the 10.9 varbind
And given the structure of the information returned under 10.1,
I'm really not convinced that 10.9 can possibly be a valid
instance OID. What does the MIB that defines this
information look like?
Dave
Tomasz Chmielewski
> On 30 March 2011 10:49, Tomasz Chmielewski<man...@wpkg.org> wrote:
>> But - thanks for your help - I think we found a bug in the application and it now replies:
>>
>> # snmpget -v 2c -c public localhost:1611 .1.3.6.1.4.1.34039.10.9 .1.3.6.1.4.1.34039.10.2
>> SNMPv2-SMI::enterprises.34039.10.9 = INTEGER: 0
>> SNMPv2-SMI::enterprises.34039.10.2 = No Such Instance currently exists at this OID
>>
>>
>> snmpd does not crash anymore. Could it be it was the cause for snmpd crashing?
>
> Yes - an incomplete response might well result in the Net-SNMP agent crashing.
> I'll look to see if I can spot how to protect against this.
Thanks for your help.
It looks the applications we've been using had a number of bugs related
to SNMP and your input helped find and fix some of them.
If you do any changes which protect against such application errors,
I'll be happy to test them - instead of crashing, some logging would be
more appropriate ;)
--
Tomasz Chmielewski
http://wpkg.org
------------------------------------------------------------------------------