How to restart snmptrapd?

[jdMelton]

I ran apt-get update and apt-get upgrade on my farm (1371) today. In
reading other posts, I noted that both snmpd and snmpdtrapd are
needed. So, before running them, I ran ps -ef > log.ps.ef to capture
the farm's post startup processes.

After the update/upgrade, I ran ps -ef again and compared it to the
log.ps.ef file. OK on www and mysqllvm role servers, but snmptrapd was
missing from the app role.

To my own discredit, I did not check /etc/init.d for snmptrapd! I did
see the binary location in the log.ps.ef file at /usr/sbin/snmptrapd.

I ran grep -r 'snmptrapd' /etc and found references to snmptrapd in /
etc/init.d/snmpd. So, I tried running /etc/init.d/snmpd restart. No
luck.

The /etc/snmp/snmptrapd.conf file is present and has not been modified
by me.

So, how does one restart snmptrapd if it is stopped?

Perhaps a related question is why does snmptrapd stop in the first
place on app roles but not www or mysqllvm?

[jdMelton]

Reply to my own post.

After posting yesterday, I continued researching, but did not come up
with any answers. My guess is that the snmp package upgrade broke
something. So I terminated and restarted my farm this morning. I still
need to resolve this problem.

[Nickolas Toursky]

snmpd init script looks into /etc/default/snmpd file for startup
variables. TRAPDRUN should be set to "yes" in order to start snmptrapd
every time snmpd is started.

[jdMelton]

I found that it was my script that holds port 162 open for some
reason. The troubleshooting steps might be of interest to someone
else, so I will post them below.

My script runs apt-get -y -q update and apt-get -y -q upgrade.

Files whose mtimes were unchanged after update/upgrade:
/etc/default/snmpd
/etc/init.d/snmpd
/etc/snmp/snmpd.conf
/etc/snmp/snmptrapd.conf

After update/upgrade, running ps -ef showed the snmptrapd process was
no longer running.
I ran /etc/init.d/snmpd stop then /etc/init.d/snmpd start.
On stdout, there was: Starting network management services: snmpd
snmptrapd.
However, snmptrapd did not start.

Running grep 'snmptrapd' /var/log/syslog revealed this as the last
entries:
Mar 16 08:43:10 ec2-75-101-205-239 snmptrapd[10987]: NET-SNMP version
5.4.1
Mar 16 08:43:10 ec2-75-101-205-239 snmptrapd[10987]: couldn't open udp:
162 -- errno 98 ("Address already in use")

Running cat /etc/services|grep 162 returned:
snmp-trap 162/tcp snmptrap # Traps for SNMP
snmp-trap 162/udp snmptrap # Traps for SNMP

So, no problem there, only snmptrap should be on port 162.

Running netstat -lnp returned:
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 0.0.0.0:11211 0.0.0.0:*
LISTEN 1933/memcached
tcp 0 0 0.0.0.0:80 0.0.0.0:*
LISTEN 10596/apache2
tcp 0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 1921/sshd
udp 0 0 0.0.0.0:161
0.0.0.0:* 10985/snmpd
udp 0 0 0.0.0.0:162
0.0.0.0:* 10070/bash
udp 0 0 0.0.0.0:68
0.0.0.0:* 1509/dhclient3
udp 0 0 10.248.110.176:123
0.0.0.0:* 4648/ntpd
udp 0 0 127.0.0.1:123
0.0.0.0:* 4648/ntpd
udp 0 0 0.0.0.0:123
0.0.0.0:* 4648/ntpd
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program
name Path

This shows bash running on port 162!

Running ps -ef | grep '10070' returned:
root 10070 1 0 Mar15 ? 00:00:00 /bin/bash /root/bin/
app.post.deploy.check.sh

Ran kill 10070 and /etc/init.d/snmpd restart.

Now snmpd and snmptrapd were running, as returned by ps -ef | grep
snmp:
snmp 11114 1 0 09:08 ? 00:00:00 /usr/sbin/snmpd -Lsd -
Lf /dev/null -u snmp -I -smux -p /var/run/snmpd.pid 0.0.0.0
root 11117 1 0 09:08 ? 00:00:00 /usr/sbin/snmptrapd -
Lsd -p /var/run/snmptrapd.pid

That was my script! Why was it holding open that port? The script does
not do this on www or mysqllvm role instances. I will have to figure
that out.