Zenoss 4.2.5 - Incoming Traps - Flood control

[Sam Nirmal]

Hello,

With the flood, the backlog goes up. Zeneventserver will consume the events in the queue and triggers/notifications will follow. I think this puts a great deal of burden on zeneventserver and

would impact its performance to timely processing the events of all.  Most of the devices sending trap have no control on their side. 

I wonder if there is a way in Zenoss or by others means on the Zenoss host to have flood control of traps?

How people deal with the flood or just let it go through?

Thanks

[Jane Curry]

What version of Zenoss are you on, Sam?  If V4, what patch level?  It makes a big difference to the way trap filtering is handled by Zenoss.

Cheers,

Jane

[Sam Nirmal]

Zenoss version is 4.2.5 with SUP743. 

Thanks

[Jane Curry]

So.... with a huge caveat that this is regarding Zenoss 4.2.5 SUP743 and  NOT Zenoss 5 or 6........

There is a virtually undocumented feature in zentrap.conf that allows you to specify a trapfilterfile parameter - not sure when this was introduced. I have:

trapfilterfile zentrap.filter.conf         just give the filename, not the full path; filename can be anything

You then create your file (zentrap.filter.conf in my case) in $ZENHOME/etc (opt/zenoss/etc in my case). For example:

#File for filtering traps - one OID per line
.1.3.6.1.4.1.123.1234
#.1.3.6.1.4.1.123.1234  # This doesnt break things but the comment at the end  is also part of the filter string
#.1.3.6.garbage.1.4.1.123.1234  # This doesn't break things - but clearly isnt very useful
.1.3.6.1.4.1.123.1235
.1.3.6.1.4.1.123.1236
.1.3.6.1.4.1.123.0.1236

hubenabledtrapfilter in zentrap.conf also has to be set to False (which is the default), for this filtering to work.

The nasty bit is that I cannot find anything that clearly specifies whether this is supposed to filter IN or filter OUT.  The code actually filters IN ie. you only see the TRAP event if the oid field in the Event Details exactly matches one of your filter lines.  Personally, this isn't what I want at all. I want all TRAPs to appear as an event UNLESS they are exactly matched by a line in my trapfilterfile.

If you look at zentrap.py in /opt/zenoss/Products/ZenEvents, look for the definition of the TrapFilter class around line 774.  The _read_oids function adds $ZENHOME/etc to the trapfilterfile you specified and creates a set of the OIDs, one per line, stripping off leading and trailing dots and white space.

It is the  _dropOid function which I think is wrong (or at least doesn't suit my requirements):

    def _dropOid(self, oid):
        # JC
        # I think to drop events in self._oids this should NOT have the not
        #return oid not in self._oids
        return oid in self._oids

As my 3 comment lines indicate I want to drop events that ARE listed in the file (which, by now, is self._oids) so my live code returns:

oid in self.(oids)

The transform  function calls _dropOid after it has got the oid off the event (which you can check in the Event Console, Event Details).:

trapOid = event.get('oid', None)

...

if self._dropOid(trapOid):
                    log.warn("Dropping trap %s", trapOid)
                    result = TRANSFORM_DROP

You need to restart the zentrap daemon (and only zentrap) if you change the name of the trapfilterfile in zentrap.conf, or change the contents of your trapfilterfile, or edit the code in zentrap.py.

Run the zentrap daemon in debug mode (logseverity 10 in zentrap.conf) or you can turn on debugging on-the-fly with:

zentrap debug                        this is a toggle so another zentrap debug will turn debug level off again

I have done limited testing and this seems to be doing what I want.  If you have a very limited set of TRAPs that you DO want to see then run the zentrap code as-is so that it filters IN.

For anyone running Zenoss 6 there has clearly been a huge amount of change in this area.  Similar parameters exist in zentrap.conf but the format of the filter file has changed and a sample is provided in $ZENHOME/etc/zentrap.filter.conf.  There is a separate $ZENHOME/Products/ZenEvents/TrapFilter.py module which has lots more complexity and checking in it.

As ever, this is a small piece of research NOT guaranteed production code. If you are going to try implementing it, test well.

Good luck,

Jane

[Sam Nirmal]

Probably "trapfilterfile" feature was introduced in 5.1.  As for 4.2.5, the only option I can think of is to use iptables rules on the collector.

Thanks