Core thread changes (git) 11/25/2011

[Champ Clark III [Quadrant]]

Made some core changes to how Sagan handles threads.   Traditionally,  there where thread variables for each type of output plugin (excluding alert/ASCII and unified2).   This made the code a bit more complex and caused Sagan to spawn multiple threads for multiple plugins.  In the case of output plugins,  we're trying to avoid any I/O blocking that would cause Sagan not to do it's job and drop events. The most recent change creates one thread for all output plugin tasks.  

This reduces the memory footprint and most likely increases the number of events Sagan can process.  

When Sagan needs to log to a possible I/O bound process,  a single thread is created.  This thread will handle _all_ output plugins that required I/O.   This make the code easier to read/follow as well. 

A new thread handler has been created for CPU intensive events.  These are not related to output plugins,  but rather,   log analysis events.  One example might be for Sagan to do GeoIP lookups based on source IP addresses within logs.    While the processing time isn't great (a few milliseconds?),  we'd still want Sagan to be processing new events.   Currently,  no "processors" have been created,  but several are planned.   

Restructuring of the code now allows for future development of "processor" plugins.  For example:

1. Detecting when known machines that log stop logging.

2. Log lines anomalies (to few or to many logs)

3. GeoIP lookup's of IP addresses in logs (For example,  you _might_ want to flag if a log message originates for russia)

4) Lookup of known hostile IP address (Russian Business Networks,  C&C servers, etc)

Just to name a few. 

Champ Clark III
(office) 904.253.7856

(mobile) 850.443.2440 

(SOC) 800.538.9357 ext 101

ccl...@quadrantsec.com
www.quadrantsec.com