feature request: pipe packet and metadata to external program

[Jason Haar]

Hi there

I just brought this up in the "Email" thread on snort-users. How about a new "output" option in barnyard that calls an external program? I'd envisage it sending snort metadata as environment variables and the "virtual packet" the rule triggered on via a temp file - something like that?


eg

output external: prog=/usr/local/bin/snortalert %packet

Today I use swatch to monitor the syslog events to trigger emails. But I do things in that email script using the snort details shown in syslog to drop the email if the event involves a "known good" source, etc. There are way too many things people could think of that could be cool to do, so a generic "output" mechanism could very well be the best way of achieving it?

Jason

[Jason Haar]

Weird - I got beenph's reply via email, but it's not showing up in this browser view?

Anyway, the issue is that I can't see an output option besides the unified ones that contain both packet information and metadata (classification, sid, etc) - the syslog/alert.file options only contain metadata as far as I can see? The "text mode" option writes an ASCII representation of the packet content out - but it's not truly the packet - and docs say it's slow too as barnyard has to do the extra thinking and writing I suppose. There are those other output modules to cef/prelude - but again I think they are only sent the metadata - not the packet itself? (I don't use them so I dunno)

So it seems to me that I basically have to use unified and write my own barnyard replacement in order to be able to "fiddle" with the internals of an alert? Hence my question about adding it as a feature to barnyard instead :-)

What about writing to a socket instead of executing a program? That way I could have a program that monitors that socket, and does whatever it wants when a line of data comes in, and barnyard would just rattle on its way, not caring what happens next? Load then becomes the script's problem.

The issue I have is that today I use a rather complicated script triggered off of syslog to decide whether or not a snort event should generate a true alert (eg email). I do "extra" checks like downloading the server cert on events that involve port 443, or doing DNS PTR lookups, etc. On top of that I want to do things like look inside the packet that caused the event - to pull out URLs or filenames - so that they can influence script behaviour and show up in the email directly - ie I want to gain CONTEXT in order to make decisions about how important this raw IDS event actually is. That I can't do at the moment using the output modules such as syslog.

Jason

[beenph]

On Thu, Aug 30, 2012 at 5:22 PM, Jason Haar <jason...@gmail.com> wrote:
>
> Weird - I got beenph's reply via email, but it's not showing up in this
> browser view?

Browser view?
https://groups.google.com/forum/?fromgroups#!topic/barnyard2-users/yvRLdBREKQM
...
i agree it dosen't seem to show up even tho i effectively replied to the list.

>
> Anyway, the issue is that I can't see an output option besides the unified
> ones that contain both packet information and metadata (classification, sid,
> etc) - the syslog/alert.file options only contain metadata as far as I can
> see? The "text mode" option writes an ASCII representation of the packet
> content out - but it's not truly the packet - and docs say it's slow too as
> barnyard has to do the extra thinking and writing I suppose. There are those
> other output modules to cef/prelude - but again I think they are only sent
> the metadata - not the packet itself? (I don't use them so I dunno)
>

syslog_full in 2.1-10 with (output log_syslog_full: operation_mode
complete, etc...)
will output the full hexed packet for example.

> So it seems to me that I basically have to use unified and write my own
> barnyard replacement in order to be able to "fiddle" with the internals of
> an alert? Hence my question about adding it as a feature to barnyard instead
> :-)

Nothing wrong with writing your own unified2 processing script if you have
more specific needs, or you can write your own output plugin and adapt
it for your needs.

> What about writing to a socket instead of executing a program? That way I
> could have a program that monitors that socket, and does whatever it wants
> when a line of data comes in, and barnyard would just rattle on its way, not
> caring what happens next? Load then becomes the script's problem.

Unix Domain Socket? UDS to be reliable will be blocking on I/O and if
you have to wait until it unblock then it can have an impact on event
processing/event logging if
you have other output plugin enabled at the same time. (Which is not
really an issue if its only this
output plugin that is enabled).

>
> The issue I have is that today I use a rather complicated script triggered
> off of syslog to decide whether or not a snort event should generate a true
> alert (eg email). I do "extra" checks like downloading the server cert on
> events that involve port 443, or doing DNS PTR lookups, etc. On top of that
> I want to do things like look inside the packet that caused the event - to
> pull out URLs or filenames - so that they can influence script behaviour and
> show up in the email directly - ie I want to gain CONTEXT in order to make
> decisions about how important this raw IDS event actually is. That I can't
> do at the moment using the output modules such as syslog.
>

Because with the details i read in the original e-mail, and mabey im
still not understanding, but i do not see
how invocation of a script on each event will allow the script be make
better decision
than having a larger event base to correlate and evaluate the information.

If i interpret the need correctly, i think that if you want to make
decision based on contextual interpretation of an event,
you need to be able to look back and store some information to be able
to adjust whatever the storage mechanism is ex: file database,
relational database
or NO SQL Store.

If you only base your decision on a runtime analysis ex (being called
by a external program) wouldnt there be smarter ways to make
decisions?

-elz

[Jason Haar]

On Friday, 31 August 2012 10:20:59 UTC+12, bob walsh wrote:

Because with the details i read in the original e-mail, and mabey im
still not understanding, but i do not see
how invocation of a script on each event will allow the script be make
better decision
than having a larger event base to correlate and evaluate the information.

Alerting is what this is all about. When I say "alert" - I am referring to declaring that a "security event" such as a snort event is interesting enough that a human needs to be notified. One thing I have to worry about is that we have a very small security team (2), and 99% of incident response is handled by non-security operational personnel. I have to be extremely sensitive to not flooding them with false positives (because they will get bored and stop reacting), and that the alerts that do get to them contain as much detail as possible - as helpdesk staff can not be expected to be able to analyse low-level data shown to them via base, etc. Our current system is hooked into our Asset Tracking and Active Directory systems, so internal IPs mentioned in an event are converted into user@hostname, and reference the (internal) sitename, the user's email and phone number, etc. None of this is an IDS problem/responsibility, but I think it's critical to make an IDS be usable within a large organization context. Sourcefire have separate products too for doing this sort of thing.

Generating alerts based on statistical decisions (which is what I guess you are talking about?) is not what I am after. I'm just after making more information from snort events available to alerting.

eg, for ET rule "Possible Dynamic DNS Exploit Pack Payload" which alerts on urls containing specific uri components, it would be much better if that was "Possible Dynamic DNS Exploit Pack Payload seen on http://bbs.skykiwi.com/forum.php". Not only does that encapsulate basically the entire content of the metadata for that particular alert, but it is "normal IT person readable". It also gives me the ability to filter that out as not been interesting - without adding "pass" rules to snort, or excluding the entire rule. Another example: I've written snort rules so that we can monitor remote access connections to our DMZ hosts - eg via terminal server. RDP actually sends the username of the person connecting in the first few packets, so I have a snort rule for recording that - but snort doesn't give me a mechanism to extract that username out and put into the "msg" string. Being able to post-process the packet as part of an event would enable me to create much more usable alerts or even logging records: "RDP connection to DMZ host 1.2.3.4 by "frank""

You mention this new syslog_full option that's coming in 2.1-10 - that would do it I guess. But a snort packet can be >>1024 bytes - what's the maximum size of a syslog record? ie will the bigger ones fit?

Thanks for listening! ;-)

Jason

[beenph]

Lets say you consider that you do not need behavioral analysis or
history to determine
if your prototyped security event sent to the team is valid, then i
guess you could launch a script
to do this, but again i do not see how mutch overly complicated it
would be to use eg: a database that also has
the ability to determine threshold (occurences) of events, determine
different sources,destination timeline , ports etc.

Once the data is there it stays there and you can flush it /archive it
make reports without using "base" or sguil or snoby or a defined UI.
You can just have the database storing the information for you and
this can allow you to handle it as you want run hourly cron have other
tables
stored with "computed information" from previous execution, define
specific rule in the database interpreted by your script etc..

I think it mainly remove the need of Runtime interpretation and can
allow you to developp buisness rules arround it.

Just a tought.

> You mention this new syslog_full option that's coming in 2.1-10 - that would
> do it I guess. But a snort packet can be >>1024 bytes - what's the maximum
> size of a syslog record? ie will the bigger ones fit?
>

Stable branch already have it so you can go on github and test it see
if it fits your needs.

from syslog(3) man page..

The kernel log buffer
The kernel has a cyclic buffer of length LOG_BUF_LEN (4096,
since 1.3.54: 8192, since 2.1.113: 16384; in recent kernels the size
can be set at compile time) in which messages given as argument to the
kernel
function printk() are stored (regardless of their loglevel).

-elz