Support for logging new Unified2 "extra data" output in spo_alert_full and spo_database in 1.9 and 1.10-beta2

[Brett Edgar]

I have attached patch files that add support for logging the Unified2
"extra data" information to the alert_full and database output
plugins. There is a patch file for 1.9, and one for the git
repository of 1.10-beta2 as of this morning. The patch includes the
schema change for MySQL, but that should be easily ported to the other
DB types by someone who can test such a change.

Regards,

Brett

[beenph]

Thanks Brett,
We will have a look into it.

The overall patch seem to be fine, but since there is no support from
the UI's that
i am aware off (yet) application of full EXTRA_DATA support will
probably be posponed.

Since there is a few changes in the pipe incomming.

One thing tho is that the schema you proposed shouldn't drop tables,
this could lead to people
droping their databases and thus loosing some information (if they are
not carefull).

-elz

[Brett Edgar]

Apologies, those DROPs were in there for my testing purposes. That
said, if you are running the "create_*sql" script, you better believe
it is going to create you a clean database. It's not going to work
for upgrading the schema, anyway, since it would fail at the first
"create table" statement for an existing table. If you are crazy
enough to run the create_*sql script, I would hope you know what you
are doing.

It is true that no _public_ UIs support the extra data, but they
probably will be more inclined to support it if the data is there!
Chicken and egg problem... Barnyard2 exists to get the information
from the unified2 output to the database, not to support any
particular UI. I feel like if Barnyard2 says it supports the unified2
format, it better read everything that it can from that file and get
it into some output somewhere...

-Brett

[beenph]

On Tue, Sep 20, 2011 at 4:46 PM, Brett Edgar <brett...@gmail.com> wrote:
> Apologies, those DROPs were in there for my testing purposes.  That
> said, if you are running the "create_*sql" script, you better believe
> it is going to create you a clean database.  It's not going to work
> for upgrading the schema, anyway, since it would fail at the first
> "create table" statement for an existing table.  If you are crazy
> enough to run the create_*sql script, I would hope you know what you
> are doing.
>

You do not need to excuse anything it was only a remark.

> It is true that no _public_ UIs support the extra data, but they
> probably will be more inclined to support it if the data is there!
> Chicken and egg problem... Barnyard2 exists to get the information
> from the unified2 output to the database, not to support any
> particular UI.
> I feel like if Barnyard2 says it supports the unified2
> format, it better read everything that it can from that file and get
> it into some output somewhere...
>

Everything is read from unified2, but not everything is logged (currently).
Don't feel like your patch effort is not appreciated.

But to fully enable some features, others part of the code will
get modified and this is where we are heading.

And there is a few improvements comming to some of the core features.

On an other hand, nothing stop anyone arround from creating their output plugin,
for their internal UI. But since some major changes can have an influence on
alot of projects, some coordination will be required.

The beauty of the ML is that people who are interested to fastforward
toward EXTRADATA and can applies your patch and upgrade their schema
and comment on improvements or possible bugs/issue.

Hopefully you wont fell unheard.

-elz

[Brett Edgar]

On Tue, Sep 20, 2011 at 4:20 PM, beenph <bee...@gmail.com> wrote:
> On Tue, Sep 20, 2011 at 4:46 PM, Brett Edgar <brett...@gmail.com> wrote:
>> Apologies, those DROPs were in there for my testing purposes.  That
>> said, if you are running the "create_*sql" script, you better believe
>> it is going to create you a clean database.  It's not going to work
>> for upgrading the schema, anyway, since it would fail at the first
>> "create table" statement for an existing table.  If you are crazy
>> enough to run the create_*sql script, I would hope you know what you
>> are doing.
>>
>
> You do not need to excuse anything it was only a remark.
>

Understand.

>> It is true that no _public_ UIs support the extra data, but they
>> probably will be more inclined to support it if the data is there!
>> Chicken and egg problem... Barnyard2 exists to get the information
>> from the unified2 output to the database, not to support any
>> particular UI.
>> I feel like if Barnyard2 says it supports the unified2
>> format, it better read everything that it can from that file and get
>> it into some output somewhere...
>>
>
> Everything is read from unified2, but not everything is logged (currently).
> Don't feel like your patch effort is not appreciated.

I don't. :)

> But to fully enable some features, others part of the code will
> get modified and this is where we are heading.
>
> And there is a few improvements comming to some of the core features.
>
> On an other hand, nothing stop anyone arround from creating their output plugin,
> for their internal UI. But since some major changes can have an influence on
> alot of projects, some coordination will be required.
>
> The beauty of the ML is that people who are interested to fastforward
>  toward EXTRADATA and can applies your patch and upgrade their schema
> and comment on improvements or possible bugs/issue.
>
> Hopefully you wont fell unheard.

Not at all!

[DigitalX]

Coming in late on this...two questions, is there an upgrade schema we can use, and will any snort frontends read the extra data yet? Thanks.

James

Sent from my iPhone

[Brett Edgar]

On Wed, Sep 21, 2011 at 9:44 AM, DigitalX <digit...@gmail.com> wrote:
> Coming in late on this...two questions, is there an upgrade schema we can use, and will any snort frontends read the extra data yet?  Thanks.

Look at the "create_mysql" script in the patch and add the last table.
You'll also need to modify the entry in the `schema` table to version
108.

There are no frontends that read the extra data yet.

[Jose]

Hello Brett,

I tried to use your patch, but couldn't get any information logged to the extra table ...

I'm using barnyard2-1.9 with your patch, and checked with latest snort tool u2spewfoo that the extra headers are present in the unified2 file.

I tried configuring the database plugin both to "log" and "alert" facilities, without success. Also, the alert_full log file isn't showing the extra info either.

Barnyard logs don't show anything  ... :(

Is there any test I can make to get it working ?

Thank you very much,

Jose.

[Brett Edgar]

Use "log-extra" instead of just "log".

[Jose]

Hello again Brett,

Finally it was "alert-extra", instead of "log-extra". I found it looking at this code (in spo_database.c):

    /* Add the processor function into the function list */
    if (strncasecmp(data->facility, "log", 3) == 0)
    {
        AddFuncToOutputList(Database, OUTPUT_TYPE__LOG, data);
    }
    else
    {
        AddFuncToOutputList(Database, OUTPUT_TYPE__ALERT, data);
    if (strncasecmp(data->facility, "alert-extra", 11) == 0)
        AddFuncToOutputList(Database, OUTPUT_TYPE__EXTRA, data);
    }


Can I just change it this way to have "log-extra" also working ?

    /* Add the processor function into the function list */
    if (strncasecmp(data->facility, "log", 3) == 0)
    {
        AddFuncToOutputList(Database, OUTPUT_TYPE__LOG, data);
    if (strncasecmp(data->facility, "log-extra", 11) == 0)
        AddFuncToOutputList(Database, OUTPUT_TYPE__EXTRA, data);
    }
    else
    {
        AddFuncToOutputList(Database, OUTPUT_TYPE__ALERT, data);
    if (strncasecmp(data->facility, "alert-extra", 11) == 0)
        AddFuncToOutputList(Database, OUTPUT_TYPE__EXTRA, data);
    }


Also, do you know if there is any known difference between both facilities?

Thank you very much for your help and quick response.

Best regards,

Jose.

[Brett Edgar]

That's right. Sorry, I was writing that from memory early in the
morning. You should use alert-extra. The "log-extra" facility
doesn't make sense. Here's why:

In By2, the "log" facility only writes to the database when there is a
Packet structure associated with an alert. That means that for any
event fired by Snort and written to the unified2 file but that does
not have any associated Packet data, the event would not get written
into the database. Using the "alert" facility, *all* events get
written into the database, including Packet data, if it is attached to
the event. I discovered this the hard way when I switched over to By2
and noticed that there were fewer alerts in it versus my old Barnyard
database.

-Brett

[psy phii]

Greetings!

I have a question regarding enabling only alert-extra or log-extra as output plugins.  Basically  I am using FLoP in order to be able to extract full pcap's from the database and need to keep this functionality ( especially concerning tagged packets), therefore I wish to enable unified2 simply to get the extra_data events *only* to be added to my DB.  I noticed in the patch the following line:
data->shared->cid - 1, //"extra" data is always for the previous event
 
So if I understand this correctly the cid for extra_data events is always set to the cid of the previous event by2 inserted into the DB, so in the case of only having *-extra enabled in by2 the event cid's would always be set to the previous extra_data event logged?

[Jose Vila]

Hi all,

We've been using without any remarkable problem Brett's patch to barnyard, and also wrote a small and simple patch to get snort extra headers shown in BASE. The patch only changes the alert detailed view, adding space for the extra headers. No other functionality has been changed.

We don't take any responsibility of possible problems using this patch, but encourage you to use it and report back to us any further questions or updates you have.

You can find it as an update to this blog post (a picture showing the patch is also included there):

http://www.securityartwork.es/2012/05/18/snort-la-uri-en-la-alerta-ante-una-respuesta/

Kind regards,

Jose Vila.

[Jose Luis SPaNKeR]

Hi!

Is this patch compatible with the last 2.10 release?

Thanks!

[psy phii]

It was failing to apply with normal patch options like patch -Np# however adding a healthy fuzz (2048+ lines) caused all hunks except one to succeed.

The failing hunk was the patch to the schema version declaration which has moved to a header (.h) file.

With that said I still ran into an issue when building that may be related to the version of GCC I was using but Im not sure.....

Alternatively If the extra-data output is supported via syslog or alert-full etc it may be possible to get the events into the snortdb via sagan.

[beenph]

On Wed, Dec 5, 2012 at 6:24 AM, psy phii <cyber.t...@gmail.com> wrote:
>
> It was failing to apply with normal patch options like patch -Np# however adding a healthy fuzz (2048+ lines) caused all hunks except one to succeed.
>
> The failing hunk was the patch to the schema version declaration which has moved to a header (.h) file.
>
> With that said I still ran into an issue when building that may be related to the version of GCC I was using but Im not sure.....
>
> Alternatively If the extra-data output is supported via syslog or alert-full etc it may be possible to get the events into the snortdb via sagan.
>
>

That patch will not work against 2-1.10 or 2-1.11 becuase of the obvious plugin code changes.

 

So even if you would apply changes by hands it would not work.

 

The port your have to respect the way the output plugin now works.

 

-elz