Issue with snort.u2 data not loading into my snort mysql database.

[K Kammer]

Hello to all;

First, sorry for the length. I just wanted to give everyone as much info as possible and try not to waste anyone’s time.

 

I’m having an issue where I can’t get my mysql database loaded with data from my snort.u2.timestamp file.

 

Details of what’s running:

Ubuntu 12.04.1

snort Version 2.9.3 IPv6 GRE (Build 37),

Barnyard2 reports Version 2.1.10 (Build 310)

Interfacing it all with SnortReports 1.3.3

Mysql - Server version: 5.5.24

 

I start snort with this:

sudo /usr/local/snort/bin/snort -u snort -g snort –c /usr/local/snort/etc/snort.conf -i eth1

 

Snort starts and runs with no problems.

I’m using unified2 files. I see the ‘snort.u2.timestamp’ file(s) incrementing in size as snort runs, as well as a file named ‘alert’. Also looks like barnyard2.waldo gets updated accordingly. I have a local rule for ICMP to generate activity for testing.

 

I kickoff barnyard2 with this:

sudo /usr/local/bin/barnyard2 -c /usr/local/snort/etc/barnyard2.conf -G /usr/local/snort/etc/gen-msg.map -S /usr/local/snort/etc/sid-msg.map -d /var/log/snort -f snort.u2 -w /var/log/snort/barnyard2.waldo

 

Output line in barnyard2.conf is this:

output database: log, mysql, user=snort password=xxxxxxxxx dbname=snort host=localhost

(I have also tried alert, “output database: alert, mysql, etc”)

 

During startup I get these “dup signature” warning messages. I get 23969 of them (signature id goes to 23969 on last warning):

 

WARNING: While processing data parsed from SIGNATURE FILE a duplicate entry was found [DUPLICATE ARE NOT PROCESSED]:

        Generator ID:[1]        Signature ID:[660]      Revision:[0] Classification ID:[0]

        Message  [SERVER-MAIL expn root]

WARNING: While processing data parsed from SIGNATURE FILE a duplicate entry was found [DUPLICATE ARE NOT PROCESSED]:

        Generator ID:[1]        Signature ID:[661]      Revision:[0] Classification ID:[0]

 

I assume there is some duplicity in my sig files but I’m not sure if this is a problem because it says (DUP ARE NOT PROCESSED) and it says they are warnings.

 

Here are more by2 startup messages:

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM data WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM event WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM icmphdr WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM iphdr WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM opt WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM tcphdr WHERE sid='1';]

database: [SynchronizeEventId()]: Problems executing [SELECT MAX(cid) FROM udphdr WHERE sid='1';]

[SignatureReferencePullDataStore()]: No Reference found in database ...

database: compiled support for (mysql)

database: configured to use mysql

database: schema version = 107

database:           host = localhost

database:           user = snort

database:  database name = snort

database:    sensor name = localhost:eth1

database:      sensor id = 1

database:     sensor cid = 1

database:  data encoding = hex

database:   detail level = full

database:     ignore_bpf = no

database: using the "log" facility

 

by2 messages showing it using waldo:

 

        --== Initialization Complete ==--

 

  ______   -*> Barnyard2 <*-

 / ,,_  \  Version 2.1.10 (Build 310)

 |o"  )~|  By Ian Firns (SecurixLive): http://www.securixlive.com/

 + '''' +  (C) Copyright 2008-2012 Ian Firns <>

 

Using waldo file '/var/log/snort/barnyard2.waldo':

    spool directory = /var/log/snort

    spool filebase  = snort.u2

    time_stamp      = 1349268804

    record_idx      = 4099

Opened spool file '/var/log/snort/snort.u2.1349268804'

Waiting for new data

 

Then, as snort continues to run I get these messages from by2:

 

Closing spool file '/var/log/snort/snort.u2.1349268804'. Read 4099 records

Opened spool file '/var/log/snort/snort.u2.1349269934'

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x9486188]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x956c1b0]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x94e6330]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x9470860]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0xaa95578]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0xa908fc0]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0xa58c4f0]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x9cb8220]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x98f2848]

WARNING database [Database()]: Called with Event[0x0] Event Type [0] (P)acket [0x952a440]

 

I get quite a few of these warnings. (one for every record snort puts into the ‘snort.u2.timestamp’ file and by2 tries to process???) I’m guessing that by2 is trying to process the snort.u2 file into the snort database but can’t (for some reason) thus giving me this WARNING message.

 

Things done up to now:

I’ve double-check all mysql password, grants and permissions and all seem ok.

Deleted and recreated and reloaded the snort database.

 

Here’s my snort tables:

mysql> show tables;

+------------------+

| Tables_in_snort  |

+------------------+

| data             |

| detail           |

| encoding         |

| event            |

| icmphdr          |

| iphdr            |

| opt              |

| reference        |

| reference_system |

| schema           |

| sensor           |

| sig_class        |

| sig_reference    |

| signature        |

| tcphdr           |

| udphdr           |

+------------------+

 

Found out ‘event’ is supposed to look like this:

mysql select * from event;

+-----+-----+-----------+---------------------+

| sid | cid | signature | timestamp           |

+-----+-----+-----------+---------------------+

|   1 |   1 |         1 | 2004-11-06 03:24:51 |

|   1 |   2 |         1 | 2004-11-06 03:24:57 |

|   1 |   3 |         2 | 2004-11-06 03:32:41 |

|   1 |   4 |         2 | 2004-11-06 03:32:47 |

|   1 |   5 |         3 | 2004-11-06 03:33:29 |

|   1 |   6 |         3 | 2004-11-06 03:33:35 |

|   1 |   7 |         4 | 2004-11-06 03:33:41 |

|   1 |   8 |         4 | 2004-11-06 03:33:47 |

|   1 |   9 |         5 | 2004-11-06 03:34:53 |

|   1 |  10 |         5 | 2004-11-06 03:34:59 |

|   1 |  11 |         6 | 2004-11-06 03:35:05 |

|   1 |  12 |         6 | 2004-11-06 03:35:11 |

|   1 |  13 |         7 | 2004-11-06 03:35:17 |

|   1 |  14 |         7 | 2004-11-06 03:35:23 |

|   1 |  15 |         7 | 2004-11-06 03:37:42 |

+-----+-----+-----------+---------------------+

 

But mine is empty and looks like this:

mysql> select * from event;

Empty set (0.00 sec)

 

Here is my sensor table: (I have just one sensor at the time)

mysql> select * from sensor;

+-----+----------------+-----------+--------+--------+----------+----------+

| sid | hostname       | interface | filter | detail | encoding | last_cid |

+-----+----------------+-----------+--------+--------+----------+----------+

|   1 | localhost:eth1 | eth1      | NULL   |      1 |        0 |        0 |

+-----+----------------+-----------+--------+--------+----------+----------+

1 row in set (0.00 sec)

 

Also, re-installed barnyard2 after re-downloading, thinking something was corrupt.

 

Thanks to anyone for your time and assistance.  -Kevin

[beenph]

On Wed, Oct 3, 2012 at 11:46 AM, K Kammer <kj.t...@gmail.com> wrote:
> Hello to all;


>
> First, sorry for the length. I just wanted to give everyone as much info as
> possible and try not to waste anyone’s time.
>
>

Greetings Kevin,

how is your snort.conf " output unified2 directive configured?"

If it output unified2, output log_unified2 or output alert_unified2?

It should be set at output unified2.

If it wasent, make sure you delete old unified2 file before restarting
barnyard2 else you will see
the same issue until new files are generated.

>
> During startup I get these “dup signature” warning messages. I get 23969 of
> them (signature id goes to 23969 on last warning):
>
>
>
> WARNING: While processing data parsed from SIGNATURE FILE a duplicate entry
> was found [DUPLICATE ARE NOT PROCESSED]:
>
> Generator ID:[1] Signature ID:[660] Revision:[0]
> Classification ID:[0]
>
> Message [SERVER-MAIL expn root]
>
> WARNING: While processing data parsed from SIGNATURE FILE a duplicate entry
> was found [DUPLICATE ARE NOT PROCESSED]:
>
> Generator ID:[1] Signature ID:[661] Revision:[0]
> Classification ID:[0]
>
>
>
> I assume there is some duplicity in my sig files but I’m not sure if this is
> a problem because it says (DUP ARE NOT PROCESSED) and it says they are
> warnings.
>
>

Well it not a "problem" but if you fix those files, then you will get
the messages :)

If you have a defined ruleset i would suggest that you use PulledPork
to generate those files

-elz

[K Kammer]

Hello Bob!

At first I was using 'output unified2', but I didn't get data in snort.u2 files. Saw where someone suggested trying  'log_unified2', so I did too and then I started to receive data in "u2".

I must have fixed some other things along the way, because I back to as you said, to use  'output unified2', and I am still receiving data in 'u2' when snort runs. I no longer have an "alert" dataset though.

So, 

-made the change back to output unified2

-removed everything from /var/log/snort (deleted all snort.u2 files, the "alert" file,  deleted and then recreated barnyard2.waldo) everything looks good at this point.

-also clear out the tables in the snort database

-start by2 like before and I still get the SIG FILE "duplicate entry"  warnings

-by2 starts to process the snort.u2 file and I get this message:

10/03-12:01:57.632391  [**] [1:10000001:0] Snort Alert [1:10000001:0] [**] [Classification ID: (null)] [Priority ID: 0] {ICMP} x.x.x.x -> x.x.x.x

WARNING [dbProcessSignatureInformation()]: [Event: 1] with [gid: 1] [sid: 10000001] [rev: 0] [classification: 0] [priority: 0]

         was not found in barnyard2 signature cache, this could lead to display inconsistency.

         To prevent this warning, make sure that your sid-msg.map and gen-msg.map file are up to date with the snort process logging to the spool fil.

         The new inserted signature will not have its information present in the sig_reference table.

         Note that the message inserted in the signature table will be snort default message "Snort Alert [gid:sid:revision]"

         You can allways update the message via a SQL query if you want it to be displayed correctly by your favorite interface

barnyard2: spo_database.c:1665: dbProcessSignatureInformation: Assertion `data->mc.cacheSignatureHead->obj.db_id != 0' failed.

I'm assumeing it something with the sid and gen-msg.map files (like the warning says).

Here's sort-of a side question... What's the difference between updating rules and up to date sid- and gen-msg.map files (like in the message)? What's the relationship between them?

I understand what the rules are (as new vulnerabilities are found new rules are written (or new rule-sets are downloaded) to address those vulnerabilities).

But I'm not sure what the sid and gen-msg.maps are?

Last sentence in the warning says: "You can always update the message via a SQL query if you want it to be displayed correctly by your favorite interface." What would this syntax look like???

I'll concentrate on fixing the sid and gen files like you mentioned and drop in PulledPork.

I just wanted to give you an update, Bob, and not leave you wondering. Thank for your time and input and I'll update later.

-Kevin

[beenph]

On Wed, Oct 3, 2012 at 3:02 PM, K Kammer <kj.t...@gmail.com> wrote:
> Hello Bob!
>
> At first I was using 'output unified2', but I didn't get data in snort.u2
> files. Saw where someone suggested trying 'log_unified2', so I did too and
> then I started to receive data in "u2".
> I must have fixed some other things along the way, because I back to as you
> said, to use 'output unified2', and I am still receiving data in 'u2' when
> snort runs. I no longer have an "alert" dataset though.
>
> So,
> -made the change back to output unified2
> -removed everything from /var/log/snort (deleted all snort.u2 files, the
> "alert" file, deleted and then recreated barnyard2.waldo) everything looks
> good at this point.
> -also clear out the tables in the snort database

Good stuff.

Barnyard2 does not parse rule files, it parses sid-msg.map file,
gen-msg.map file and
classification.config file and reference.config

sid-msg.map file contain signature ID, signature message and reference
information
gen-msg.map file contain preprocessor and SO objects generator id,
generator sid and message
classification.config contain different leterals for known rule classification.
reference.config contains information about reference literals.

When you update your rules files and restart snort, it will generate
events containing those signature information,
those information are written in the unified2 file and barnyard2
process those file, those information are then
cross referenced with those file to fill tables if some information is missing.


Now i am a bit confused why your hitting the assert, could you send me
your barnyard2.conf (minus your db password), your sid-msg.map file
your gen-msg.map file your classification.config file and your
reference.config file?

>
> Last sentence in the warning says: "You can always update the message via a
> SQL query if you want it to be displayed correctly by your favorite
> interface." What would this syntax look like???
>

You would need to connect to your database and then issue a query like
the following:

UPDATE signature set sig_name="MESSAGE IN TABLE" WHERE sig_gid="XXX"
sig_sid="XXX" sig_rev="XXX" sig_class_id="XXX";

XXX being information you can get from the message bring printed

Or you can search for uninitialized signature in

SELECT * FROM signature where sig_msg LIKE "%Snort Alert%";

But if you have an up to date sid-msg.map and gen-msg.map file being
loaded by barnyard2 you shouldn't get those messages.

> I'll concentrate on fixing the sid and gen files like you mentioned and drop
> in PulledPork.
>
> I just wanted to give you an update, Bob, and not leave you wondering. Thank
> for your time and input and I'll update later.
>

No problems, its appreciated, also if others experiences similar
problems, they can
reference to the thread.

I just hope this answer some of your questions.


-elz

[beenph]

Made a typo

SELECT * FROM signature where sig_msg LIKE "%Snort Alert%";

SHOULD be

SELECT * FROM signature where sig_name LIKE "%Snort Alert%";

-elz

[K Kammer]

Hi Eric.

I need to attend to some other things and will send the files first thing in the morning.

I did download and update snortrules *2931 from snort.org but got the same results, (previously I was using snapshot 2930). 

I will also try the "SELECT" command you offered.

I will also send the instruction set which I am following for the install.

Again, thanks for all your help and have a good night.

[beenph]

On Wed, Oct 3, 2012 at 4:00 PM, K Kammer <kj.t...@gmail.com> wrote:
> Hi Eric.
> I need to attend to some other things and will send the files first thing in
> the morning.
> I did download and update snortrules *2931 from snort.org but got the same
> results, (previously I was using snapshot 2930).
> I will also try the "SELECT" command you offered.
> I will also send the instruction set which I am following for the install.
>

Well only updating your rules files for snort will not necessarly mean
by2 point to the good files

Also you have to make sure that you have associative file defined in
barnyard2.conf

config reference_file: PATHTOFILE/reference.config
config classification_file: PATHTOFILE/classification.config
config gen_file: PATHTOFILE/gen-msg.map
config sid_file: PATHTOFILE/sid-msg.map

And verify that those files are the one that came with your rules
package / generated by pulled pork.


-elz

[K Kammer]

**Test Post** I keep getting "An error (#357) occurred while communicating with the server."  when trying to submit a follow-up post to this group. -Kevin                   

[K Kammer]

It worked, does Google Groups have any size restrictions?

[K Kammer]

Hellooo Eric;

Before I go through and install PulledPork I wanted to get to you the files (plus a couple extra) that you requested. The "PATHTOFILE"'s are correct in by2.conf in reference to your last note. I've also run the mysql SELECT but it's empty.

So here the list of files I'm sending:

-barnyard2.conf

-sid-msg.map

-gen-msg.map

-classification.config

-reference.config 

-DB OUTPUT from SELECT from sig-class table.txt

-DB OUTPUT from SELECT from signature table.txt

-DB OUTPUT from SELECT from signature where sig-name like snort-alert RESULT.txt

-DB OUTPUT from SELECT from sig-reference table.txt

-list of usr-local-snort-rules.txt

-open-test.conf

-snort.conf

-snortinstallguide293.pdf

The 'list of usr-local-snort-rules.txt' file is simply a "ls -la" of /usr/local/snort/rules/ directory.

The 'snortinstallguide293.pdf' is the install guide I'm following (so you know what you're up against).  :)

The "DB OUTPUT" files are results of mysql query's. One is the SELECT stmt you suggested and the other three are simple, displays of some tables. Question... Why, in the SIGNATURE table, does the "sig_id" start with 949?  Are "sig's" getting loaded, only I have a partial table? And maybe along the same line... Why does sid-msg.map start with 103?

More than you "signed-on" for I'm sure but once again thanks for your input and time. -Kevin

*Note* Might have to send files in a couple of posts... keep getting errors.

[K Kammer]

[beenph]

On Thu, Oct 4, 2012 at 11:35 AM, K Kammer <kj.t...@gmail.com> wrote:
> Hellooo Eric;
>
> Before I go through and install PulledPork I wanted to get to you the files
> (plus a couple extra) that you requested. The "PATHTOFILE"'s are correct in
> by2.conf in reference to your last note. I've also run the mysql SELECT but
> it's empty.

Ok, i think might have figured this out.

Can you remove your -G and -S Command line arguments and rerun the process?

So your command line should look like that (Striped from initial post)
sudo /usr/local/bin/barnyard2 -c /usr/local/snort/etc/barnyard2.conf

-d /var/log/snort -f snort.u2 -w /var/log/snort/barnyard2.waldo

As of google group limit, next time mabey you would like to put them
in a tarball or zip file :)


Let us know how this goes.

-elz

[Soledad Soledad]

Hi all, I've met the same problem with by2. I install Snort and by2 on Fedora 17, snort 2.9.3.1 and by2 version 2.1.10 (Build 310). Please help me! Thanks

[beenph]

Did you create a test rule?

If so, make sure that your test rule as a revision argument : rev:XXX;
in the rule body where XXX is greater or equal to 1.

Also clean your previously created unified2 file and it should be fine.

-elz

[Soledad Soledad]

I created simple test rule:
          alert icmp any any <> any any (msg:"ICMP"; sid: 1000001;)
and tried to clean everything as your comment on this subject (u2 files, snort's log files, waldo...) but no luck. This problem doesn't occur in snort ver 2.9.2.1 with by2 version 2.1.10-beta2 (Build 266).
Thanks for reply!
-Luong-

[beenph]

On Tue, Oct 16, 2012 at 4:14 AM, Soledad Soledad <not.s...@gmail.com> wrote:
> I created simple test rule:
> alert icmp any any <> any any (msg:"ICMP"; sid: 1000001;)

As previously said in my message:
<SNIP>

Did you create a test rule?
If so, make sure that your test rule as a revision argument : rev:XXX;
in the rule body where XXX is greater or equal to 1.

</SNIP>

So you rule should look like the following:

alert icmp any any <> any any (msg:"ICMP"; sid: 1000001; rev:1;)

And it should work without a problem.

-elz

> and tried to clean everything as your comment on this subject (u2 files,
> snort's log files, waldo...) but no luck. This problem doesn't occur in
> snort ver 2.9.2.1 with by2 version 2.1.10-beta2 (Build 266).
> Thanks for reply!
> -Luong-
>

> --
>
>
>

[Soledad Soledad]

Thanks Lauzon. It was ok now

[K Kammer]

Hey Eric:

A very, very late update, but did want to get back to you.

I did start SNORT as suggested:

>Can you remove your -G and -S Command line arguments and rerun the process? 

This did get rid of the “SIG FILE "duplicate entry” warnings.

Thank you for that.

 

I still get the:

“barnyard2: spo_database.c:1665: dbProcessSignatureInformation: Assertion `data->mc.cacheSignatureHead->obj.db_id != 0' failed.” error.

 

Moving forward update:

I was able to find, (scavenge), a better server (core 2 duo) and was able to build everything under 64bit. This system is working great. I must have “dinked” something up in the other build, so I’m abandoning that one.

I just want to thank you for all your help and assistance.

It truly is/was appreciated.

-Kevin

[beenph]

On Mon, Oct 22, 2012 at 3:26 PM, K Kammer <kj.t...@gmail.com> wrote:
> Hey Eric:
>
> A very, very late update, but did want to get back to you.
>
> I did start SNORT as suggested:
>
>>Can you remove your -G and -S Command line arguments and rerun the process?
>
> This did get rid of the “SIG FILE "duplicate entry” warnings.
>
> Thank you for that.
>
>

Np.

>
> I still get the:
>
> “barnyard2: spo_database.c:1665: dbProcessSignatureInformation: Assertion
> `data->mc.cacheSignatureHead->obj.db_id != 0' failed.” error.
>
>

Hey Kevin,

well thats probably because you didin't delete the unified2 file
containing event with revision of 0.

Since if i read below and see that you installed on a new system (i
assume you gave revision to your test rules)
and thus the unified2 file where not having events with revision 0 and
barnyard2 was happy.

glad you got it work in the end!

-elz

[K Kammer]

I did delete all unified2 files, deleted and recreated waldo, and clear all the entries/info that were in the database tables of the snort database.

Kevin

[sumit kamboj]

Hello Eric

I also having the same problem with snort 2.9.4, barnyard2 v2-1.11. My snort logs are full of alerts/logs but barnyard always read it as 0 records.
I had done all you suggest here till now but does not work. Please help

Note-Just for first time when i start barnyard it logged the events to the database after that it stopped working
.

[beenph]

On Thu, Jan 24, 2013 at 3:16 PM, sumit kamboj <sumitk...@gmail.com> wrote:
> Hello Eric
>
> I also having the same problem with snort 2.9.4, barnyard2 v2-1.11. My snort
> logs are full of alerts/logs but barnyard always read it as 0 records.
> I had done all you suggest here till now but does not work. Please help
>
> Note-Just for first time when i start barnyard it logged the events to the
> database after that it stopped working
> .
>

You snort logs?
What is your snort output configuration?
Feed in some details so we can help out.