Barnyard v2-1.13 released.

[firnsy]

G'day All,

We are happy to announce the latest STABLE release v2.1-13 which was tagged a few hours ago (https://github.com/firnsy/barnyard2/tags)

This release is a bug fix release that also introduce a few new features and enhancements.

UPGRADE REQUIREMENTS

If you are upgrading to barnyard2 2-1.13 (build 327) or above from a previous version and using output database.

You will need to delete every row in your sig_reference table. (DELETE FROM sig_reference;)

The table will be re-populated at startup, and has no impact on historical data.

FEATURE REQUESTS

• Phil Daws - add interface and hostname field to spo_alert_csv if specified.
• Jorge Pinto - spo_syslog_full support for ASCII,BASE64 payload
• Jason Brvenik - variables ... (a long time ago, sorry :P)
• Martin Olsson - remove some useless verbosity unless ./configure --enable-debug is specified and proper flag are used (spo_database and sid-msg.mapv2)
• All other barnyard2 users who help and contribute.
  

BUG REPORTS

• Martin Olsson - bug in sig_reference generation and good discussions. Rewrote the code & al
• John Eure and others - autogen.sh could cause some issue on some system so [autoreconf -fv --install] is not set to autoreconf -fvi
• John Naggets - spo_database: could stop barnyard2 from processing new event if some packets with ip option where processed and option_len was null.
• Fäbu Hufi - spo_syslog_full: in complete mode was printing wrong ip version information and ip header length.
• Jeremy Hoel - identified issue with suppression range in 2-1.13-BETA (fixed in release)
• Bill Green - identified is with signature insertion mainly preprocessor in 2-1.13-BETA (fixed in release) 
• All other barnyard2 users who help and contribute.

NEW FEATURES

1. Support for sid-msg.map version 2 format.

A new sig-msg.map format can be generated by pulledpok (upcomming release, already in svn).

Detection of sid-msg.map version is done by a simple header in the file that shouldn't be altered if you want it to be processed correctly.

The sig-msg.map version 2 format extends the information already present in the sid-msg.map file created from rules.

This new format version allow signature pre-population if users are using output database method with barnyard2 2-1.13 and above.

sid-msg.map v1 format:

SID || MSG || REF 1 || REF N

sid := integer

msg := string

ref := string

sid-msg.map v2 format:

GID || SID || REV || CLASSIFICATION || PRIORITY || MSG || REF 1 || REF N

gid := integer

sid := integer

rev := integer

classification := string (if NULL set to NOCLASS)

priority := integer (if prio == 0, classification priority is used)

msg := string

ref := string

=====================

generator (GID, gen-msg.map) are defaulted to the following value

if their information is not overruled in sid-msg.map v2 file via processing of preprocessor.rules:

revision 1

classification 0

priority 3 

If generator message is present in the sid-msg.map v2 file, and gen-msg.map message are longer 

(more comprehensive by string length), 

gen-msg.map messages are used instead of sid-msg.map v2 file generator messages.

=====================

2. Signature/event logging suppression at spooler level.

Read doc/README.sig_suppression

3. Configuration file variables.

You can now use [var VARNAME value] in the barnyard2 configuration file and every instance of $VARNAME will get replaced by value.

Note that variable declaration order is important only you include a variable with in a variable.

 

 EX (is VALID): 

 var INTERFACE ethX

 var PATH /var/log/IDS

 var LOG $PATH/$INTERFACE/log

 var ARCHIVE $PATH/$INTERFACE/archive

 

 EX (is INVALID): 

 var LOG $PATH/$INTERFACE/log

 var ARCHIVE $PATH/$INTERFACE/archive

 var INTERFACE ethX

 var PATH /var/log/IDS

4. New output database configuration keyword.

Keywords connection_limit and reconnect_sleep_time where added in 2-1.10 but where "undocumented" and shouldn't be modified unless you encounter an issue.

  connection_limit <integer>: default 10

The maximum number of time that barnyard2 will tolerate a transaction faillure and or database connection failure.

  reconnect_sleep_time <integer> : default 5

The number of seconds to sleep betwen connection retry.

  disable_signature_reference_table

Tell the output plugin not to synchronize the sig_reference table in the schema.

Note: This option will speedup the process, especialy if you use sid-msg.mapv2 file or have alot of signature already in databases. (Make sure that you do not need that information before enabling this)

So we hope you enjoy the new release, as a side note the RELEASE.NOTES file has not been updated and will be removed in the next version. It's honestly the most laborious part of release time ;)

Regards,

The barnyard2 team. 

[Starner, Mark]

I downloaded the new 2.1.13 this afternoon.

Compiled it on my i686 and x86_64 architecture machines.

 

1)      Stopped the old barnyard (2.1.11) on all the sensors (7 of them) talking to my QA Database.

2)      Did the DELETE FROM sig_reference; command to clear the table in the MySQL DB.

3)      Then installed the new barnyard 2.1.13 onto each sensor and started it up.

 

On some sensors it worked fine, on others I got these errors and barnyard2 exited:

May 14 19:42:07 xxxxxxx barnyard2: ERROR database: Returned signature_id [4597] is not equal to updated signature_id [4838] in [dbSignatureInformationUpdate()] 

May 14 19:42:07 xxxxxxx barnyard2: [dbProcessSignatureInformation()] Line[1556], call to dbSignatureInformationUpdate failed for :  [gid :119] [sid: 32] [upd_rev: 1] [upd class: 18] [upd pri 3]

May 14 19:42:07 xxxxxxx barnyard2: FATAL ERROR: [dbProcessSignatureInformation()]: Failed, stoping processing 

 

Not sure what this is telling me.

 

The sig_reference table has been repopulated with data.

And some of the barnyard instances seem happy with it….

 

But some of them keep throwing an exactly as above (2 different sensors, all saying the same signature_id (4597) is not equal to signature_id (4838) for gid: 119, sid: 32, upd_rec: 1, etc….

 

One of the other sensors is saying (very similar, just one less signature_Id and sid:31):

May 14 19:49:18 uuuuuu barnyard2: ERROR database: Returned signature_id [4596] is not equal to updated signature_id [4839] in [dbSignatureInformationUpdate()]

May 14 19:49:18 uuuuuu barnyard2: [dbProcessSignatureInformation()] Line[1556], call to dbSignatureInformationUpdate failed for :  [gid :119] [sid: 31] [upd_rev: 1] [upd class: 18] [upd pri 3]

 

Neither sig_id 4597 or sig_id 4838 is in the signature_reference table in this database.

 

So what is it trying to tell me???

--
 
---
You received this message because you are subscribed to the Google Groups "barnyard2-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to barnyard2-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Starner, Mark]

I even tried the new option disable_signature_reference_table to see if that made a difference, it didn’t.

 

output database: alert, mysql, user=xxxxxx password=xxxxxx dbname=snort2 host=xxxxxxx sensor_name=xxxxxxx disable_signature_reference_table

[beenph]

On Tue, May 14, 2013 at 4:16 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> I downloaded the new 2.1.13 this afternoon.
>

<SNIP>
</SNIP>

>
>
>
> Neither sig_id 4597 or sig_id 4838 is in the signature_reference table in
> this database.
>

Hi Mark,

This has nothing to do with the sig_reference table.
The sig_reference table is only used for UI to display reference text
appended to signature
found in sid-msg.map file.

>
> So what is it trying to tell me???
>

Seem's like you might have unclean preprocessors entries in your
signature table, have you ran 2-1.13-BETA on it
before migrating to 2-1.13 release?


1. Are all your sensors using the same databe or are you using
separated database for each sensor?

2. Which database are your using? (PostgreSQL or MySQL)?

3. If you are using MySQL are you using innodb storage engine?
( you can execute the following query on the database or the
database where you have the problematic sensor:
SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE
table_schema=DATABASE(); )

4. could you execute the following query and return the resultset:
SELECT * FROM signature WHERE sig_gid='119';

Thanks you
-elz

[Starner, Mark]

I was running 2.1.11 prior to testing 2.1.13. I never tried the BETA.

Answers:
1) multiple sensors going to a single DB (6 in this case)
2) MySQL
3) innodb storage engine (was MyISAM, but I Altered them into innodb)
4) The query you specified returned 707 lines, I narrowed it down to just
gid:119, sid:32 and got 30 rows returned. (they are at the bottom of my
reply)
So I assume I shouldn't have this many rows for one signature, and only one
of them looks fully populated with all the data, and that is the one that is
referenced by signature_id (4597) in the error message:

May 14 19:42:07 xxxxxxx barnyard2: ERROR database: Returned signature_id
[4597] is not equal to updated signature_id [4838] in
[dbSignatureInformationUpdate()]
May 14 19:42:07 xxxxxxx barnyard2: [dbProcessSignatureInformation()]
Line[1556], call to dbSignatureInformationUpdate failed for : [gid :119]
[sid: 32] [upd_rev: 1] [upd class: 18] [upd pri 3]

One odd thing: We have 32 bit and 64 bit systems as snort sensors. The 64
bit systems are not giving this error message. The 32 bit systems are. Not
sure what that means until I understand whats happening. Both were compiled
from the same 2.1.13 source minutes apart.

Thanks

Here is the SQL output:
mysql> SELECT * FROM signature WHERE sig_gid='119' AND sig_sid='32';
+--------+------------------------------+--------------+--------------+-----
----+---------+---------+
| sig_id | sig_name | sig_class_id | sig_priority |
sig_rev | sig_sid | sig_gid |
+--------+------------------------------+--------------+--------------+-----
----+---------+---------+
| 4350 | http_inspect: SIMPLE REQUEST | 0 | 0 |
0 | 32 | 119 |
| 4597 | http_inspect: SIMPLE REQUEST | 18 | 3 |
1 | 32 | 119 |
| 4838 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 5221 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 5603 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 5985 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 6367 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 6749 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 7131 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 7513 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 7895 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 8277 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 8659 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 9041 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 9423 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 9805 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 10187 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 10569 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 10951 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 11333 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 11715 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 12097 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 12479 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 12861 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 13243 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 13625 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 14007 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 14389 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 14771 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
| 15153 | http_inspect: SIMPLE REQUEST | 0 | 0 |
1 | 32 | 119 |
+--------+------------------------------+--------------+--------------+-----
----+---------+---------+
30 rows in set (0.03 sec)

[beenph]

On Wed, May 15, 2013 at 6:40 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> I was running 2.1.11 prior to testing 2.1.13. I never tried the BETA.
>
> Answers:
> 1) multiple sensors going to a single DB (6 in this case)
> 2) MySQL
> 3) innodb storage engine (was MyISAM, but I Altered them into innodb)

Instead of directly converting mabey it would have been good to filter
out some stuff,
nevertheless ill suggest a possible workarround.

> 4) The query you specified returned 707 lines, I narrowed it down to just
> gid:119, sid:32 and got 30 rows returned. (they are at the bottom of my
> reply)

Yhea i envisaged this situation. Its a multi step operation to fix,
but it is technically its doable.

> So I assume I shouldn't have this many rows for one signature, and only one
> of them looks fully populated with all the data, and that is the one that is
> referenced by signature_id (4597) in the error message:
> May 14 19:42:07 xxxxxxx barnyard2: ERROR database: Returned signature_id
> [4597] is not equal to updated signature_id [4838] in
> [dbSignatureInformationUpdate()]
> May 14 19:42:07 xxxxxxx barnyard2: [dbProcessSignatureInformation()]
> Line[1556], call to dbSignatureInformationUpdate failed for : [gid :119]
> [sid: 32] [upd_rev: 1] [upd class: 18] [upd pri 3]
>

This can be manuall long and technically could be scripted.

So i wrote this little stored proc where you will have to execute it
for each generator id
this should fix your signature table and also fix your event table.

Once every preprocessor has been executed you should be able to
restart your 2-1.13 barnyard2 processes without an issue.

<STOREDPROC MYSQL>
DROP PROCEDURE fixsigs;
delimiter $$
CREATE PROCEDURE fixsigs(IN proc_gid INT,OUT return_val varchar(50))
BEGIN
DECLARE GID_COUNT INT;
DECLARE GID_SID_MIN INT;
DECLARE GID_SID_MAX INT;
DECLARE C_SID INT;
DECLARE cursorGIDSid CURSOR FOR SELECT MIN(sig_sid),MAX(sig_sid) FROM
signature WHERE sig_gid=proc_gid;
DECLARE cursorGIDcount CURSOR FOR SELECT COUNT(sig_sid) FROM signature
WHERE sig_gid=proc_gid GROUP BY sig_gid;
OPEN cursorGIDcount;
FETCH cursorGIDcount INTO GID_COUNT;
IF GID_COUNT > 1 THEN
SET return_val = 'OPERATED';
OPEN cursorGIDSid;
FETCH cursorGIDSid INTO GID_SID_MIN,GID_SID_MAX;
SET return_val = CONCAT(return_val,' MIN:',GID_SID_MIN,' MAX: ',GID_SID_MAX);

SET C_SID = GID_SID_MIN;
WHILE C_SID <= GID_SID_MAX DO
UPDATE event SET signature=(SELECT sig_id FROM signature WHERE
sig_gid=proc_gid AND sig_sid=C_SID AND sig_class_id <> 0 and
sig_priority <> 0) WHERE signature IN (SELECT sig_id FROM signature
WHERE sig_gid=proc_gid AND sig_sid=C_SID AND sig_class_id ='0' and
sig_priority='0');
DELETE FROM signature WHERE sig_gid=proc_gid AND sig_sid=C_SID AND
sig_class_id='0' AND sig_priority='0';

SET C_SID = C_SID + 1;
END WHILE;
CLOSE cursorGIDSid;

ELSE
SET return_val = 'NOTHING TO DO';
END IF;
CLOSE cursorGIDcount;
END$$
delimiter ;
</STOREDPROC MYSQL>

Once the above proc has been saved to your database

you can use it as follow.

binf@SINGULAR:~/SIDV2/test$ grep 111 gen-msg.map
111 || 1 || spp_stream4: Stealth Activity Detected
111 || 2 || spp_stream4: Evasive Reset Packet
111 || 3 || spp_stream4: Retransmission
111 || 4 || spp_stream4: Window Violation
111 || 5 || spp_stream4: Data on SYN Packet
111 || 6 || spp_stream4: Full XMAS Stealth Scan
111 || 7 || spp_stream4: SAPU Stealth Scan
111 || 8 || spp_stream4: FIN Stealth Scan
111 || 9 || spp_stream4: NULL Stealth Scan
111 || 10 || spp_stream4: NMAP XMAS Stealth Scan
111 || 11 || spp_stream4: VECNA Stealth Scan
111 || 12 || spp_stream4: NMAP Fingerprint Stateful Detection
111 || 13 || spp_stream4: SYN FIN Stealth Scan
111 || 14 || spp_stream4: TCP forward overlap detected
111 || 15 || spp_stream4: TTL Evasion attempt
111 || 16 || spp_stream4: Evasive retransmitted data attempt
111 || 17 || spp_stream4: Evasive retransmitted data with the data split attempt
111 || 18 || spp_stream4: Multiple acked
111 || 19 || spp_stream4: Shifting to Emergency Session Mode
111 || 20 || spp_stream4: Shifting to Suspend Mode
111 || 21 || spp_stream4: TCP Timestamp option has value of zero
111 || 22 || spp_stream4: Too many overlapping TCP packets
111 || 23 || spp_stream4: Packet in established TCP stream missing ACK
111 || 24 || spp_stream4: Evasive FIN Packet
111 || 25 || spp_stream4: SYN on established

mysql> call fixsigs(111,@a);
Query OK, 1 row affected (53.39 sec)
mysql> select @a;
+------------------------+
| @a |
+------------------------+
| OPERATED MIN:1 MAX: 25 |
+------------------------+
1 row in set (0.00 sec)

Operation should be done for every preprocessor signature you find in
gen-msg.map where you have duplicated entries and all of them should
be done before you restart barnyard2.

Once your done you can allways use the previous query on the signature
table to see if you have duplicate.

Let us know how it goes.

-elz

[Starner, Mark]

Thanks!

Working on this now...

Two questions..
1) Do I need to do this for generator 1, 2 and 3? Or just the pre-processor
generators?

2) some of the MIN and MAX values are not matching my gen-msg.map. Is this
OK?
For example, generator 140
mysql> call fixsigs(140,@a);
Query OK, 0 rows affected (33.70 sec)

mysql> select @a;
+------------------------+
| @a |
+------------------------+

| OPERATED MIN:1 MAX: 27 |

+------------------------+
1 row in set (0.00 sec)

But my gen-msg.map says:
140 || 1 || sip: Maximum sessions reached
140 || 2 || sip: Empty request URI
140 || 3 || sip: URI is too long
140 || 4 || sip: Empty call-Id
140 || 5 || sip: Call-Id is too long
140 || 6 || sip: CSeq number is too large or negative
140 || 7 || sip: Request name in CSeq is too long
140 || 8 || sip: Empty From header
140 || 9 || sip: From header is too long
140 || 10 || sip: Empty To header
140 || 11 || sip: To header is too long
140 || 12 || sip: Empty Via header
140 || 13 || sip: Via header is too long
140 || 14 || sip: Empty Contact
140 || 15 || sip: Contact is too long
140 || 16 || sip: Content length is too large or negative
140 || 17 || sip: Multiple SIP messages in a packet
140 || 18 || sip: Content length mismatch
140 || 19 || sip: Request name is invalid
140 || 20 || sip: Invite replay attack
140 || 21 || sip: Illegal session information modification
140 || 22 || sip: Response status code is not a 3 digit number
140 || 23 || sip: Empty Content type
140 || 24 || sip: SIP version other than 2.0, 1.0, and 1.1 are invalid
140 || 25 || sip: Mismatch in Method of request and the CSEQ header
140 || 26 || sip: The method is unknown


Thanks!

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Wednesday, May 15, 2013 1:24 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Thu, May 16, 2013 at 8:05 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Thanks!
>
> Working on this now...
>
> Two questions..
> 1) Do I need to do this for generator 1, 2 and 3? Or just the pre-processor
> generators?

1,2 and 3 should be fine unless you have multple row's.

Mabey you have a outdated gen-msg.map file?

grep 140 gen-msg.map
<SNIP>

140 || 21 || sip: Illegal session information modification
140 || 22 || sip: Response status code is not a 3 digit number
140 || 23 || sip: Empty Content type
140 || 24 || sip: SIP version other than 2.0, 1.0, and 1.1 are invalid
140 || 25 || sip: Mismatch in Method of request and the CSEQ header
140 || 26 || sip: The method is unknown

140 || 27 || sip: Maximum dialogs in a session reached
</SNIP>

keep us up to date.
-elz

[Starner, Mark]

Ok... I stopped Barnyard on all 6 sensors, ran fixsigs againast all the
generators except 1, 2 and 3 and restarted barnyard on all 6 sensors.....

They are running and happy.

Can fixsigs be applied to a DB being fed by Barnyard 2.1.11? And will it
stick?

So I can fix my production DB, while still running 2.1.11 and then when I
upgrade to 2.1.13 I won't have to run fixsigs again?

I know I need to upgrade to 2.1.13 all at once for all sensors feeding a
specific DB. No Mixed 2.1.11 and 2.1.13.

Great support!!!!!
Thanks!
Mark

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Thursday, May 16, 2013 9:50 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[Starner, Mark]

Oh, as for my #2, the gen-msg.map I was grepping was an old one since my
Mgmt station doesn't actually run snort itself. As expected the one running
on the sensors matched your output.

Thanks.

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph

Sent: Thursday, May 16, 2013 9:50 AM

To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Thu, May 16, 2013 at 10:03 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Ok... I stopped Barnyard on all 6 sensors, ran fixsigs againast all the
> generators except 1, 2 and 3 and restarted barnyard on all 6 sensors.....
>
> They are running and happy.
>
> Can fixsigs be applied to a DB being fed by Barnyard 2.1.11? And will it
> stick?
>

Can't put a warranty on this. But i think the culprit was 2-1.9 in this case.

> So I can fix my production DB, while still running 2.1.11 and then when I
> upgrade to 2.1.13 I won't have to run fixsigs again?
>

I would run it if you encounter the same issue using the same method.

> I know I need to upgrade to 2.1.13 all at once for all sensors feeding a
> specific DB. No Mixed 2.1.11 and 2.1.13.
>

Well its greatly recommended.


> Great support!!!!!

Np, good to hear it worked!

Cheers,
-elz

[Starner, Mark]

A followup.... this has been running fine now since I performed the fixsigs
procedure and cleaned up all the duplicate entries.

Until this last Friday. Barnyard 2.1.13 on 1 sensor of 6 pointing to one of
my Snort DBs refused to start with:
Jun 10 16:30:19 xxxxxx barnyard2: ERROR database: Returned signature_id
[2976] is not equal to updated signature_id [3635] in
[dbSignatureInformationUpdate()]
Jun 10 16:30:19 xxxxxx barnyard2: [dbProcessSignatureInformation()]
Line[1556], call to dbSignatureInformationUpdate failed for : [gid :125]
[sid: 4] [upd_rev: 1] [upd class: 5] [upd pri 3]
Jun 10 16:30:19 xxxxxx barnyard2: FATAL ERROR:
[dbProcessSignatureInformation()]: Failed, stoping processing

So I looked at this gid:sid and had 430 records:
mysql> SELECT * FROM signature WHERE sig_gid='125' AND sig_sid='4';
+--------+---------------------------------+--------------+--------------+--
-------+---------+---------+

| sig_id | sig_name | sig_class_id | sig_priority |
sig_rev | sig_sid | sig_gid |

+--------+---------------------------------+--------------+--------------+--
-------+---------+---------+
| 1073 | ftp_pp: FTP malformed parameter | 0 | 0 |
0 | 4 | 125 |
| 1313 | ftp_pp: FTP malformed parameter | 0 | 0 |
0 | 4 | 125 |
| 1360 | ftp_pp: FTP malformed parameter | 0 | 0 |
0 | 4 | 125 |
| 1666 | ftp_pp: FTP malformed parameter | 0 | 0 |
0 | 4 | 125 |
| 1831 | ftp_pp: FTP malformed parameter | 0 | 0 |
0 | 4 | 125 |
| 2976 | ftp_pp: FTP malformed parameter | 5 | 3 |
1 | 4 | 125 |
| 3635 | ftp_pp: FTP malformed parameter | 0 | 0 |
1 | 4 | 125 |
| 3704 | ftp_pp: FTP malformed parameter | 0 | 0 |
1 | 4 | 125 |
| 3791 | ftp_pp: FTP malformed parameter | 0 | 0 |
1 | 4 | 125 |
Etc....

All of them had 0 class_od and 0 priority except the 5th one below.

I confirmed ALL of my sensors are running 2.1.13 (and I had cleaned up this
DB before upgrading to 2.1.13.
And all the sensors are running the same version of the rules.

I stopped all 6 barnyard processes and ran the fixsigs to remove all the
duplicates
And then restarted barnyard and it seems to have addressed the problem for
now.

Are there still issues with 2.1.13 and this table?

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of Starner, Mark
Sent: Thursday, May 16, 2013 8:05 AM
To: barnyar...@googlegroups.com
Subject: RE: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

Shouldn't be, when you say i had cleaned up this db before upgrading,
what do you mean?

-elz

[Starner, Mark]

I stopped all barnyard instances (they were all running 2.1.11)
Ran the fixsigs stored procedure you sent me on all the generator ids and
made sure all the duplicates were gone
Then started up barnyard 2.1.13
All was fine with, multiple barnyard restarts, ruleset updates, etc... since
the middle of May.
Then on Friday evening, barnyard would not start on one sensor of six and
when I was able to look at it today, I had lots of duplicates like below
again.

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Tuesday, June 11, 2013 11:19 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Tue, Jun 11, 2013 at 11:23 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> I stopped all barnyard instances (they were all running 2.1.11)
> Ran the fixsigs stored procedure you sent me on all the generator ids and
> made sure all the duplicates were gone
> Then started up barnyard 2.1.13
> All was fine with, multiple barnyard restarts, ruleset updates, etc... since
> the middle of May.
> Then on Friday evening, barnyard would not start on one sensor of six and
> when I was able to look at it today, I had lots of duplicates like below
> again.
>

Is it possible that you forgot gid 125 for some reason, when running fixsigs?

[Starner, Mark]

Nope -- I looked at my script that I ran when I ran fixsigs and 125 was
there... there were also dups in 119.
I made a script that I ran against the DB to be sure I didn't miss any.

Here is an excerpt:
call fixsigs(122,@a);
select @a;
call fixsigs(123,@a);
select @a;
call fixsigs(124,@a);
select @a;
call fixsigs(125,@a);
select @a;
call fixsigs(126,@a);
select @a;
call fixsigs(128,@a);

[beenph]

On Tue, Jun 11, 2013 at 11:34 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Nope -- I looked at my script that I ran when I ran fixsigs and 125 was
> there... there were also dups in 119.
> I made a script that I ran against the DB to be sure I didn't miss any.
>

Are you sure you do not have a older by2 instance that would be
running in the background,
or an instance where your gen-msg.map file would miss some entries?

Could you try this.

Stop all your barnyard2 instances and choose one that you will perform
this test on.

1. run the following query against the signature table

SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;


2. start your choosen instance within a short period of time like 5 to
10 minutes stop it,
restart it. and rerun the query above.


I personally have never noticed preprocessor being repopulated (beside
some occurences in early 2-1.10 developpement process)
and at this point, i have the impression that the "converted" storage
enging MyIASM -> InnoDB
could possibly play a part in this.


Do you think you could create a new database using innoDB by default
and retest the above senario also?


-elz

[Starner, Mark]

I am 100% sure there are no other instances of barnyard talking to this
database, in the background or otherwise.

I compared the gen-msg.map file on all 6 sensors.. the md5 checksums match
on all 6, they are identical.

I will try this when I can get a chance... they are in production, so I need
to submit an outage and get it approved before I can run the test.

I will have to see if there is a way I can do new databases... some of our
other production DB's for snort have hundreds of thousands of events in
them. Starting over may not be possible. I will have to investigate that.

Just for grins, I ran the below SQL on the database I just ran fixsigs on
this morning, and got 418 rows returned. (I didn't stop anything, just ran
the query) -- shouldn't it be 0 rows returned?
One of the rows was:
| 456 | 116 | 1 |

So I ran: SELECT * FROM signature WHERE sig_gid='116' AND sig_sid='456';
And got
mysql> SELECT * FROM signature WHERE sig_gid='116' AND sig_sid='456';
+--------+---------------------------------------------------------+--------
------+--------------+---------+---------+---------+

| sig_id | sig_name |
sig_class_id | sig_priority | sig_rev | sig_sid | sig_gid |

+--------+---------------------------------------------------------+--------
------+--------------+---------+---------+---------+
| 181950 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
| 182502 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
| 182541 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
| 182613 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
| 182853 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
| 183160 | snort_decoder: WARNING: too many IPV6 extension headers |
0 | 0 | 1 | 456 | 116 |
+--------+---------------------------------------------------------+--------
------+--------------+---------+---------+---------+

And this was just cleaned up this morning....

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Tuesday, June 11, 2013 12:45 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Tue, Jun 11, 2013 at 1:05 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> I am 100% sure there are no other instances of barnyard talking to this
> database, in the background or otherwise.
>
> I compared the gen-msg.map file on all 6 sensors.. the md5 checksums match
> on all 6, they are identical.
>
> I will try this when I can get a chance... they are in production, so I need
> to submit an outage and get it approved before I can run the test.
>
> I will have to see if there is a way I can do new databases... some of our
> other production DB's for snort have hundreds of thousands of events in
> them. Starting over may not be possible. I will have to investigate that.
>

Well you can allways backup the data and re-insert it in the new database.
meanwhile your snort process still run, and your unified2 files will grow.

When you restart barnyard2 pointing them to the new database, the data will
get re-inserted.

> Just for grins, I ran the below SQL on the database I just ran fixsigs on
> this morning, and got 418 rows returned. (I didn't stop anything, just ran
> the query) -- shouldn't it be 0 rows returned?

Yup.

Honestly right now im thinking that you need to create a native InnoDB database.

Also i re-read the previously e-mail in the thread and you never
returned the result of the
following query executed on your db.

<SNIP>

SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE
table_schema=DATABASE(); )

</SNIP>

Which process did you use to convert from MyIASM to InnoDB?
-elz

[Starner, Mark]

I was sure I did -- but here it is
mysql> SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE
-> table_schema=DATABASE();
+------------------+--------+
| table_name | engine |
+------------------+--------+
| acid_ag | InnoDB |
| acid_ag_alert | InnoDB |
| acid_event | InnoDB |
| acid_ip_cache | InnoDB |
| base_roles | InnoDB |
| base_users | InnoDB |
| data | InnoDB |
| detail | InnoDB |
| encoding | InnoDB |
| event | InnoDB |
| icmphdr | InnoDB |
| iphdr | InnoDB |
| opt | InnoDB |
| reference | InnoDB |
| reference_system | InnoDB |
| schema | MyISAM |
| sensor | InnoDB |
| sig_class | InnoDB |
| sig_reference | InnoDB |
| signature | InnoDB |
| tcphdr | InnoDB |
| udphdr | InnoDB |
+------------------+--------+

I used the ALTER TABLE command within MySQL.

Guess I need to figure out how to do the backup and reinsert. I have changed
my MySQL default to be InnoDB so anything new I create will be InnoDB.

Anyone here have a script that will back it all up and then put it back once
I recreate the DB?

Thanks
Mark

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Tuesday, June 11, 2013 1:14 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[Starner, Mark]

Ok.. I did the following on my QA database.

Stopped all BY2 instances writing to the DB

Ran: SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;

Ran fixsigs on all the generator IDs that were listed as a result of that
A Note -- oddly, the following would never go away, no matter how
many times I ran fixsigs:
| 5 | 138 | 1 |
| 9 | 125 | 1 |
| 27 | 140 | 1 |
| 58 | 116 | 1 |

Also, there are lots of GID 1 and 3 (and one gid 2) entries that show
up with a count >= 2
I never ran fixsigs on 1,2, or 3.

Then started ONE instance of 2.1.13 barnyard
______ -*> Barnyard2 <*-
/ ,,_ \ Version 2.1.13 (Build 327)
|o" )~| By Ian Firns (SecurixLive): http://www.securixlive.com/
+ '''' + (C) Copyright 2008-2013 Ian Firns
<fir...@securixlive.com>


Waited 15 minutes or so (it was waiting for new data)

Ran: SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;
| 2 | 138 | 1 |
| 3 | 138 | 1 |
| 4 | 138 | 1 |
| 5 | 138 | 1 |
| 6 | 138 | 1 |
| 9 | 125 | 1 |
| 27 | 140 | 1 |
| 58 | 116 | 1 |
| 59 | 116 | 1 |
In addition to all the GID 1 and 3 entries (and the one GID 2 entry)

I will try the same thing with a new DB and point this BY 2 instance to that
DB.
And see what happens.

I really could use help figuring out how to backup and restore this DB.

Thanks!
Mark

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Tuesday, June 11, 2013 12:45 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[Starner, Mark]

I created a brand new snort DB. (InnoDB)

Pointed one sensor at it.

Let it populate everything to start up.

Ran: SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;
It Returned: Empty set (0.00 sec)

Restarted Barnyard
Let it run to waiting for new data
Ran: SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;
It Returned: Empty set (0.00 sec)

I will set a few more sensors to go to it and see if it holds up.

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Tuesday, June 11, 2013 12:45 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Tue, Jun 11, 2013 at 4:41 PM, Starner, Mark <mark.s...@unisys.com> wrote:

Re hi mark, sorry that i couldn't reply in a timely fashion since my
last e-mail but i was
not near any interconnected silicon material.

1. I do not know if this could be the case but did you restart your
database server (mysql) service since you
did the alter table?

if not, you might want to stop all your barnyard2 process,stop
mysql, restart mysql, fix duplicate sigs and then
restart barnyard2.

2. As for backuping you have multiple options depending on alot of factors.

Factor 1. Which UI do you use?
Factor 2. Do you archive your unified2 files?
Factor 3. How mutch responce downtime can you tolerate.

Without knowing any of the above atm, i would do the following:

1. Create a new db and make sure its InnoDB.
2. Copy over the sensor table so each sensor have the same sid from
your previous db.
3. Point your UI to the new database.
4. adjust access so your sensor can connect to the new database.
5. modify all barnyard2 configuration to make them point to the new
database and restart them.

There is many way you can use to copy data over, that new instance and
this is a bit out of scope of this thread but if you want to minimize
downtime
and be able to access your old data just point your UI to the old db,
thats how i would it quickly and without hurt.

Now for copying your data i think you should start a new thread that
could mabey also benefit other users.

Hope this helps for now.
-elz

[Starner, Mark]

1) Yes, restarted mysql

Here is what I have done since yesterday
1) created brand new empty DB (InnoDB engine)
2) pointed one sensor at it and let it populate everything.
3) Then about an hour later pointed another sensor at it
4) this morning when I looked, all looked ok

I then took one of my other databases
1) ran fixsigs on all the generator ids with dups
2) did mysqldump to dump the db to a file
3) dropped the database
4) reloaded the Database from the dump
5) started barnyard on all the 6 sensors
6) waited about 20 minutes, then ran

SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;

+---------+---------+---------+
| sig_sid | sig_gid | sig_rev |
+---------+---------+---------+

| 2 | 138 | 1 |
| 3 | 138 | 1 |
| 4 | 138 | 1 |
| 5 | 138 | 1 |
| 6 | 138 | 1 |
| 9 | 125 | 1 |
| 27 | 140 | 1 |
| 58 | 116 | 1 |
| 59 | 116 | 1 |

Now, the fixsigs never would get rid of these:

| 5 | 138 | 1 |
| 9 | 125 | 1 |
| 27 | 140 | 1 |
| 58 | 116 | 1 |

But now there are more gid:138 entries than there were before and another
116.

Question:
What happens when more than one barnyard process is starting at the same
time on different sensors.

For example:
I run a script that connects to each sensor in order and starts barnyard2.
As soon as barnyard2 returns to the prompt the next sensor in the list is
started, etc...
So we have more than one barnyard instance (on different sensors) starting
at the same time... could that cause an issue?
Since barnyard forks and the parent exits ,while the child continues to do
all the prep and background work, could this cause duplicates to be placed
into the table?
Do I have to wait until barnyard is done initializing and is processing the
unified2 file before starting another instance on another sensor that writes
to the same database?

If so, it would be nice if there was an option that told barnyard2 not to
fork until its starts processing data file (or some mechanism so I can tell
all the prep work is done and it is ok to start another instance somewhere.

[beenph]

On Wed, Jun 12, 2013 at 9:14 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> 1) Yes, restarted mysql
>
> Here is what I have done since yesterday
> 1) created brand new empty DB (InnoDB engine)
> 2) pointed one sensor at it and let it populate everything.
> 3) Then about an hour later pointed another sensor at it
> 4) this morning when I looked, all looked ok
>
> I then took one of my other databases
> 1) ran fixsigs on all the generator ids with dups
> 2) did mysqldump to dump the db to a file
> 3) dropped the database
> 4) reloaded the Database from the dump
> 5) started barnyard on all the 6 sensors
> 6) waited about 20 minutes, then ran
> SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
> sig_sid,sig_gid,sig_rev HAVING count(*) >=2;

Do you have multiple database? Im trying to understand the above and i see
all ok and i see i waited and i get these which is confusing me.

Mabey you could name them with made up name so we can better communicate here.

Something like DB_native_innodb and DB_converted_innodb ..

>
> But now there are more gid:138 entries than there were before and another
> 116.
>
> Question:
> What happens when more than one barnyard process is starting at the same
> time on different sensors.
>

Signature insertion is transaction isolated and barnyard2 instance
racing against
one an other shouldn't cause a problem if you are using InnoDB under MySQL.

> For example:
> I run a script that connects to each sensor in order and starts barnyard2.
> As soon as barnyard2 returns to the prompt the next sensor in the list is
> started, etc...
> So we have more than one barnyard instance (on different sensors) starting
> at the same time... could that cause an issue?

No.

> Since barnyard forks and the parent exits ,while the child continues to do
> all the prep and background work, could this cause duplicates to be placed
> into the table?

It forks because you daemonize the process but the process forking has
no impact on
when happen ex: daemonized process vs normal process.

> Do I have to wait until barnyard is done initializing and is processing the
> unified2 file before starting another instance on another sensor that writes
> to the same database?
>

No.

-elz

[Starner, Mark]

Here is a more documented version.

Here is what I have done since yesterday

1) created brand new empty DB named snort3_natve_innodb
2) pointed one sensor at snort3_natice_innodb and let it populate
everything.
3) Then about an hour later pointed another sensor at snort3_native_innodb
4) this morning when I looked, all looked ok, no duplicates



I then took one of my other databases (snort2_converted_innodb) that has 2
weeks worth of alerts in it
1) ran fixsigs on all the generator ids with dups on snort2_coverted_innodb
2) did mysqldump to dump the snort2_converted_innodb to a file
3) dropped the snort2_converted_innodb database
4) created a new database snort2_native_innodb
5) reloaded the Database from the dump in #2 above into
snort2_native_innodb
6) verified all tables are InnoDB in snort2_native_innodb
7) started barnyard on all the 6 sensors sending to snort2_native_innodb
8) waited about 20 minutes, then ran the following against
snort2_native_innodb:

SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;

It looks like mysqldump command should properly recreate the DB as a with
native innodb tables, since recreates the tables and adds each record.

Are there any settings required on the innodb engine that are not the
defaults to make sure the engine is configured properly????

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Wednesday, June 12, 2013 9:30 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Wed, Jun 12, 2013 at 9:46 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Here is a more documented version.
>
> Here is what I have done since yesterday
> 1) created brand new empty DB named snort3_natve_innodb
> 2) pointed one sensor at snort3_natice_innodb and let it populate
> everything.
> 3) Then about an hour later pointed another sensor at snort3_native_innodb
> 4) this morning when I looked, all looked ok, no duplicates
>
>
>
> I then took one of my other databases (snort2_converted_innodb) that has 2
> weeks worth of alerts in it
> 1) ran fixsigs on all the generator ids with dups on snort2_coverted_innodb
> 2) did mysqldump to dump the snort2_converted_innodb to a file
> 3) dropped the snort2_converted_innodb database
> 4) created a new database snort2_native_innodb
> 5) reloaded the Database from the dump in #2 above into
> snort2_native_innodb
> 6) verified all tables are InnoDB in snort2_native_innodb
> 7) started barnyard on all the 6 sensors sending to snort2_native_innodb
> 8) waited about 20 minutes, then ran the following against
> snort2_native_innodb:
> SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
> sig_sid,sig_gid,sig_rev HAVING count(*) >=2;
>

The stored proc i wrote could have some issue i might have to review
it, since it has
primarly has been something i wrote down for you on the corner.

But i would need more info on duplicates before fixsigs and after
fixsigs to see whats happening.

SELECT * FROM signature WHERE sig_sid='5' AND sig_gid='138';
SELECT * FROM signature WHERE sig_sid='9' AND sig_gid=125';
SELECT * FROM signature WHERE sig_sid='27' AND sig_gid='140';
SELECT * FROM signature WHERE sig_sid='58' AND sig_gid='116';

I would consider that database to be tainted for now thus testing
against it its more or less usefull.

> It looks like mysqldump command should properly recreate the DB as a with
> native innodb tables, since recreates the tables and adds each record.
>

Yup, but see above.

> Are there any settings required on the innodb engine that are not the
> defaults to make sure the engine is configured properly????
>

I am not sure i understand the above.

-elz

[Starner, Mark]

Do you mean testiing against this DB is more or less useless? (you said
usefull)

It almost sounds like I may need to start from scratch and just archive all
the existing data.


I don't have a before handy, but I have the after.... (I'll have to let it
happen again and try to grab some before and afters)
But here is what's left AFTER running fixsigs.

Looking at the code, it only deletes where the class and prio are 0 and in
some of these these fields are populated.
Not sure where the Snort Alert sig name is coming from, unless its not
coming from the pre-population, but I verified that all my sensors have
these defined in their gen-msg.map file.


mysql> SELECT * FROM signature WHERE sig_sid='5' AND sig_gid='138';
+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+

| sig_id | sig_name | sig_class_id |
sig_priority | sig_rev | sig_sid | sig_gid |

+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+
| 483 | Snort Alert [138:5:0] | 24 |
2 | 1 | 5 | 138 |
| 4020 | sensitive_data: sensitive data - eMail addresses | 24 |
2 | 1 | 5 | 138 |
+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+
2 rows in set (0.01 sec)

mysql> SELECT * FROM signature WHERE sig_sid='9' AND sig_gid='125';
+--------+-------------------------------------------------------+----------
----+--------------+---------+---------+---------+

| sig_id | sig_name |
sig_class_id | sig_priority | sig_rev | sig_sid | sig_gid |

+--------+-------------------------------------------------------+----------
----+--------------+---------+---------+---------+
| 50 | ftp_pp: Evasive Telnet command on FTP command channel |
3 | 3 | 1 | 9 | 125 |
| 4585 | Snort Alert [125:9:1] |
5 | 2 | 1 | 9 | 125 |
+--------+-------------------------------------------------------+----------
----+--------------+---------+---------+---------+
2 rows in set (0.01 sec)

mysql> SELECT * FROM signature WHERE sig_sid='27' AND sig_gid='140';
+--------+-------------------------------------------+--------------+-------
-------+---------+---------+---------+

| sig_id | sig_name | sig_class_id |
sig_priority | sig_rev | sig_sid | sig_gid |

+--------+-------------------------------------------+--------------+-------
-------+---------+---------+---------+
| 4612 | sip: Maximum dialogs in a session reached | 5 |
2 | 1 | 27 | 140 |
| 4642 | sip: Maximum dialogs in a session reached | 5 |
2 | 1 | 27 | 140 |
| 4643 | sip: Maximum dialogs in a session reached | 5 |
2 | 1 | 27 | 140 |
+--------+-------------------------------------------+--------------+-------
-------+---------+---------+---------+
3 rows in set (0.01 sec)

mysql> SELECT * FROM signature WHERE sig_sid='58' AND sig_gid='116';
+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+

| sig_id | sig_name | sig_class_id |
sig_priority | sig_rev | sig_sid | sig_gid |

+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+
| 201 | snort_decoder: Experimental TCP options | 3 |
3 | 1 | 58 | 116 |
| 566 | snort_decoder: WARNING: Experimental TCP options | 3 |
3 | 1 | 58 | 116 |
+--------+--------------------------------------------------+--------------+
--------------+---------+---------+---------+
2 rows in set (0.01 sec)

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Wednesday, June 12, 2013 10:04 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

SELECT * FROM signature WHERE sig_sid='9' AND sig_gid='125';

SELECT * FROM signature WHERE sig_sid='27' AND sig_gid='140';
SELECT * FROM signature WHERE sig_sid='58' AND sig_gid='116';

I would consider that database to be tainted for now thus testing
against it its more or less usefull.



> It looks like mysqldump command should properly recreate the DB as a with
> native innodb tables, since recreates the tables and adds each record.
>
Yup, but see above.

> Are there any settings required on the innodb engine that are not the
> defaults to make sure the engine is configured properly????
>
I am not sure i understand the above.

-elz

[beenph]

On Wed, Jun 12, 2013 at 11:27 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Do you mean testiing against this DB is more or less useless? (you said
> usefull)
>
> It almost sounds like I may need to start from scratch and just archive all
> the existing data.
>
>
> I don't have a before handy, but I have the after.... (I'll have to let it
> happen again and try to grab some before and afters)
> But here is what's left AFTER running fixsigs.
>
> Looking at the code, it only deletes where the class and prio are 0 and in
> some of these these fields are populated.
> Not sure where the Snort Alert sig name is coming from, unless its not
> coming from the pre-population, but I verified that all my sensors have
> these defined in their gen-msg.map file.
>
>

Here's what you need to do in your new database.

1. stop all barnyard2 instance logging to the new InnoDB database.

2. run the script found below.

3. restart your barnyard2 intances on the new InnoDB database.

<SCRIPT>
UPDATE signature SET sig_name='sensitive_data: sensitive data - eMail
addresses' WHERE sig_id ='438';
UPDATE event SET sig_id='438' WHERE sig_id='4020';
DELETE FROM signature WHERE sig_id='4020';

UPDATE event SET sig_id='50' WHERE sig_id='4585';
DELETE FROM signature WHERE sig_id='4585';

UPDATE event SET sig_id='4612' WHERE sig_id IN (4642,4643);
DELETE FROM signature WHERE sig_id='4642';
DELETE FROM signature WHERE sig_id='4643';

UPDATE signature SET sig_name='snort_decoder: WARNING: Experimental
TCP options' WHERE sig_id ='201';
UPDATE event SET sig_id='201' WHERE sig_id='566';
DELETE FROM signature WHERE sig_id='566';
</SCRIPT>


-elz

[Starner, Mark]

Thanks... I did that (sig_id didn't exist in event, replaced it with
signature)

Will run it for a while and see if any new duplicates crop up.

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Wednesday, June 12, 2013 11:41 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Wed, Jun 12, 2013 at 12:07 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> Thanks... I did that (sig_id didn't exist in event, replaced it with
> signature)

Sorry for that ;)

>
> Will run it for a while and see if any new duplicates crop up.
>

Ok.

-elz

[Starner, Mark]

Sorry to dredge this old thread up....

I am still running 2.1.13 on all sensors.

______ -*> Barnyard2 <*-
/ ,,_ \ Version 2.1.13 (Build 327)
|o" )~| By Ian Firns (SecurixLive): http://www.securixlive.com/
+ '''' + (C) Copyright 2008-2013 Ian Firns <fir...@securixlive.com>

I have had occasional dups show up in the signature table. Only a few, and
so far, always for GID 1, but various SID's.
I have fixed them with the three lines you sent me last time after stopping
all barnyard processes:
Sample is (of course, all the relevant things have been changed to match the
affected table entry):

UPDATE signature SET sig_name='sensitive_data: sensitive data -
eMail addresses' WHERE sig_id ='438';

UPDATE event SET signature='1353' WHERE signature='1354';
DELETE FROM signature WHERE sig_id='1354';

Three more cropped up today.
mysql> SELECT sig_sid,sig_gid,sig_rev FROM signature GROUP BY
sig_sid,sig_gid,sig_rev HAVING count(*) >=2;

+---------+---------+---------+
| sig_sid | sig_gid | sig_rev |
+---------+---------+---------+

| 25384 | 1 | 2 |
| 25387 | 1 | 2 |
| 27092 | 1 | 1 |

+---------+---------+---------+
3 rows in set (0.01 sec)

mysql> SELECT * FROM signature WHERE sig_sid='25384' AND sig_gid='1';
+--------+------------------------------------------------------------------
-+--------------+--------------+---------+---------+---------+

| sig_id | sig_name
| sig_class_id | sig_priority | sig_rev | sig_sid | sig_gid |

+--------+------------------------------------------------------------------
-+--------------+--------------+---------+---------+---------+
| 1393 | EXPLOIT-KIT Multiple Exploit Kit Payload detection - contacts.exe
| 59 | 1 | 2 | 25384 | 1 |
| 1394 | EXPLOIT-KIT Multiple Exploit Kit Payload detection - contacts.exe
| 59 | 1 | 2 | 25384 | 1 |
+--------+------------------------------------------------------------------
-+--------------+--------------+---------+---------+---------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM signature WHERE sig_sid='25387' AND sig_gid='1';
+--------+-----------------------------------------------------------------+
--------------+--------------+---------+---------+---------+

| sig_id | sig_name |
sig_class_id | sig_priority | sig_rev | sig_sid | sig_gid |

+--------+-----------------------------------------------------------------+
--------------+--------------+---------+---------+---------+
| 1385 | EXPLOIT-KIT Multiple Exploit Kit Payload detection - readme.exe |
59 | 1 | 2 | 25387 | 1 |
| 1386 | EXPLOIT-KIT Multiple Exploit Kit Payload detection - readme.exe |
59 | 1 | 2 | 25387 | 1 |
+--------+-----------------------------------------------------------------+
--------------+--------------+---------+---------+---------+
2 rows in set (0.00 sec)

mysql> SELECT * FROM signature WHERE sig_sid='27092' AND sig_gid='1';
+--------+------------------------------------------------+--------------+--

------------+---------+---------+---------+
| sig_id | sig_name | sig_class_id |
sig_priority | sig_rev | sig_sid | sig_gid |

+--------+------------------------------------------------+--------------+--
------------+---------+---------+---------+
| 1383 | EXPLOIT-KIT Cool/Styx exploit kit landing page | 59 |
1 | 1 | 27092 | 1 |
| 1384 | EXPLOIT-KIT Cool/Styx exploit kit landing page | 59 |
1 | 1 | 27092 | 1 |
+--------+------------------------------------------------+--------------+--
------------+---------+---------+---------+
2 rows in set (0.00 sec)

mysql>

I verified ALL are running the above mentioned version of Barnyard2.

Is there anything else you want me to look at before I clean it up?

Any idea what is making this happen?

Thanks
Mark

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Wednesday, June 12, 2013 12:09 PM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

Hi Mark,
IIRC you converted from MyIASM to InnoDB in production right?

I personally think that this can still cause issue (in your other tables also).

Whenever you have a chance you should move to a native InnoDB database.

Do you have duplicate in your sid-msg.map file, because when i look at
the examples you print, they seem's to all be consecutive (eg: 1st
sig, 2nd sig == previous sig_id+1).

Do you use sig-msg.map v1 or sig-msg.map v2?

Let me know.

-elz

[Starner, Mark]

Originally, yes.... but that is not the case...

All databases were scrapped, new DB's created using InnoDB and then started
over.... so NO, there are no Converted Databases anymore -- and that
definitely helped and I hadn't seen any in quite a long time. Just recently
(over the past week) have these few started showing up.

I use sid_msg.map version 1.

I looked and see no duplicate entries....
Only one line in the sid-msg.map file for this signature
25384 || EXPLOIT-KIT Multiple Exploit Kit Payload detection - contacts.exe
||
url,blog.webroot.com/2011/10/31/outdated-operating-system-this-blackhole-exp
loit-kit-has-you-in-its-sights/ || cve,2012-4681 || cve,2012-1889 ||
cve,2012-1723 || cve,2012-0507 || cve,2012-0188 || cve,2011-3544 ||
cve,2011-2110 || cve,2011-0559 || cve,2010-1885 || cve,2009-0927 ||
cve,2008-2992 || cve,2008-0655 || cve,2007-5659 || cve,2006-0003

Now, when I update rules, There are multiple barnyard's starting at real
close to the same time, so, is it possible that barnyard looks, doesn't see
the entry, but before it can add it, another barnyard process on another
sensor adds it, then the original adds it again? It seems unlikely, since
they are always consecutive, but just shooting in the dark

OR

If the same alert, is seen by two sensors, at almost exactly the same time,
and both barnyard2 processes notice that the alert needs to be added, and
they both add it to the table without realizing another barnyard2 just did
it?

Mark


-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Thursday, July 11, 2013 10:49 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

#2 is more plausible. Even if there is check and delay's.

If you would use sig-msg.map v2 this path would not happen.

-elz

[Starner, Mark]

Ok... what needs to be changed to change to v2?

Does snort need to be told? Barnyard needs to be told?
And then I need the script to create v2? Instead of v1?

Thanks

-----Original Message-----
From: barnyar...@googlegroups.com
[mailto:barnyar...@googlegroups.com] On Behalf Of beenph
Sent: Thursday, July 11, 2013 11:06 AM
To: barnyar...@googlegroups.com
Subject: Re: [barnyard2-users] Barnyard v2-1.13 released.

[beenph]

On Thu, Jul 11, 2013 at 11:09 AM, Starner, Mark <mark.s...@unisys.com> wrote:
> Ok... what needs to be changed to change to v2?
>
> Does snort need to be told? Barnyard needs to be told?
> And then I need the script to create v2? Instead of v1?
>

Pulledpork can generate sid-msg.map v2 files.
Do you use pulledpork?

[Starner, Mark]

Yes -- using pulledpork 0.6.1 but I don't see any *method* to set the
version of the sid-msg.map output.
If there is a new version, its not where I always got pulledpork
http://code.google.com/p/pulledpork/

[beenph]

On Thu, Jul 11, 2013 at 12:03 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> Yes -- using pulledpork 0.6.1 but I don't see any *method* to set the
> version of the sid-msg.map output.
> If there is a new version, its not where I always got pulledpork
> http://code.google.com/p/pulledpork/
>
>

Check http://code.google.com/p/pulledpork/source/browse/trunk/etc/pulledpork.conf

set
sid_msg_version= to 2

ex:
sid_msg_version=2

[Starner, Mark]

Ok... I see this is for version 0.7.0
version=0.7.0

which I assume has not been *released* yet, since the main pulledpork page
still only shows you 0.6.1.

Hopefully coming soon.

[beenph]

On Thu, Jul 11, 2013 at 12:16 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> Ok... I see this is for version 0.7.0
> version=0.7.0
>
> which I assume has not been *released* yet, since the main pulledpork page
> still only shows you 0.6.1.
>

You could use it without and issue.

[Jeremy Hoel]

Syn versions of pulledpork down't build right on CentOS.. just an FYI.
we are at older perl versions and so it won't run without errors.
See issues 127

https://code.google.com/p/pulledpork/issues/detail?id=127

[Lucas Miguel]

Hello,

Can anyone help me?

I'm having the error bellow when compiling the Barnyard2.

make[3]: Entering directory '/home/snort/snort_src/barnyard2-master/src/output-plugins'

gcc -DHAVE_CONFIG_H -I. -I../..  -I.. -I ../sfutil -I/usr/include/mysql -DENABLE_MYSQL  -g -O2 -Wall -c -o spo_database.o spo_database.c

In file included from spo_database.c:103:0:

../output-plugins/spo_database.h:360:5: error: unknown type name ‘my_bool’

     my_bool mysql_reconnect; /* We will handle it via the api. */

     ^~~~~~~

Makefile:391: recipe for target 'spo_database.o' failed

make[3]: *** [spo_database.o] Error 1

make[3]: Leaving directory '/home/snort/snort_src/barnyard2-master/src/output-plugins'

Makefile:498: recipe for target 'all-recursive' failed

make[2]: *** [all-recursive] Error 1

make[2]: Leaving directory '/home/snort/snort_src/barnyard2-master/src'

Makefile:412: recipe for target 'all-recursive' failed

make[1]: *** [all-recursive] Error 1

make[1]: Leaving directory '/home/snort/snort_src/barnyard2-master'

Makefile:344: recipe for target 'all' failed

make: *** [all] Error 2

Regards,

LM

terça-feira, 14 de Maio de 2013 às 12:52:38 UTC+1, firnsy escreveu:

G'day All,

We are happy to announce the latest STABLE release v2.1-13 which was tagged a few hours ago (https://github.com/firnsy/barnyard2/tags)

This release is a bug fix release that also introduce a few new features and enhancements.

UPGRADE REQUIREMENTS

If you are upgrading to barnyard2 2-1.13 (build 327) or above from a previous version and using output database.

You will need to delete every row in your sig_reference table. (DELETE FROM sig_reference;)

The table will be re-populated at startup, and has no impact on historical data.

FEATURE REQUESTS

• Phil Daws - add interface and hostname field to spo_alert_csv if specified.
• Jorge Pinto - spo_syslog_full support for ASCII,BASE64 payload
• Jason Brvenik - variables ... (a long time ago, sorry :P)
• Martin Olsson - remove some useless verbosity unless ./configure --enable-debug is specified and proper flag are used (spo_database and sid-msg.mapv2)
• All other barnyard2 users who help and contribute.
  

BUG REPORTS

• Martin Olsson - bug in sig_reference generation and good discussions. Rewrote the code & al
• John Eure and others - autogen.sh could cause some issue on some system so [autoreconf -fv --install] is not set to autoreconf -fvi
• John Naggets - spo_database: could stop barnyard2 from processing new event if some packets with ip option where processed and option_len was null.
• Fäbu Hufi - spo_syslog_full: in complete mode was printing wrong ip version information and ip header length.
• Jeremy Hoel - identified issue with suppression range in 2-1.13-BETA (fixed in release)
• Bill Green - identified is with signature insertion mainly preprocessor in 2-1.13-BETA (fixed in release) 
• All other barnyard2 users who help and contribute.

NEW FEATURES

1. Support for sid-msg.map version 2 format.

A new sig-msg.map format can be generated by pulledpok (upcomming release, already in svn).

Detection of sid-msg.map version is done by a simple header in the file that shouldn't be altered if you want it to be processed correctly.

The sig-msg.map version 2 format extends the information already present in the sid-msg.map file created from rules.

This new format version allow signature pre-population if users are using output database method with barnyard2 2-1.13 and above.

sid-msg.map v1 format:

SID || MSG || REF 1 || REF N

sid := integer

msg := string

ref := string

sid-msg.map v2 format:

GID || SID || REV || CLASSIFICATION || PRIORITY || MSG || REF 1 || REF N

gid := integer

sid := integer

rev := integer

classification := string (if NULL set to NOCLASS)

priority := integer (if prio == 0, classification priority is used)

msg := string

ref := string

=====================

generator (GID, gen-msg.map) are defaulted to the following value

if their information is not overruled in sid-msg.map v2 file via processing of preprocessor.rules:

revision 1

classification 0

priority 3 

If generator message is present in the sid-msg.map v2 file, and gen-msg.map message are longer 

(more comprehensive by string length), 

gen-msg.map messages are used instead of sid-msg.map v2 file generator messages.

=====================

2. Signature/event logging suppression at spooler level.

Read doc/README.sig_suppression

3. Configuration file variables.

You can now use [var VARNAME value] in the barnyard2 configuration file and every instance of $VARNAME will get replaced by value.

Note that variable declaration order is important only you include a variable with in a variable.

 

 EX (is VALID): 

 var INTERFACE ethX

 var PATH /var/log/IDS

 var LOG $PATH/$INTERFACE/log

 var ARCHIVE $PATH/$INTERFACE/archive

 

 EX (is INVALID): 

 var LOG $PATH/$INTERFACE/log

 var ARCHIVE $PATH/$INTERFACE/archive

 var INTERFACE ethX

 var PATH /var/log/IDS

4. New output database configuration keyword.

Keywords connection_limit and reconnect_sleep_time where added in 2-1.10 but where "undocumented" and shouldn't be modified unless you encounter an issue.

  connection_limit <integer>: default 10

The maximum number of time that barnyard2 will tolerate a transaction faillure and or database connection failure.

  reconnect_sleep_time <integer> : default 5

The number of seconds to sleep betwen connection retry.

  disable_signature_reference_table

Tell the output plugin not to synchronize the sig_reference table in the schema.

Note: This option will speedup the process, especialy if you use sid-msg.mapv2 file or have alot of signature already in databases. (Make sure that you do not need that information before enabling this)

So we hope you enjoy the new release, as a side note the RELEASE.NOTES file has not been updated and will be removed in the next version. It's honestly the most laborious part of release time ;)

Regards,

The barnyard2 team.