Duplicates in signature table with different sig_priority and sig_class_id

[Bill Green]

Hi folks,

I have a fresh install on Centos 6.4:

snort (Version 2.9.4 GRE (Build 40))
BY2 (Version 2.1.13-BETA (Build 325)) - going to MySQL via unfied2 [output unified2: filename snort.log, limit 128]
PulledPork (PulledPork v0.6.1 the Smoking Pig)

Also Snorby 2.6.1 on MySQL. 5.1.69

I have a set of 8 network taps split between two bridge interfaces (to consolidate the simplex taps), and there are two snort processes and two BY2 processes running, one for each bridge interface. I have one of the snort processes in a split-config, one for each of several VLANS. So far, so good.

The problem I'm seeing is when I view event data either with Snorby, or directly in the DB. If I drop the DB and recreate it fresh, and then start the snort and BY2 processes, BY2 seems to populate the signature table with the entries in gen-msg.map...

But then, after the BY2 process starts to actually put events into the DB, I see things like this in the signature table:

+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+
| sig_id | sig_class_id | sig_name                                                     | sig_priority | sig_rev | sig_sid | sig_gid | events_count |
+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+
|    167 |            0 | frag3: Fragments smaller than configured min_fragment_length |            3 |       1 |      13 |     123 |            0 |
|    479 |            7 | Snort Alert [123:13:1]                                       |            2 |       1 |      13 |     123 |          737 |
+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+

My questions are:

1. Where does BY2 get the data it initially populates into the signature table, and where does it get the priority and class from?
2. When snort creates an event and BY2 picks it up and goes to stick it into the DB, where does the priority and class come from in a case like this where it's a preprocessor alert?
3. Why do these two entries differ in their class and priority? I see that dbProcessSignatureInformation() checks to ensure that these fields must all match before the signature is declared 'found' in the table.

I've checked all the obvious things I can think of, can somebody point me in the right direction? I would greatly appreciate it, thanks!

[beenph]

On Thu, May 9, 2013 at 12:22 AM, Bill Green <green...@gmail.com> wrote:
> Hi folks,
>
Hi Bill,

As of 2-1.13-BETA (2-1.13) it will depends on the version of the
sid-msg.map file.

If PulledPork is generating v1 sid-msg.map file, priority and
classification and revision comes from the first event seen.

If PulledPork is generating a v2 sid-msg.map file then those field
comes from the file.

As of now i think that only current PulledPork trunk
my $VERSION = "PulledPork v0.6.2dev the Cigar Pig <////~";

Support generation of v2 sid-msg.map file.

> 2. When snort creates an event and BY2 picks it up and goes to stick it into
> the DB, where does the priority and class come from in a case like this
> where it's a preprocessor alert?

In v1 map file, those come from the first event seen except for
preprocessor which
where pre-populated and updated on first trigger.

> 3. Why do these two entries differ in their class and priority? I see that
> dbProcessSignatureInformation() checks to ensure that these fields must all
> match before the signature is declared 'found' in the table.
>

As mentioned above its probably due to the new feature introduced,
which introduced a
possible regression when working in v1 context, since the new addition
of the v2 support
has been merged in 2-1.13

> I've checked all the obvious things I can think of, can somebody point me in
> the right direction? I would greatly appreciate it, thanks!
>

In the meantime you could also give a tries to PP trunk and use a v2
file on a fresh database and it should work
without a issue.

I will try to keep the tread up to date when i isolate the culprit.


-elz

[beenph]

Hey Bill,

Can you give a run from this branch:

https://github.com/binf/barnyard2/tree/fix-signals

Added a check for v1 vs v2 codepath at the generator level that should
fix the issue.

Let me know how it goes.

-elz

[Bill Green]

I grabbed the other trunk of PP and re-ran the rules download to get a new v2 sid-msg.map. I cleared the DB, and restarted snort and BY2. After some time, I now have:

mysql> select * from signature where sig_gid=123 and sig_sid=13;

+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+
| sig_id | sig_class_id | sig_name                                                     | sig_priority | sig_rev | sig_sid | sig_gid | events_count |
+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+

|    571 |            7 | frag3: Fragments smaller than configured min_fragment_length |            2 |       1 |      13 |     123 |            0 |
|   8011 |            7 | frag3: Fragments smaller than configured min_fragment_length |            2 |       1 |      13 |     123 |            0 |
+--------+--------------+--------------------------------------------------------------+--------------+---------+---------+---------+--------------+
2 rows in set (0.29 sec)

Now the class and priority are the same? ID 571 was inserted at the same time this was posted in the logs as BY2 was initializing:

May  8 22:54:59 ids01 barnyard2[12964]: [SignaturePullDataStore()]: No signature found in database ...
May  8 22:55:39 ids01 barnyard2[12978]: [SignaturePullDataStore()]: No signature found in database ...

Then id 8011 showed up some 10 minutes or so later, so this doesn't *seem* like a race condition.

I will try your suggested branch tomorrow and let you know. If there is anything else I can provide which would be helpful, please let me know.

Thank you very much for looking at this.

[beenph]

Are your using MyIASM storage for mysql or InnoDB?

You need to be using InnoDB.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "barnyard2-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to barnyard2-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[beenph]

You can find out which engine is running your database by using the
mysql client with your barnyard2 credential
by running the following query:

SELECT table_name,engine FROM INFORMATION_SCHEMA.TABLES WHERE
table_schema=DATABASE();

-elz

[Bill Green]

It's InnoDB.

[Bill Green]

So, I've uninstalled MySQL 5.1 and installed MariaDB 5.5. Verified that Snorby (which creates the DB) is using Innodb. I switched PP back to v0.6.1, and recomiled to your BY2 fix Version 2.1.13-BETA (Build 325).

Restarted snort and BY2, and after a while now have the duplicates below... I'll email the files you asked for.

MariaDB [snorby]> select sig_name, count(*) from signature group by sig_sid,sig_gid having count(*) > 1 order by count(*) desc;
+-------------------------------------------------------------------------------------------------------------------------+----------+
| sig_name                                                                                                                | count(*) |
+-------------------------------------------------------------------------------------------------------------------------+----------+
| dnp3: DNP3 Link-Layer Frame was dropped.                                                                                |        2 |
| pop: No memory available for decoding. Memcap exceeded.                                                                 |        2 |
| pop: Unknown POP3 response                                                                                              |        2 |
| pop: Unknown POP3 command                                                                                               |        2 |
| imap: Unknown IMAP4 response                                                                                            |        2 |
| imap: Unknown IMAP4 command                                                                                             |        2 |
| sip: Content length is too large or negative                                                                            |        2 |
| sip: URI is too long                                                                                                    |        2 |
| sensitive_data: sensitive data - U.S. phone numbers                                                                     |        2 |
| sensitive_data: sensitive data - U.S. social security numbers with dashes                                               |        2 |
| smtp: Attempted header name buffer overflow                                                                             |        2 |

| frag3: Fragments smaller than configured min_fragment_length                                                            |        2 |

| http_inspect: UNKNOWN METHOD                                                                                            |        2 |
| http_inspect: POST W/O CONTENT-LENGTH OR CHUNKS                                                                         |        2 |
| http_inspect: NON-RFC DEFINED CHAR                                                                                      |        2 |
| snort_decoder: WARNING: ICMP Destination Unreachable Communication with Destination Host is Administratively Prohibited |        2 |
| snort_decoder: WARNING: ICMP Destination Unreachable Communication Administratively Prohibited                          |        2 |
| snort_decoder: WARNING: ICMP redirect host                                                                              |        2 |
| snort_decoder: WARNING: ICMP6 type not decoded                                                                          |        2 |
| snort_decoder: WARNING: IPV4 packet both DF and offset set                                                              |        2 |
| snort_decoder: WARNING: Experimental TCP options                                                                        |        2 |
+-------------------------------------------------------------------------------------------------------------------------+----------+
21 rows in set (0.03 sec)

[Starner, Mark]

Is this a new requirement? I don't recall anything related to this when I
first setup the DB.

I just checked and my MySQL is using MyISAM and not InnoDB.

Can it be changed from MyISAM to InnoDB without starting over?

What is the difference?

[beenph]

On Thu, May 9, 2013 at 2:22 PM, Starner, Mark <mark.s...@unisys.com> wrote:
> Is this a new requirement? I don't recall anything related to this when I
> first setup the DB.
>
> I just checked and my MySQL is using MyISAM and not InnoDB.
>
> Can it be changed from MyISAM to InnoDB without starting over?
>
> What is the difference?
>
>

http://dev.mysql.com/doc/refman/5.6/en/storage-engines.html

<SNIP>
InnoDB: A transaction-safe (ACID compliant) storage engine for MySQL
that has commit, rollback, and crash-recovery capabilities to protect
user data.
InnoDB row-level locking (without escalation to coarser granularity
locks) and Oracle-style consistent nonlocking reads increase
multi-user concurrency and
performance. InnoDB stores user data in clustered indexes to reduce
I/O for common queries based on primary keys. To maintain data
integrity,
InnoDB also supports FOREIGN KEY referential-integrity constraints.
InnoDB is the default storage engine in MySQL 5.6.
</SNIP>

[Bill Green]

Just for fun, I recreated the DB, restarted with just one snort / BY2 process, and the signature table still had duplicates

--

---
You received this message because you are subscribed to a topic in the Google Groups "barnyard2-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/barnyard2-users/fnjiqbvWCFE/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to barnyard2-use...@googlegroups.com.

[beenph]

On Thu, May 9, 2013 at 3:20 PM, Bill Green <green...@gmail.com> wrote:
> Just for fun, I recreated the DB, restarted with just one snort / BY2
> process, and the signature table still had duplicates
>

That's funky stuff.
I did a test running two simultaneous instances running in daemonized
mode using two different sensor_id with empty signature information
and I can't reproduce, using your map files.

mysql> SELECT * FROM (SELECT count(*) AS a,sig_sid,sig_gid FROM
signature WHERE sig_gid <> 1 group by sig_sid,sig_gid) as tq WHERE
tq.a>1;
Empty set (0.10 sec)
mysql> select * from sensor;

+-----+--------------+-----------+--------+--------+----------+----------+
| sid | hostname | interface | filter | detail | encoding | last_cid |
+-----+--------------+-----------+--------+--------+----------+----------+
| 1 | tho:eth0aeex | eth0aeex | NULL | 1 | 0 | 308142 |
| 2 | tho:eth0affx | eth0affx | NULL | 1 | 0 | 1 |
+-----+--------------+-----------+--------+--------+----------+----------+
2 rows in set (0.02 sec)
mysql> SELECT * FROM (SELECT count(*) AS a,sig_sid,sig_gid FROM
signature WHERE sig_gid <> 1 group by sig_sid,sig_gid) as tq WHERE
tq.a>1;
Empty set (0.28 sec)


binf@SINGULAR:/tmp$ ps -adefl | grep barn
1 R binf 31782 1 17 80 0 - 42146 - 15:20 ?
00:00:54 /home/binf/213Release/Comp/src/barnyard2 -c
../../SIDV2/test/barnyard2.conf -d ../../SIDV2/test/log/ -l
../../SIDV2/test/log/ -f snort.log -D
1 R binf 31784 1 18 80 0 - 42151 - 15:20 ?
00:00:53 /home/binf/213Release/Comp/src/barnyard2 -c
../../SIDV2/test/barnyard2-2.conf -d ../../SIDV2/test/log/ -l
../../SIDV2/test/log/ -f snort.log -D

Im trying to see what could be producing that.

[Bill Green]

Well, dbProcessSignatureInformation() checks the sig_priority and sig_class id to determine whether an entry exists in the signature table, right? The question I have is why are the class id and priority in the new entry being added different from those which were added at initialization ? The info for class and priority should all be coming from the same source I should think, why the difference?

[Bill Green]

I grabbed the fix-signals branch, and re-ran with a clean database. That fixed my issue.

Thanks Eric!

[beenph]

On Fri, May 10, 2013 at 11:34 AM, Bill Green <green...@gmail.com> wrote:
> I grabbed the fix-signals branch, and re-ran with a clean database. That
> fixed my issue.
>
> Thanks Eric!
>

Good stuff, so to summerize it up, when i merged sid-msgv2 map format support
i forgot some corner cases that where specific to sid-msgv1 format
and this had as a drawback when processing preprocessor events with an
"inserted at initialization signature but never had an event
inserted".

Thanks to your vigilance Bill in helping make this issue bite the dust.

Cheers.

-elz