Fatal errors with new barnyard

[Miguel Alvarez]

Hi,

I re-installed snorby with a clean database and am now adding my sensors to it.  I'm also upgrading my barnyard2 version to 1.13 as part of the process.

The re-adding of sensors completed successfully on the first nine but now on each subsequent sensor, I'm getting the following error:

# sudo grep -A2 FATAL /var/log/messages
May 27 16:57:46 sensor10 barnyard2[3318]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('44243','1911','2');]
May 27 16:57:46 sensor10 barnyard2[3318]: Barnyard2 exiting
--
May 27 17:00:34 sensor10 barnyard2[3365]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('44243','1911','2');]
May 27 17:00:34 sensor10 barnyard2[3365]: Barnyard2 exiting
--
May 27 17:19:43 sensor10 barnyard2[4729]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('44243','1911','2');]
May 27 17:19:43 sensor10 barnyard2[4729]: Barnyard2 exiting

# sudo grep barnyard2 /var/log/messages |grep -A2 FATAL
May 27 17:09:48 sensor11 barnyard2[27243]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:09:48 sensor11 barnyard2[27243]: Barnyard2 exiting
May 27 17:09:48 sensor11 barnyard2[27243]: database: Closing connection to database "snorby"
--
May 27 17:10:07 sensor11 barnyard2[27248]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:10:07 sensor11 barnyard2[27248]: Barnyard2 exiting
May 27 17:10:07 sensor11 barnyard2[27248]: database: Closing connection to database "snorby"
--
May 27 17:10:21 sensor11 barnyard2[27251]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:10:21 sensor11 barnyard2[27251]: Barnyard2 exiting
May 27 17:10:21 sensor11 barnyard2[27251]: database: Closing connection to database "snorby"
--
May 27 17:10:30 sensor11 barnyard2[27254]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:10:30 sensor11 barnyard2[27254]: Barnyard2 exiting
May 27 17:10:30 sensor11 barnyard2[27254]: database: Closing connection to database "snorby"
--
May 27 17:10:50 sensor11 barnyard2[27257]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:10:50 sensor11 barnyard2[27257]: Barnyard2 exiting
May 27 17:10:50 sensor11 barnyard2[27257]: database: Closing connection to database "snorby"
--
May 27 17:11:00 sensor11 barnyard2[27260]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:11:00 sensor11 barnyard2[27260]: Barnyard2 exiting
May 27 17:11:00 sensor11 barnyard2[27260]: database: Closing connection to database "snorby"
--
May 27 17:11:15 sensor11 barnyard2[27318]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:11:15 sensor11 barnyard2[27318]: Barnyard2 exiting
May 27 17:11:15 sensor11 barnyard2[27318]: database: Closing connection to database "snorby"
--
May 27 17:11:17 sensor11 barnyard2[27338]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:11:17 sensor11 barnyard2[27338]: Barnyard2 exiting
May 27 17:11:17 sensor11 barnyard2[27338]: database: Closing connection to database "snorby"
--
May 27 17:11:28 sensor11 barnyard2[27341]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:11:28 sensor11 barnyard2[27341]: Barnyard2 exiting
May 27 17:11:28 sensor11 barnyard2[27341]: database: Closing connection to database "snorby"
--
May 27 17:28:47 sensor11 barnyard2[27551]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:28:47 sensor11 barnyard2[27551]: Barnyard2 exiting
May 27 17:28:47 sensor11 barnyard2[27551]: database: Closing connection to database "snorby"
--
May 27 17:29:01 sensor11 barnyard2[27554]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:29:01 sensor11 barnyard2[27554]: Barnyard2 exiting
May 27 17:29:01 sensor11 barnyard2[27554]: database: Closing connection to database "snorby"
--
May 27 17:29:21 sensor11 barnyard2[27557]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:29:21 sensor11 barnyard2[27557]: Barnyard2 exiting
May 27 17:29:21 sensor11 barnyard2[27557]: database: Closing connection to database "snorby"
--
May 27 17:29:31 sensor11 barnyard2[27560]: FATAL ERROR: database mysql_error: Duplicate entry '1911-2' for key 'PRIMARY'
May 27 17:29:31 sensor11 barnyard2[27560]: Barnyard2 exiting
May 27 17:29:31 sensor11 barnyard2[27560]: database: Closing connection to database "snorby"

The new sensors and their interfaces appear in the snorby UI so I'm not quite sure what's going on.  Any suggestions?

Thank you!

MA

[Miguel Alvarez]

I didn't think I'd need to as the db was clean but doing a 'DELETE from sig_reference;' (as mentioned in the upgrade requirement notes) resolved the problem.

Cheers,

MA

[beenph]

If you started your clean db with a version before 2-1.13 and then
upgraded in the process, yes you needed to clean the table.

Cheers,
-elz

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "barnyard2-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to barnyard2-use...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Miguel Alvarez]

There were no previous versions that were writing to this database and I got paged in the middle of the night when our automated rule update process restarted snort and barnyard. Barnyard failed to start on every one of my sensors with the following:

# sudo grep -A2 'May 28 04:' /var/log/messages |grep -A2 'FATAL ERROR'
May 28 04:10:50 sensor12 barnyard2[31780]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:10:50 sensor12 barnyard2[31780]: Barnyard2 exiting
--
May 28 04:11:11 sensor12 barnyard2[31783]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:11:11 sensor12 barnyard2[31783]: Barnyard2 exiting
--
May 28 04:11:32 sensor12 barnyard2[31788]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:11:32 sensor12 barnyard2[31788]: Barnyard2 exiting
--
May 28 04:11:41 sensor12 barnyard2[31791]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:11:41 sensor12 barnyard2[31791]: Barnyard2 exiting
--
May 28 04:11:58 sensor12 barnyard2[31794]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:11:58 sensor12 barnyard2[31794]: Barnyard2 exiting
--
May 28 04:12:08 sensor12 barnyard2[31817]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:12:08 sensor12 barnyard2[31817]: Barnyard2 exiting
--
May 28 04:12:18 sensor12 barnyard2[31820]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:12:18 sensor12 barnyard2[31820]: Barnyard2 exiting
--
May 28 04:12:27 sensor12 barnyard2[31825]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:12:27 sensor12 barnyard2[31825]: Barnyard2 exiting
--
May 28 04:12:31 sensor12 barnyard2[31828]: FATAL ERROR: database mysql_error: Duplicate entry '1963-2' for key 'PRIMARY'
        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('14464','1963','2');]
May 28 04:12:31 sensor12 barnyard2[31828]: Barnyard2 exiting

So I downgraded back to 1.12 and haven't had a problem since.  This is MariaDB 5.5.31 on Ubuntu 12.04.2 LTS -- not sure if that matters.  But yes, it was a brand new, clean database with no pre-existing data.  After I ran into the problem the first time and cleaned out sig_reference, everything was good until barnyard was restarted for the rule update and it blew up on every one of my sensors.

Thank you,

MA

[beenph]

Which process do you use for your rule update?
Is it home made?

Also if you do not use the sig_reference table you can allway use the
disable_signature_reference_table for the database output plugin.

-elz

[beenph]

On Tue, May 28, 2013 at 10:56 AM, beenph <bee...@gmail.com> wrote:
> Which process do you use for your rule update?
> Is it home made?
>
> Also if you do not use the sig_reference table you can allway use the
> disable_signature_reference_table for the database output plugin.
>
> -elz
>

I would be curious to see which rules would do this,

You can execute the following SQL query

SELECT sig_sid FROM signature where sig_id='14464';

From there you should have the signature id that could be different
in your sid-msg.map file.

So mabey you would like to compare the row from sensor12 to other
sensors you have.

Thanks in advance Miguel,
-elz

[Miguel Alvarez]

Sorry for the delayed response, but no, we use pulledpork.  

And would you happen to know how to delete a sensor from the DB?  I deleted one from the snorby UI on Monday but it's still there (and present in the DB).  Would it be something like this?

delete from sensor where sid = 'xx'?

I'm just not sure if there's anything more to it?

Thank you,

MA

[beenph]

On Wed, May 29, 2013 at 3:49 PM, Miguel Alvarez <migue...@gmail.com> wrote:
> Sorry for the delayed response, but no, we use pulledpork.
>

No problems Miguel,
but did you look at the map files with the signature to see if you had
any differences??

> And would you happen to know how to delete a sensor from the DB? I deleted
> one from the snorby UI on Monday but it's still there (and present in the
> DB). Would it be something like this?
>


> delete from sensor where sid = 'xx'?
>

yes.
This will completly delete the information from the database.
Also when you mentionned earlier starting from a new db, did you did a
delete with snorby or you create a completly new database?

> I'm just not sure if there's anything more to it?
>

Well you could also allways delete the events associated with that
sensor, if snorby does not do it but i can't say
which operation snorby does on the schema.

-elz

[Miguel Alvarez]

Hi Eric,

On Thu, May 30, 2013 at 5:13 AM, beenph <bee...@gmail.com> wrote:
>
> On Wed, May 29, 2013 at 3:49 PM, Miguel Alvarez <migue...@gmail.com> wrote:
> > Sorry for the delayed response, but no, we use pulledpork.
> >
> No problems Miguel,
> but did you look at the map files with the signature to see if you had
> any differences??

Yes but interestingly, it returned no data:

MariaDB [snorby]> select sig_sid FROM signature where sig_id='14464';
Empty set (0.00 sec)

MariaDB [snorby]> select * FROM signature where sig_id='14464';
Empty set (0.00 sec)

> > And would you happen to know how to delete a sensor from the DB?  I deleted
> > one from the snorby UI on Monday but it's still there (and present in the
> > DB).  Would it be something like this?
> >
> >
> > delete from sensor where sid = 'xx'?
>
> yes.
> This will completly delete the information from the database.
> Also when you mentionned earlier starting from a new db, did you did a
> delete with snorby or you create a completly new database?

Great, thanks!

 
> Well you could also allways delete the events associated with that
> sensor, if snorby does not do it but i can't say
> which operation snorby does on the schema.

Understood -- thank you again, Eric!

MA