barnyard2 reliability

[Fedor Sakharov]

Hi all.

I've been using barnyard2 for a while and I have encountered a few problems. My setup includes a number of sensors in different geographic locations. Those sensors use barnyard2 to send alerts to central server. However, the setup proves to be not very reliable and here is why:
1) A sensor may be behind NAT that fails from time to time. If barnyard2 looses connect to the server it will fail and stop sending alerts which means that all alerts that happened during the periods of poor network connectivity between sensors and the central server will be lost. Most likely barnyard2 will stop working in case of losing connectivity and will not restart on it's own, so some monitoring has to be done.
2) The connection to remote mysql itself is not very good. By default the connection happens over an unencrypted channel which is not good either. In the case when there are a lot of sensors each one of them has access to all data in snorby DB. If one of them is compromised, the eavesdropper obviously gets access to the whole DB.
3) If the ruleset differs from one sensor to the other this will result in conflicts between sensors -> some of them will fail to work.

So my current setup is not very reliable for production use. Could you give me some advice on fixing it? I've been thinking about extending barnyard2 to send alerts over some reliable message passing like rabbitmq which could solve both the connectivity problem and direct DB access problem.

[beenph]

On Mon, May 12, 2014 at 6:21 AM, Fedor Sakharov
<fedor.s...@gmail.com> wrote:
> Hi all.
>

Hi Fedor

> I've been using barnyard2 for a while and I have encountered a few problems.
> My setup includes a number of sensors in different geographic locations.
> Those sensors use barnyard2 to send alerts to central server. However, the
> setup proves to be not very reliable and here is why:
> 1) A sensor may be behind NAT that fails from time to time. If barnyard2
> looses connect to the server it will fail and stop sending alerts which
> means that all alerts that happened during the periods of poor network
> connectivity between sensors and the central server will be lost. Most
> likely barnyard2 will stop working in case of losing connectivity and will
> not restart on it's own, so some monitoring has to be done.
> 2) The connection to remote mysql itself is not very good. By default the
> connection happens over an unencrypted channel which is not good either. In
> the case when there are a lot of sensors each one of them has access to all
> data in snorby DB. If one of them is compromised, the eavesdropper obviously
> gets access to the whole DB.

It would be interesting to know which version you use because since
2-1.10 the database
output plugin has been re-written and you will not loose any data if
your connection betwen
the barnyard2 instance and the database die.

Data will continue to accumulate in the unified2 file while the
connection is down and or barnayrd2
is not running and as soon as it comes up again it will process the backlog.

Also since 2-1.19 you can use a SSL setup betwen mysql and or postgresql.

Also current bug-fix-release 2-1.13 build 333
ref: https://github.com/binf/barnyard2/tree/bug-fix-release

Is the version you should use.



As for schema security you can isolate them using different database
without an issue but thats up to you to do
and also you can tighten security of your schema by following the
rules found here and restrict schema access.
ref: https://github.com/binf/barnyard2/tree/bug-fix-release

> 3) If the ruleset differs from one sensor to the other this will result in
> conflicts between sensors -> some of them will fail to work.

Before going there we need to know which version you are using and
then advise on what to do.

> So my current setup is not very reliable for production use. Could you give
> me some advice on fixing it? I've been thinking about extending barnyard2 to
> send alerts over some reliable message passing like rabbitmq which could
> solve both the connectivity problem and direct DB access problem.
>

Cheers,
-elz

[Fedor Sakharov]

It would be interesting to know which version you use because since
2-1.10 the database
output plugin has been re-written and you will not loose any data if
your connection betwen
the barnyard2 instance and the database die.

I am using 2.1.13 from  https://github.com/firnsy/barnyard2 master.

Data will continue to accumulate in the unified2 file while the
connection is down and or barnayrd2
is not running and as soon as it comes up again it will process the backlog.

The behaviour as i see it that barnyard2 may only run with -n flag and will send only newly arrived events (that were written to unified files after it's launched) or without this flag and on startup it will send all events in all unified files it sees.
 

Also since 2-1.19 you can use a SSL setup betwen mysql and or postgresql.

Also current bug-fix-release 2-1.13 build 333
ref: https://github.com/binf/barnyard2/tree/bug-fix-release

Is the version you should use.

Ok I will switch to it.

As for schema security you can isolate them using different database
without an issue but thats up to you to do
and also you can tighten security of your schema by following the
rules found here and restrict schema access.
ref: https://github.com/binf/barnyard2/tree/bug-fix-release

I am using snorby with the DB so i cannot really separate them. 

[beenph]

On Tue, May 13, 2014 at 4:51 AM, Fedor Sakharov
<fedor.s...@gmail.com> wrote:
>
>
>> It would be interesting to know which version you use because since
>> 2-1.10 the database
>> output plugin has been re-written and you will not loose any data if
>> your connection betwen
>> the barnyard2 instance and the database die.
>>
> I am using 2.1.13 from https://github.com/firnsy/barnyard2 master.
>

Then how did you observe or conclude that you would be loosing events?

>>
> Ok I will switch to it.
>>
>>
>> As for schema security you can isolate them using different database
>> without an issue but thats up to you to do
>> and also you can tighten security of your schema by following the
>> rules found here and restrict schema access.
>> ref: https://github.com/binf/barnyard2/tree/bug-fix-release
>>
>
> I am using snorby with the DB so i cannot really separate them.
>

Separate? You can either create multiple database and probably snortby
can support that
or create different logging users for your different instance also the
way you might want
to do that will depend on the database backend you use.

-elz

[Fedor Sakharov]

Then how did you observe or conclude that you would be loosing events?

I had my barnyard2 running on some server and at some point connectivity between this server and db server was lost. B2 detected that connection was lost and exited. It generally stops working in case it fails to connect do db, so i had to write a monit script that watches barnyard2 and re-starts it in case it fails. however it can be restarted with -n flag -> so i will see only newly arrived events or without it -> it would resend everything on each restart. It would be a great thing if barnyard2 could somehow 'remember' the last successfully sent event and start sending all events that happened later in case it is restarted.

[beenph]

Mabey you want to check out what the waldo file is used for.
Since that behavoir would only explain that you never have used waldo file.

And it would stop if it can re-establish the connection in a timely
fashion, thus it means that you might have uplink issue betwen your
sensor and your database.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "barnyard2-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to barnyard2-dev...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.