ipv6 again
Martijn van Oosterhout
Looking at the barnyard2 source I see that ipv6 is still not supported in the database plugin. But searching through the lists it seems there isn't even a consensus as to what the database schema would look like. The few examples I've seem involve changing the type of the src/dst fields in the iphdr table, but this is patently absurd. It breaks backward compatibility. The IPv6 protocol looks nothing like IPv4, it has a lot less fields for starters. The iphdr is a complete breakdown of the IPv4 header, a correct solution would be a new table just for IPv6, called ip6hdr, something like this:
Table ip6hdr:
sid int(10) unsigned,
cid int(10) unsigned,
ip6_src varchar(40),
ip6_dst varchar(40),
ip6_ver tinyint(3) unsigned,
ip6_len smallint(5) unsigned,
ip6_flow integer unsigned,
ip6_proto tinyint(3) unsigned,
This is at least a forward compatible way. Obviously for databases with a real datatype you'd use that instead of varchar. I think it would be useful to at least get the database structure finalised and committed so people know what it looks like and can start coding against it.
As an aside, the iphdr/tcphdp/udphdr/icmphdr tables are all literal copies of the headers. Does anyone actually use all those fields? Would it not be more useful to make a database-lite which stored just addresses and ports numbers?
Have a nice day,
Brett Edgar
> the database plugin. But searching through the lists it seems there isn't
> even a consensus as to what the database schema would look like.
Yep, several months back I received the "no frontend uses it, so we're
not going to support it right now" reply. Which is a chicken and egg
problem.
> The few
> examples I've seem involve changing the type of the src/dst fields in the
> iphdr table, but this is patently absurd. It breaks backward compatibility.
> The IPv6 protocol looks nothing like IPv4, it has a lot less fields for
> starters. The iphdr is a complete breakdown of the IPv4 header, a correct
> solution would be a new table just for IPv6, called ip6hdr, something like
> this:
>
> Table ip6hdr:
> sid int(10) unsigned,
> cid int(10) unsigned,
> ip6_src varchar(40),
> ip6_dst varchar(40),
> ip6_ver tinyint(3) unsigned,
> ip6_len smallint(5) unsigned,
> ip6_flow integer unsigned,
> ip6_proto tinyint(3) unsigned,
That would be the way to go to stay with a similar schema to the old
ACID Snort DB.
> I think it would be useful
> to at least get the database structure finalised and committed so people
> know what it looks like and can start coding against it.
You and I are in agreement. :)
> As an aside, the iphdr/tcphdp/udphdr/icmphdr tables are all literal copies
> of the headers. Does anyone actually use all those fields? Would it not be
> more useful to make a database-lite which stored just addresses and ports
> numbers?
Indeed it would. I've made my own plugin that keeps the flow
five-tuple in one table, which speeds things up quite a bit since you
don't have to join (or query) multiple tables. In addition, it logs
IPv6 and all the new Snort 2.9.x "extra" events. I call it "simple
db". Message me off-list and I'll be glad to share it with you. The
main by2 people don't seem to be interested in this, though, so I
haven't shared it here.
-Brett
Dave Hajoglou
> Looking at the barnyard2 source I see that ipv6 is still not supported in
> the database plugin. But searching through the lists it seems there isn't
> even a consensus as to what the database schema would look like.
Yep, several months back I received the "no frontend uses it, so we're
not going to support it right now" reply. Which is a chicken and egg
problem.
> The few
> examples I've seem involve changing the type of the src/dst fields in the
> iphdr table, but this is patently absurd. It breaks backward compatibility.
> The IPv6 protocol looks nothing like IPv4, it has a lot less fields for
> starters. The iphdr is a complete breakdown of the IPv4 header, a correct
> solution would be a new table just for IPv6, called ip6hdr, something like
> this:
>
> Table ip6hdr:
> sid int(10) unsigned,
> cid int(10) unsigned,
> ip6_src varchar(40),
> ip6_dst varchar(40),
> ip6_ver tinyint(3) unsigned,
> ip6_len smallint(5) unsigned,
> ip6_flow integer unsigned,
> ip6_proto tinyint(3) unsigned,
That would be the way to go to stay with a similar schema to the old
ACID Snort DB.
beenph
- The new schema will clearly support IPv6 and IPv4 and EXTRA_DATA events.
- The current schema will not be modified to support it.
People can patch the code to their desire but there is mutch more than
the output plugin and a few hacks here and there
that need to be redone, and it is comming.
Unfortunatly in the last month i had less than a few hours to put on
the code to finish the ODBC support
but the schema will get exposed as soon as time allow it.
Layout has already been done, its just not have been pushed to any
repo since its not even in usable state, and since it
touches more than just the output plugin it will go thru testing phases.
But yes it will come, and people are welcome to submit their mods/code
and share it.
-elz
Martijn van Oosterhout
From Brett:
> Yep, several months back I received the "no frontend uses it, so we're
> not going to support it right now" reply. Which is a chicken and egg
> problem.
Well, I'm in the position were I need to write a frontend now and
would like to use a "standard" layout so I know tools in the future
will work with it. Looks like I'll have to go with an in between
solution. Which is unfortunate, since using a custom schema increases
the future maintenance burden.
From Dave re iphdr table:
> The solution below would work though, in reality, does it really make
> sense? How many fields are unnecessarily duplicated? The ip6_ver
> field is unnecessary since the ver field is supposed to indicate v4 or v6.
> The only advantage I see would be query optimization based on version
> since you'd be only querying on v4 or v6 data by table.
Well, I was just following the existing style of the database. I don't
think the existing style makes sense but consistency trumps when
maintaining existing codebases. The existing iphdr table is not really
appropriate since it includes fields which don't exist in ipv6. If the
only fields you're interested in are the src/dst/version then it makes
sense to stick them together. That's why I referred to a possible
snort-lite schema.
Using varbinary is clever, but backward incompatible, which is awkward
if you have many databases to upgrade. Flag days, transition plans,
etc. And I'd use varchar anyway, because I always find it irritating
that I can't just log into the database and query addresses by hand.
On 4 April 2012 04:59, beenph <bee...@gmail.com> wrote:
> I tought i was clear in the past that regarding the IPV6 issue,
> - The new schema will clearly support IPv6 and IPv4 and EXTRA_DATA events.
> - The current schema will not be modified to support it.
But it is not publicly available which means it can't be coded against
either. Maybe the code it not ready, but you could at least publish
the schema, right? It's also a backward incompatible change but if the
benefits are significant (like being able to support multiple packets
per event) then it may be worthwhile.
> But yes it will come, and people are welcome to submit their mods/code
> and share it.
In this sort of situations I'm a fan of release-early-release-often.
Have a nice day,
--
Martijn van Oosterhout <kle...@gmail.com> http://svana.org/kleptog/
beenph
>> I tought i was clear in the past that regarding the IPV6 issue,
>> - The new schema will clearly support IPv6 and IPv4 and EXTRA_DATA events.
>> - The current schema will not be modified to support it.
>
> But it is not publicly available which means it can't be coded against
> either. Maybe the code it not ready, but you could at least publish
> the schema, right? It's also a backward incompatible change but if the
> benefits are significant (like being able to support multiple packets
> per event) then it may be worthwhile.
>
Well it has not been announced because there is still some checks that
need to go in.
But it will be released and people will be encouraged to code against.
And it will be in the public domain.
But again even if the schema is out if you have no data to test
against its not optimal,
this is why we would like to release both at the same time.
We probably see what is the best and make a decision about it shortly.
-elz
Dave Hajoglou
Using varbinary is clever, but backward incompatible, which is awkward
if you have many databases to upgrade. Flag days, transition plans,
etc. And I'd use varchar anyway, because I always find it irritating
that I can't just log into the database and query addresses by hand.
Jim Hranicky
Using varbinary is clever, but backward incompatible, which is awkward
if you have many databases to upgrade. Flag days, transition plans,
etc. And I'd use varchar anyway, because I always find it irritating
that I can't just log into the database and query addresses by hand.
However, with varchar you push off IP comparisons into the code itself and
the DB can't help you (at least not as nicely).
For example, looking for hits on the 10.1.0.0/22 subnet:
select * from ip_table where ip_src >= inet_aton('10.1.0.0') and ip_src <= inet_aton(' 10.1.3.254')
If ip_src is a varchar you'll have to pull each record and check the IP
against the subnet in the code, unless I'm mistaken, or use something
like this:
select * from ip_table where ip_src like '10.1.0.%' or ip_src like '10.1.1.%'
or ip_src like '10.1.2.%' or ip_src like '10.1.3.%'
which is kinda kludgy.
Too bad mysql doesn't have an 'inet' type like postgres - works for ip4 and ip6
and allows this:
select * from ip_table where ip <<= '10.1.1.0/24';
--
Jim Hranicky
IT Security Engineer
Office of Information Security and Compliance
University of Florida
Martijn van Oosterhout
> But again even if the schema is out if you have no data to test
> against its not optimal,
> this is why we would like to release both at the same time.
Not entirely true, barnyard2 isn't the only tool to insert alerts into
a database. With test driven development you make you own test data.
In any case, I think that right I now will just use the schema I
posted earlier. A backward incompatible schema will have to wait until
the next review of this area of code. Barnyard2 will have to support
both formats for a long time anyway, so just because there's a new
format is no excuse to ignoring the old one.
For good or for bad, there are other tools (like BASE) which expect
the current schema and general support for a new schema is going to
take time.
On 4 April 2012 20:08, Jim Hranicky <phra...@gmail.com> wrote:
> However, with varchar you push off IP comparisons into the code itself and
> the DB can't help you (at least not as nicely).
>
> For example, looking for hits on the 10.1.0.0/22 subnet:
>
> select * from ip_table where ip_src >= inet_aton('10.1.0.0') and ip_src <=
> inet_aton(' 10.1.3.254')
This is true. You could say: where inet_aton(ip_src) >
inet_aton('10.1.0.0') but I don't think mysql is smart enough to be
able to optimise that. For that kind of stuff I'd use postgres anyway.
Dave Hajoglou
On Wed, Apr 4, 2012 at 3:05 AM, Martijn van Oosterhout <kle...@gmail.com> wrote:
Using varbinary is clever, but backward incompatible, which is awkward
if you have many databases to upgrade. Flag days, transition plans,
etc. And I'd use varchar anyway, because I always find it irritating
that I can't just log into the database and query addresses by hand.
However, with varchar you push off IP comparisons into the code itself and
the DB can't help you (at least not as nicely).
beenph
<kle...@gmail.com> wrote:
> From beenph:
>> But again even if the schema is out if you have no data to test
>> against its not optimal,
>> this is why we would like to release both at the same time.
>
> Not entirely true, barnyard2 isn't the only tool to insert alerts into
> a database. With test driven development you make you own test data.
> In any case, I think that right I now will just use the schema I
> posted earlier. A backward incompatible schema will have to wait until
> the next review of this area of code. Barnyard2 will have to support
> both formats for a long time anyway, so just because there's a new
> format is no excuse to ignoring the old one.
>
There will be no backward compatible schema.
Mainly because there is no reason to have it.
And seriously if you have something in production. There is no way
you can hope to get performance by using such a schema.
The goal of the new schema is to close that gap so that people can actually
leverage its use until they hit an other threshold.
-elz
Martijn van Oosterhout
> There will be no backward compatible schema.
> Mainly because there is no reason to have it.
Being "backward compatible" *is* the reason.
> And seriously if you have something in production. There is no way
> you can hope to get performance by using such a schema.
I know the performance isn't great, but that isn't really an issue.
How many alerts do you get per second anyway? Even this lame schema
can do dozens of alerts per second without sweating.
BTW, does this mean that the new database module will completely
replace the old one, so upgrading will break everything?
beenph
<kle...@gmail.com> wrote:
> On 6 April 2012 16:15, beenph <bee...@gmail.com> wrote:
>> There will be no backward compatible schema.
>> Mainly because there is no reason to have it.
>
> Being "backward compatible" *is* the reason.
>
>> And seriously if you have something in production. There is no way
>> you can hope to get performance by using such a schema.
>
> I know the performance isn't great, but that isn't really an issue.
> How many alerts do you get per second anyway? Even this lame schema
> can do dozens of alerts per second without sweating.
>
For testing or small network it should be just fine,
but if you work with large number of events
from multiple sensor ....you clearly can't use it reliably on the long run.
> BTW, does this mean that the new database module will completely
> replace the old one, so upgrading will break everything?
>
No the old module will still continue to exist but it will be patched so it
have somewhat of backward compatibility even if the spooler change.
Have you tried the re-writen datbase module by the way.
-elz