sid-msg map issues
russell.fulton
I have recently found that by2 sometimes appears not to pick up things
from the sid-msg.map file and I end up with things like Snort Alert
[1:9900050:0] in the database. Once there they are a pain to to get
rid of (read you have to delete them out of the table in the database
by hand). Once in the database barnyard continues to use them even
though it has the real ones available.
One way this can happen is if you restart snort with new rules and do
not restart barnyard. Since barnyard reads the map file at startup it
will not have the messages for the new rules and generates a place
holder.
I have recently been trying to clean up my DB but a few high frequency
rule are still getting default messages. Hopefully this is a just a
syncing issue but I was pretty sure that I had restarted all my by
instances with good map files before I last deleted the place holders
from the signature table so I am asking:
Are there any known issues with parsing or loading the map files (mine
are generated by pulledpork)?
One thing I would find useful is a way of telling barnyard to reload
the map files ( SIGHUP ?). I'd like to be able to decouple snort and
barnyard and manage the processes separately... so I could just send
by a signal whenever I feed snort new rules.
Russell
russell.fulton
I have now tried shutting down all my barnyard2 processes, deleting
the offending entries from the signature table and the restarting one
barnyard2 process on a busy sensor that I know has a good sid-msg.map
file. Immediately the entries reappear with new sig_id numbers. I
have verified that by2 is reading the right file by running it under
strace and lastly have 'upgraded' to the beta version. nothing seems
to make any difference. It is as if by2 is not using the sid-msg-map
file at all.
In the past I have always been able to force barnyard to load the
proper messaged by deleting the incorrect ones from the signature
table (assuming of course that there is an appropriate entry in the
map).
It must be something I am screwing up since I am sure I have done this
before with the same version of barnyard2.
Any ideas as to how I am shooting my feet?
Umm.. a thought! what access does by2 to the database -- I have
select and insert on all tables and update on sensors (to allow by2 to
update last_cid).
I have changed the db perms a while back... (after accidentally
posting a password in this forum ;)
Russell
beenph
both kinda talk about
the same issue.
I understand the issue, one thing we could do is effectively reload
rules by SIGHUP.
As of whats the result of spo_database interaction with the schema,
this is quite old
code and some stuff that it is doing it really not optimized for
multiple sensor, alot of rule updates etc.
One logical issue that the schema has is , that it shoudln't get
signature message from
barnyard2. All this should be handled by a signature table that is
maintaned by a 3rd party tool
that interface with it and fill tables with the appropriate messages
when parsing new rule file.
Thus, we might easely consider to enable by2 to relaod rules, the core
of the problem lies
deeper and will get addressed in the future.
On Wed, Aug 31, 2011 at 12:35 AM, russell.fulton
<russell...@gmail.com> wrote:
> A bit more on this one.
>
> I have now tried shutting down all my barnyard2 processes, deleting
> the offending entries from the signature table and the restarting one
> barnyard2 process on a busy sensor that I know has a good sid-msg.map
> file. Immediately the entries reappear with new sig_id numbers. I
> have verified that by2 is reading the right file by running it under
> strace and lastly have 'upgraded' to the beta version. nothing seems
> to make any difference. It is as if by2 is not using the sid-msg-map
> file at all.
>
It will also lookup in the database for the signature. Which is the
way it has been designed
since the schema exist.
> In the past I have always been able to force barnyard to load the
> proper messaged by deleting the incorrect ones from the signature
> table (assuming of course that there is an appropriate entry in the
> map).
>
> It must be something I am screwing up since I am sure I have done this
> before with the same version of barnyard2.
If you delete anything from the table you will have to restart
barnyard2, since the old table is not
feteched every time thus the information not refereshed in the process.
So here is what you should to make i work:
1. update rules
2. update maps.
3. stop barnyard
4. SIGHUP snort
5. restart barnyard
Technically this should work quite fine without any changes.
Hopefully this can help you.
-elz
Jim Hranicky
snort.conf and rules files itself? A HUP and you're done.
If there's enough interest I may try to throw a patch together.
I'll probably just try to rip the parsers out of snort 'cause I'm
lazy.
--
Jim Hranicky
IT Security Engineer
Office of Information Security and Compliance
University of Florida
beenph
> What about just doing away with it and having by2 parse the
> snort.conf and rules files itself? A HUP and you're done.
>
> If there's enough interest I may try to throw a patch together.
> I'll probably just try to rip the parsers out of snort 'cause I'm
> lazy.
>
The need for sid and gid map file is to translate [gid:sid:rev]
to text for some output module, but if you think deeply about this,
this type of information should already be stored somewhere in the
backend that
processes the alerts and do the textual matching afterward.
Dependance on gid and sid map file is something that will probably
phase out in the future,
For multiple reason, the main objective being to try to make UI's move
forward and not
depend on an old database schema and old behavior.
Some people could tend to disagree and it would be nice to hear your ideas,
-elz
Russell Fulton
On 1/09/2011, at 6:51 AM, beenph wrote:
>
> Dependance on gid and sid map file is something that will probably
> phase out in the future,
> For multiple reason, the main objective being to try to make UI's move
> forward and not
> depend on an old database schema and old behavior.
>
> Some people could tend to disagree and it would be nice to hear your ideas,
>
I agree. We have to move forward and to do that we inevitably have to break things. The trick is to provide some overlap and a clear way of getting from A to B. I run a sensor network for a large university, currently on my own. I can't suddenly dump large chunks of this and replace it overnight. But I can do it over a year or so, either in small steps or more likely by slowly developing a new system and then rolling it out using the configuration management system.
There has been a lot of criticism of the snort schema over the years and IIRC that was one of the main reasons for the sigul(sp?) team doing their own thing.
Russell
Russell Fulton
On 1/09/2011, at 12:42 AM, beenph wrote:
>> One thing I would find useful is a way of telling barnyard to reload
>> the map files ( SIGHUP ?). I'd like to be able to decouple snort and
>> barnyard and manage the processes separately... so I could just send
>> by a signal whenever I feed snort new rules.
>>
>
> I understand the issue, one thing we could do is effectively reload
> rules by SIGHUP.
>
> As of whats the result of spo_database interaction with the schema,
> this is quite old
> code and some stuff that it is doing it really not optimized for
> multiple sensor, alot of rule updates etc.
>
> One logical issue that the schema has is , that it shoudln't get
> signature message from
> barnyard2. All this should be handled by a signature table that is
> maintaned by a 3rd party tool
> that interface with it and fill tables with the appropriate messages
> when parsing new rule file.
Indeed! I was already thinking along these lines. It would actually be straight forward to patch pulled pork to do this. I'll look at that! I agree it that maintaing the signature table in not 'core business' for BY! Mind you I don't have a good record of getting JJ to accept my patches -- he wants to keep PP 'simple' (something I do have some sympathy with).
One quick question on this: do you add a new record to the table for a new rev or update an existing one? In a multi sensor environment I guess one really should add new records and house keep occasionally.
hmmm… still thinking. We need the classification config too. I think it would be better to do it as a standalone script.
>
> Thus, we might easely consider to enable by2 to relaod rules, the core
> of the problem lies
> deeper and will get addressed in the future.
>
Agreed!
>
>
> On Wed, Aug 31, 2011 at 12:35 AM, russell.fulton
> <russell...@gmail.com> wrote:
>> A bit more on this one.
>>
>> I have now tried shutting down all my barnyard2 processes, deleting
>> the offending entries from the signature table and the restarting one
>> barnyard2 process on a busy sensor that I know has a good sid-msg.map
>> file. Immediately the entries reappear with new sig_id numbers. I
>> have verified that by2 is reading the right file by running it under
>> strace and lastly have 'upgraded' to the beta version. nothing seems
>> to make any difference. It is as if by2 is not using the sid-msg-map
>> file at all.
>>
>
> It will also lookup in the database for the signature. Which is the
> way it has been designed
> since the schema exist.
yep, I understand that and the info from the file is used only when there isn't anything in the database.
>
>
>> In the past I have always been able to force barnyard to load the
>> proper messaged by deleting the incorrect ones from the signature
>> table (assuming of course that there is an appropriate entry in the
>> map).
>>
>> It must be something I am screwing up since I am sure I have done this
>> before with the same version of barnyard2.
>
>
> If you delete anything from the table you will have to restart
> barnyard2, since the old table is not
> feteched every time thus the information not refereshed in the process.
>
>
> So here is what you should to make i work:
>
> 1. update rules
> 2. update maps.
> 3. stop barnyard
> 4. SIGHUP snort
> 5. restart barnyard
that is pretty well what I am doing. (I actually restart snort too rather than using SIGHUP)
I'll keep prodding at it until I figure out what is going on.
Thanks!
Russell