barnyard2 repeatedly segfaulting on SO

[Kevin Branch]

I keep having barnyard2 silently crash on my up to date standalone SO box whenever it encounters a "[SURICATA TCP header length too small]" event.  Deleting some or all of my snort.unified2 files and then restarting barnyard2 does not stop the crashing.  I never had this problem with the previous barnyard version (2.1.9), and when I just now downgraded barnyard 2.1.11 on the SO box back to 2.1.9, the problem immediately went away, and now for the first time I see "[SURICATA TCP header length too small]" events showing up in ELSA.

Is anybody else having mysterious barnyard2 crashes on specific Snort and/or Suricata  events?  I wonder if this is a bug introduced in barnyard 2.1.11.

The barnyard2 log account of the crash always looks like this
.  When I run it interactively with exactly the same parameters that SO uses, the crash happens shortly after barnyard2 starts, looking like this:

root@WANIDS:~# barnyard2 -c /etc/nsm/WANIDS-eth1/barnyard2.conf -d /nsm/sensor_data/WANIDS-eth1 -f snort.unified2 -w /etc/nsm/WANIDS-eth1/barnyard2.waldo -i 1 -U
Running in Continuous mode

        --== Initializing Barnyard2 ==--
Initializing Input Plugins!
Initializing Output Plugins!
Parsing config file "/etc/nsm/WANIDS-eth1/barnyard2.conf"
: Duplicate classification "default-login-attempt"found, ignoring this line
: Duplicate classification "non-standard-protocol"found, ignoring this line
: Duplicate classification "shellcode-detect"found, ignoring this line
: Duplicate classification "string-detect"found, ignoring this line
: Duplicate classification "suspicious-filename-detect"found, ignoring this line
: Duplicate classification "suspicious-login"found, ignoring this line
: Duplicate classification "system-call-detect"found, ignoring this line
: Duplicate classification "tcp-connection"found, ignoring this line
: Duplicate classification "trojan-activity"found, ignoring this line
: Duplicate classification "unusual-client-port-connection"found, ignoring this line
: Duplicate classification "web-application-activity"found, ignoring this line
Barnyard2 spooler: Event cache size set to [2048]
Log directory = /nsm/sensor_data/WANIDS-eth1
INFO database: Defaulting Reconnect/Transaction Error limit to 10
INFO database: Defaulting Reconnect sleep time to 5 second
sguil:  sensor name = WANIDS-eth1
sguil:  agent port =  8000
sguil:  Connected to localhost on 8000.
database: compiled support for (mysql)
database: configured to use mysql
database: schema version = 107
database:           host = 127.0.0.1
database:           user = root
database:  database name = snorby
database:    sensor name = WANIDS-eth1:1
database:      sensor id = 1
database:     sensor cid = 6212
database:  data encoding = hex
database:   detail level = full
database:     ignore_bpf = no
database: using the "alert" facility

        --== Initialization Complete ==--

  ______   -*> Barnyard2 <*-
 / ,,_  \  Version 2.1.11 (Build 317) TCL
 |o"  )~|  By Ian Firns (SecurixLive): http://www.securixlive.com/
 + '''' +  (C) Copyright 2008-2012 Ian Firns <fir...@securixlive.com>

Using waldo file '/etc/nsm/WANIDS-eth1/barnyard2.waldo':
    spool directory = /nsm/sensor_data/WANIDS-eth1
    spool filebase  = snort.unified2
    time_stamp      = 1391216083
    record_idx      = 14057
Opened spool file '/nsm/sensor_data/WANIDS-eth1/snort.unified2.1391216083'
INFO [dbProcessSignatureInformation()]: [Event: 6823] with [gid: 1] [sid: 2200034] [rev: 1] [classification: 0] [priority: 3] Signature Message -> "[SURICATA TCP header length too small]"
         was not found in barnyard2 signature cache, this could mean its is the first time the signature is processed, and will be inserted
         in the database with the above information, this message should only be printed once for each signature that is not  present in the database
         The new inserted signature will not have its information present in the sig_reference table,it should be present on restart
         if the information is present in the sid-msg.map file.
         You can allways update the message via a SQL query if you want it to be displayed correctly by your favorite interface

Segmentation fault (core dumped)

Kevin Branch

[Doug Burks]

Hi Kevin,

I haven't experienced this issue.

Have you tried disabling the "SURICATA TCP header length too small" rule?

Note that we do have a Barnyard update (Barnyard2-1.13) on our Roadmap:
https://code.google.com/p/security-onion/wiki/Roadmap

Any chance you could try Barnyard2-1.13 and see if that resolves your issue?

> --
> You received this message because you are subscribed to the Google Groups
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Doug Burks

[Kevin Branch]

Doug,

After examining the source for the securityonion-barnyard2 package to figure out the right configure script build parameters, I used the following procedure to upgrade to barnyard 2.1.13.

sudo apt-get install libmysqlclient-dev libpcap-dev libtool autoconf tcl8.4-dev
wget https://github.com/firnsy/barnyard2/archive/v2-1.13.tar.gz
tar zxvf v2-1.13.tar.gz
cd barnyard2-2-1.13/
./autogen.sh
./configure --prefix=/usr --with-tcl=/usr/lib/tcl8.4/ --with-libpcap-includes=/opt/pfring/lib/ --with-mysql-libraries=/usr/lib/*-linux-gnu/ --with-mysql-includes=/usr/include/mysql/
make
mv /usr/bin/barnyard2 /usr/bin/barnyard2-old
sudo nsm_sensor_ps-stop --only-barnyard2
make install
mysql snorby -e "delete from sig_reference; delete from reference;"
sudo nsm_sensor_ps-start --only-barnyard2

The securityonion-barnyard2 source package actually referred to "--with-tcl=/usr/lib/tcl8.5/" but since tcl8.4 seems to be what in included in SO stable, I had to change it to "--with-tcl=/usr/lib/tcl8.4/".  It was also necessary to clear out the sig_reference and reference tables in the snorby database before the new barnyard2 would work, but from what I read that is recommended anyway when upgrading barnyard2.  It has run for a while with diverse events flowing through it just fine without any crash yet.  I have not yet seen a "[SURICATA TCP header length too small]" event though, so I'm going to wait and see.  I'll let you know how it goes.

Kevin


On 2/5/2014 9:52 AM, Kevin Branch wrote:

Doug,

Yes I could disable that one rule which appears to possibly be tripping a barnyard 2.1.11 bug.  I didn't try that yet because I figured whatever was causing the problem with that one rule would eventually trip me up on some other rule I just had not seen yet.
I would be happy to try barnyard 2.1.13.   Do you have an RPM already built sitting in a testing repo?  If not, I can build barnyard2 myself.

Kevin

[Kevin Branch]

Doug,

I have had SO running with the new barnyard 2.1.13 for several days now, during which a diverse set of alerts have fired, and I have had no trouble with it.   Since the barnyard upgrade, I have specifically seen the "[SURICATA TCP header length too small]" alert come through which was persistently crashing SO's stock version of barnyard (2.1.11) but which caused no problem with the old 2.1.9 version of barnyard.  I presume there was a subtle bug introduced in barnyard 2.1.10 or 2.1.11 which is no longer there in 2.1.13.  Good that deploying the new barnyard version is already on the roadmap.

I am really impressed with the Security Onion project.  Not only is it already awesome and proving very useful to me in my NSM work, but it just keeps getting better, and at an impressive rate.  Thanks!

Kevin

[Doug Burks]

Replies inline.

On Sat, Feb 8, 2014 at 12:24 PM, Kevin Branch
<branchnet...@gmail.com> wrote:
> Doug,
>
> I have had SO running with the new barnyard 2.1.13 for several days now,
> during which a diverse set of alerts have fired, and I have had no trouble
> with it. Since the barnyard upgrade, I have specifically seen the
> "[SURICATA TCP header length too small]" alert come through which was
> persistently crashing SO's stock version of barnyard (2.1.11) but which
> caused no problem with the old 2.1.9 version of barnyard. I presume there
> was a subtle bug introduced in barnyard 2.1.10 or 2.1.11 which is no longer
> there in 2.1.13. Good that deploying the new barnyard version is already on
> the roadmap.

That's good news, thanks for your feedback!

> I am really impressed with the Security Onion project. Not only is it
> already awesome and proving very useful to me in my NSM work, but it just
> keeps getting better, and at an impressive rate. Thanks!

Thanks, glad you like it!

[Kevin Branch]

I just ran into another case of the stock SO barnyard 2.1.13 segfaulting and getting restarting by the watchdog process every 5 minutes according to /var/log/nsm/watchdog.log. This is on an entirely different SO machine and network than where I encountered the issue before. I used the same barnyard upgrade method I mentioned on Feb 5, and the segfaulting immediately stopped.

Kevin

[Doug Burks]

Thanks for the followup, Kevin!

On Tue, Mar 4, 2014 at 12:24 PM, Kevin Branch

<branchnet...@gmail.com> wrote:
> I just ran into another case of the stock SO barnyard 2.1.13 segfaulting and getting restarting by the watchdog process every 5 minutes according to /var/log/nsm/watchdog.log. This is on an entirely different SO machine and network than where I encountered the issue before. I used the same barnyard upgrade method I mentioned on Feb 5, and the segfaulting immediately stopped.
>
> Kevin
>
> On Saturday, February 8, 2014 1:43:46 PM UTC-5, Doug Burks wrote:
>> Replies inline.
>>
>>
>>
>> On Sat, Feb 8, 2014 at 12:24 PM, Kevin Branch
>>
>> <branchnet...@gmail.com> wrote:
>>
>> > Doug,
>>
>> >
>>
>> > I have had SO running with the new barnyard 2.1.13 for several days now,
>>
>> > during which a diverse set of alerts have fired, and I have had no trouble
>>
>> > with it. Since the barnyard upgrade, I have specifically seen the
>>
>> > "[SURICATA TCP header length too small]" alert come through which was
>>
>> > persistently crashing SO's stock version of barnyard (2.1.11) but which
>>
>> > caused no problem with the old 2.1.9 version of barnyard. I presume there
>>
>> > was a subtle bug introduced in barnyard 2.1.10 or 2.1.11 which is no longer
>>
>> > there in 2.1.13. Good that deploying the new barnyard version is already on
>>
>> > the roadmap.
>>
>>
>>
>> That's good news, thanks for your feedback!
>>
>>
>>
>> > I am really impressed with the Security Onion project. Not only is it
>>
>> > already awesome and proving very useful to me in my NSM work, but it just
>>
>> > keeps getting better, and at an impressive rate. Thanks!
>>
>>
>>
>> Thanks, glad you like it!
>

> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Doug Burks