barnyard2 startup issues when running on multiple interfaces

[des]

Hi

first a little explanation:
my snort/barnyard setup includes 6 sensor servers, with each about 2-8 sniffing interfaces. 1 separate server works as a master server, where the config is centralised, rule updates happen and the mysql db is running. every morning, the config is copied from the master server to the sensor servers, including rules, etc. after each copying, the snort and the barnyard2 processes are restarted. my init.d/barnyard2 is adjusted so it starts barnyard2 on every required interfaces, logging is happening to separate sub directories.

since i included barnyard2 in the automatic process restart, it does have issues. not on all sensors and not all the time, but it tends to have more problems the more sniffing interfaces there are. i raised the waiting time between stopping and starting the processes, but it doesn't seem to solve it completely and all the time.

i noticed the following:

when the issues appears (startup hangs and won't end, never (tested for hours)), and check the running processes, it seems to start them up wrong (see ppid on eth1):
me@sensor:~> sudo /etc/init.d/barnyard2 start
Starting Snort Output Processor (barnyard2):
starting eth0... used -D -c /etc/snort/barnyard2.conf -d /var/log/snort/eth0 -f snort.u2 -w /var/log/snort/eth0/barnyard2.waldo -a /var/log/snort/eth0/archive -i eth0 -v
starting eth1... used -D -c /etc/snort/barnyard2.conf -d /var/log/snort/eth1 -f snort.u2 -w /var/log/snort/eth1/barnyard2.waldo -a /var/log/snort/eth1/archive -i eth1 -v

me@sensor:~> ps -ef | grep snort
snort    17122     1  0 05:05 ?        00:01:10 /usr/sbin/snort -D -i eth0 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth0
snort    17132     1  0 05:05 ?        00:01:06 /usr/sbin/snort -D -i eth1 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth1
snort    17142     1  0 05:05 ?        00:01:59 /usr/sbin/snort -D -i eth2 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth2
snort    17152     1  1 05:05 ?        00:04:22 /usr/sbin/snort -D -i eth3 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth3
snort    17163     1  0 05:06 ?        00:00:15 /usr/sbin/snort -D -i eth6 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth6
snort    17173     1  0 05:06 ?        00:00:17 /usr/sbin/snort -D -i eth7 -u snort -g snort -c /etc/snort/snort.conf -l /var/log/snort/eth7
root     23116     1 18 08:53 ?        00:00:23 /usr/bin/barnyard2 -D -c /etc/snort/barnyard2.conf -d /var/log/snort/eth0 -f snort.u2 -w /var/log/snort/eth0/barnyard2.waldo -a /var/log/snort/eth0/archive -i eth0 -v
root     23117 23114  1 08:53 pts/0    00:00:02 /usr/bin/barnyard2 -D -c /etc/snort/barnyard2.conf -d /var/log/snort/eth1 -f snort.u2 -w /var/log/snort/eth1/barnyard2.waldo -a /var/log/snort/eth1/archive -i eth1 -v
root     23118 23117 18 08:53 ?        00:00:22 /usr/bin/barnyard2 -D -c /etc/snort/barnyard2.conf -d /var/log/snort/eth1 -f snort.u2 -w /var/log/snort/eth1/barnyard2.waldo -a /var/log/snort/eth1/archive -i eth1 -v
me    24344 24290  0 08:55 pts/1    00:00:00 grep snort

I unfortunately can't find any hints in the /var/log/messages. It sometimes does help to manually stop the process, wait a while, and start it again. My guess is, that it does have issues with reading the existing u2 or waldo files, but that doesn't explain why the new processes are started with a weird PPID (usually it's always 1). Or is that normal during startup? Setup and config of barnyard2 and snort is the same on all sensors.

Do you have any hints on that? I'm running on Version 2.1.11 (Build 317), more infos attached.,

Thank you & cheers
des

[beenph]

On Wed, Mar 20, 2013 at 4:34 AM, des <d.sa...@gmail.com> wrote:
> Hi
>
Greetings D Sacher,

> first a little explanation:
> my snort/barnyard setup includes 6 sensor servers, with each about 2-8
> sniffing interfaces. 1 separate server works as a master server, where the
> config is centralised, rule updates happen and the mysql db is running.
> every morning, the config is copied from the master server to the sensor
> servers, including rules, etc. after each copying, the snort and the
> barnyard2 processes are restarted. my init.d/barnyard2 is adjusted so it
> starts barnyard2 on every required interfaces, logging is happening to
> separate sub directories.
>
> since i included barnyard2 in the automatic process restart, it does have
> issues. not on all sensors and not all the time, but it tends to have more
> problems the more sniffing interfaces there are. i raised the waiting time
> between stopping and starting the processes, but it doesn't seem to solve it
> completely and all the time.

From the quick glipmse i gave to what you paste my recommendation would be
that you have 1 startup 1 stop script for each barnyard instance. (the
same should go for snort imho)

When you look at the above you have two instance of eth1 barnyard2 running.
1 in daemonized mode and 1 that seem to be bound to a console (pts/0).
Which could explain mabey why you have problem stopping them if they all
are stoped from 1 startup/stop script.

I would also recommend that you use custom PID path for each barnyard2 instance
using the following command line argument:
--pid-path <dir> Specify the directory for the
barnyard2 PID file

Let us know if this helps out.
Cheers,

-elz

[des]

Hi elz :)

> first a little explanation:
> my snort/barnyard setup includes 6 sensor servers, with each about 2-8
> sniffing interfaces. 1 separate server works as a master server, where the
> config is centralised, rule updates happen and the mysql db is running.
> every morning, the config is copied from the master server to the sensor
> servers, including rules, etc. after each copying, the snort and the
> barnyard2 processes are restarted. my init.d/barnyard2 is adjusted so it
> starts barnyard2 on every required interfaces, logging is happening to
> separate sub directories.
>
> since i included barnyard2 in the automatic process restart, it does have
> issues. not on all sensors and not all the time, but it tends to have more
> problems the more sniffing interfaces there are. i raised the waiting time
> between stopping and starting the processes, but it doesn't seem to solve it
> completely and all the time.
From the quick glipmse i gave to what you paste my recommendation would be
that you have 1 startup 1 stop script for each barnyard instance. (the
same should go for snort imho)

ui, isn't that going to get ugly? from a DRY point of view that sounds totally ugly.. (sorry, just my first thought)
 

i do see the 2 instances, they don't make sense to me, but they happen during the startup process always when it hangs. like mentioned, on other servers with lesser sniffing interfaces, it runs through without problems, also i than never see 2 processes on the same int. this morning the issue appeared again. i stopped the startup process and closed all barnyard processes, waited for about 10 minutes or something, and at the next try it ran through without any problems. same startup scripts, no parameters changed or anything.

these are my start() und stop() parts from the /etc/init.d/barnyard2:

start() {
        echo $"Starting $desc ($prog): "
        for INT in $INTERFACES; do
                echo -n "starting $INT..."
                PIDFILE="/var/run/barnyard2-$INT.pid"
                ARCHIVEDIR="$SNORTDIR/$INT/archive"
                WALDO_FILE="$SNORTDIR/$INT/barnyard2.waldo"
                # We need Interfaced PIDs, otherwise we can't lock
                EXTRA_ARGS="-i $INT -v"
                #BARNYARD_OPTS="-D -c $CONF -d $SNORTDIR/${INT} -f $LOG_FILE -w $WALDO_FILE -a $ARCHIVEDIR -G /etc/snort/gen-msg.map -S /etc/snort/sid-msg.map -C /etc/snort/classification.config $EXTRA_ARGS"
                BARNYARD_OPTS="-D -c $CONF -d $SNORTDIR/${INT} -f $LOG_FILE -w $WALDO_FILE -a $ARCHIVEDIR $EXTRA_ARGS"
                echo " used $BARNYARD_OPTS"
                $barnyardBin $BARNYARD_OPTS
        done
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch /var/run/$prog
        return $RETVAL
}

stop() {
        echo $"Shutting down $desc ($prog): "
        #for f in `ls /var/run/$prog*.pid`;do
        #echo "killing `basename $f`"
        #/sbin/killproc -p $f -TERM $barnyardBin && rm ${f}.lck
        #done
        #RETVAL=$?
        #echo "killed $prog"
        #[ $RETVAL -eq 0 ] && rm -f /var/run/$prog
        pkill barnyard2
        RETVAL=$?
        return $RETVAL
}
 
in my opinion the pid file is specific for each interface, so that shouldn't be a problem. as it 'usually' works i'm really not sure if it's a general issue on my startup script or if it's rather a timing issue where i need to wait longer for files to be ready

thank you for your help
-des

[beenph]

Not at all.
First of all having all instances in a same script mean that you have to wait
Nth time for the Nth instance whil having parallel script you can all
start them up at the same time.

And you can stop individual instances instead of stoping them all,
less of an issue in my book.

>
> these are my start() und stop() parts from the /etc/init.d/barnyard2:
>
> start() {
> echo $"Starting $desc ($prog): "
> for INT in $INTERFACES; do
> echo -n "starting $INT..."
> PIDFILE="/var/run/barnyard2-$INT.pid"
> ARCHIVEDIR="$SNORTDIR/$INT/archive"
> WALDO_FILE="$SNORTDIR/$INT/barnyard2.waldo"
> # We need Interfaced PIDs, otherwise we can't lock
> EXTRA_ARGS="-i $INT -v"
> #BARNYARD_OPTS="-D -c $CONF -d $SNORTDIR/${INT} -f $LOG_FILE
> -w $WALDO_FILE -a $ARCHIVEDIR -G /etc/snort/gen-msg.map -S
> /etc/snort/sid-msg.map -C /etc/snort/classification.config $EXTRA_ARGS"
> BARNYARD_OPTS="-D -c $CONF -d $SNORTDIR/${INT} -f $LOG_FILE
> -w $WALDO_FILE -a $ARCHIVEDIR $EXTRA_ARGS"
> echo " used $BARNYARD_OPTS"
> $barnyardBin $BARNYARD_OPTS
> done
> RETVAL=$?
> echo
> [ $RETVAL -eq 0 ] && touch /var/run/$prog
> return $RETVAL
> }
>

having PIDFILE variable in your script and not passed to barnyard2
process in command line wont make it use a different
PIDFILE so check that out ;)

> stop() {
> echo $"Shutting down $desc ($prog): "
> #for f in `ls /var/run/$prog*.pid`;do
> #echo "killing `basename $f`"
> #/sbin/killproc -p $f -TERM $barnyardBin && rm ${f}.lck
> #done
> #RETVAL=$?
> #echo "killed $prog"
> #[ $RETVAL -eq 0 ] && rm -f /var/run/$prog
> pkill barnyard2
> RETVAL=$?
> return $RETVAL
> }
>

and issuing a SIGKILL to process is a bit nasty, you should maybe try
to send SIGINT.

> in my opinion the pid file is specific for each interface, so that shouldn't
> be a problem. as it 'usually' works i'm really not sure if it's a general
> issue on my startup script or if it's rather a timing issue where i need to
> wait longer for files to be ready
>

Nope your script is not doing that ATM, you have the variable set but
its not given to command line.
so you might want to fix it.

But overall it is your setup.

But from my observation point, right now there is enhancement to be
to make everything more reliable and to be able to diagnostic some
possible issue not related to
launch Nth process in a row.

-elz

[des]

Not at all.
First of all having all instances in a same script mean that you have to wait
Nth time for the Nth instance whil having parallel script you can all
start them up at the same time.

And you can stop individual instances instead of stoping them all,
less of an issue in my book.

true that :)

2nd true that :)
 

thank you, these were some hints i can work with :) i'll see what i come up with..

cheers
des

[beenph]

>
>
> thank you, these were some hints i can work with :) i'll see what i come up
> with..

No problems,
do not hesitate to post back if you encounter some issue,
I am sure that someone will step in and help out.

-elz

[tacti...@gmail.com]

hi so Im tryng to set up a snort box and its my first time really digging into linux and snort so thankfully I have found some good guides. So I was following a guide that has you create a script to run barnyard2 and so I copied the text in and saved it and did all the other steps.

So then I type service barnyard2 start as the instructions state and it outputs an error so it has 35 and so Im thinking that is the line that is dying in the process and so I comment it out and it works. My question is what does it do ? Im new to bash script and linux so here is the code. the problem section $prog $BARNYARD_OPTS

#!/bin/sh
#
# Init file for Barnyard2
#
#
# chkconfig: 2345 40 60
# description:  Barnyard2 is an output processor for snort.
#
# processname: barnyard2
# config: /etc/sysconfig/barnyard2
# config: /etc/snort/barnyard.conf
# pidfile: /var/lock/subsys/barnyard2.pid


[ -x /usr/sbin/snort ] || exit 1
[ -r /etc/snort/snort.conf ] || exit 1

### Default variables
SYSCONFIG="/etc/default/barnyard2"

### Read configuration
[ -r "$SYSCONFIG" ] && . "$SYSCONFIG"

RETVAL=0
prog="barnyard2"
desc="Snort Output Processor"

start() {
       echo -n $"Starting $desc ($prog): "
       for INT in $INTERFACES; do
               PIDFILE="/var/lock/barnyard2-$INT.pid"
               ARCHIVEDIR="$SNORTDIR/$INT/archive"
               WALDO_FILE="$SNORTDIR/$INT/barnyard2.waldo"
               BARNYARD_OPTS="-D -c $CONF -d $SNORTDIR/${INT} -w $WALDO_FILE -l $SNORTDIR/${INT} -a $ARCHIVEDIR -f $LOG_FILE -X $PIDFILE $EXTRA_ARGS"
               (this line here )$prog $BARNYARD_OPTS 
       done
       RETVAL=$?
       echo
       [ $RETVAL -eq 0 ] && touch /var/lock/$prog
       return $RETVAL
}

stop() {
       echo -n $"Shutting down $desc ($prog): "
       killall $prog
       RETVAL=$?
       echo
       [ $RETVAL -eq 0 ] && rm -f /var/lock/$prog
       return $RETVAL
}

restart() {
       stop
       start
}


reload() {
       echo -n $"Reloading $desc ($prog): "
       killall $prog -HUP
       RETVAL=$?
       echo
       return $RETVAL
}


case "$1" in
 start)
       start
       ;;
 stop)
       stop
       ;;
 restart)
       restart
       ;;
 reload)
       reload
       ;;
 condrestart)
       [ -e /var/lock/$prog ] && restart
       RETVAL=$?
       ;;
 status)
       status $prog
       RETVAL=$?
       ;;
dump)
       dump
       ;;
 *)
       echo $"Usage: $0 {start|stop|restart|reload|condrestart|status|dump}"
       RETVAL=1
esac

exit $RETVAL

I would appreciate any help you guys could give me on this and what it does.

[beenph]

Set prob="barnyard2" to the full path of the barnyard2 binary on your system.

ex:
prog="/usr/local/bin/barnyard2"

This will depend where the binary is found on your system.

Also look in /etc/default/barnyard2 if you have the INTERFACES
variable declared.

Hope this helps.

-elz

[tacti...@gmail.com]

Thanks I will give them a try and let you guys know if it worked.