Bro proxy crashing
bmp
I've SO setup as a standalone server running just Bro.
Server specs:
Dell R720
Dual E5-2680 2.7 GHz CPUs
96GB of RAM (6*16GB 1600 MHz RDIMMS)
NIC: Intel Ethernet Converged Network Adapter X520-SR
Disk Space: 4x 1.2TB 10K RPM SAS 6Gbps
The server is monitoring a 10G link, but its not fully utilized - the peak traffic has been approx 7Gbps.
I have Bro configured to run 26 worker processes. Initially I had assigned 15 workers, but in a previous discuss where I was running into memory issues - one suggestion was to increase the number to workers. That didn't really help with memory issue but the recent pf-rings update seems to have fixed it.
Recently I started to notice that the disk space was filling up quickly and logs weren't getting retained for more than a day. This was because the spool directory was filled will crash logs.
And it looks like the proxy keeps crashing everytime. But the manager and cluster seems to be working fine and logs are being written.
/nsm/bro/spool/tmp/post-terminate-2014-10-07-09-00-01-10408-crash/stderr.log
::::::::::::::
/opt/bro/share/bro/securityonion/./bpfconf.bro, line 81: BPFConf filename set: /etc/nsm/Dell-tracker10-eth1/bpf-bro.conf (proxy)
internal error: unknown msg type 115 in Poll()
/opt/bro/share/broctl/scripts/run-bro: line 85: 3865 Aborted (core dumped) nohup $mybro "$@"
From broctl status, the status always shows with ??? in the Peers column:
proxy proxy x.x.x.x running 16504 ??? 07 Oct 09:15:02
I did notice a similar issue mentioned on a Bro discussion group - but i think it was for a cluster setup and the solution was to add more proxies and have a dedicated proxy for each host with 14-15 workers per proxy.
Mine is a standalone setup and I wasn't sure how to add more proxies on it. I tried reducing the workers back to 15 and but had trouble starting Bro.
Any thoughts on how to resolve this? I've attached the output from sostat - i cleared old crash files in the spool folder, so my disk space is now at 22%.
Thanks,
Benson
Doug Burks
From the sostat output:
warning: broctl config has changed (run the broctl "restart --clean"
or "install" command)
Have you tried the following?
sudo nsm_sensor_ps-restart --only-bro
Load average is very high:
top - 09:43:54 up 19:09, 2 users, load average: 44.72, 44.23, 42.85
How many CPU cores do you have? Do you have more workers than CPU cores?
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com
bmp
Doug,
It has 32 CPU processors. I had Bro setup to run 15 workers initially but netstats indicated lot of drops.
After the recent update, i ran "sudo nsm_sensor_ps-restart --only-bro", bro "restart check" and also install. But it kept showing that warning. Just ran sudo nsm_sensor_ps-restart --only-bro again.
The proxy is still crashing with the same error. sostat attached.
-Benson
Doug Burks
or "install" command)
output in your reply:
# Stop Bro
sudo broctl stop
# Check for any remaining Bro processes
pgrep -lf bro
# kill them manually if necessary
# Install config
sudo broctl install
# Start Bro
sudo broctl start
bmp
So I've followed the steps you've mentioned and the warning has gone away. See attached sostat.txt
Still seeing the same error and the proxy keeps crashing.
Infact, prior to the recent pf-rings update i used to see the Peer value for proxy as ??? everytime I ran broctl status. But since the logs were being created without any problem, I didn't bother to investigate further. Its only when I ran out of disk space completely that I took notice of it and check the stderr file.
Same error was noticed by another in the past:
http://comments.gmane.org/gmane.comp.security.detection.bro/6731
-Benson
Doug Burks
# Stop Bro
sudo broctl stop
# Install config
sudo broctl install
# Start Bro
sudo broctl start
bmp
root@x:~# broctl stop
stopping Dell-tracker10-eth1-1 ...
stopping Dell-tracker10-eth1-10 ...
stopping Dell-tracker10-eth1-11 ...
stopping Dell-tracker10-eth1-12 ...
stopping Dell-tracker10-eth1-13 ...
stopping Dell-tracker10-eth1-14 ...
stopping Dell-tracker10-eth1-15 ...
stopping Dell-tracker10-eth1-16 ...
stopping Dell-tracker10-eth1-17 ...
stopping Dell-tracker10-eth1-18 ...
stopping Dell-tracker10-eth1-19 ...
stopping Dell-tracker10-eth1-2 ...
stopping Dell-tracker10-eth1-20 ...
stopping Dell-tracker10-eth1-21 ...
stopping Dell-tracker10-eth1-22 ...
stopping Dell-tracker10-eth1-23 ...
stopping Dell-tracker10-eth1-24 ...
stopping Dell-tracker10-eth1-25 ...
stopping Dell-tracker10-eth1-26 ...
stopping Dell-tracker10-eth1-3 ...
stopping Dell-tracker10-eth1-4 ...
stopping Dell-tracker10-eth1-5 ...
stopping Dell-tracker10-eth1-6 ...
stopping Dell-tracker10-eth1-7 ...
stopping Dell-tracker10-eth1-8 ...
stopping Dell-tracker10-eth1-9 ...
stopping proxy ...
stopping manager ...
after few sec ran
root@x:~# pgrep -lf bro
root@x:~#
root@Dell-tracker10:~# broctl install
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/site ... done.
removing old policies in /nsm/bro/spool/installed-scripts-do-not-touch/auto ... done.
creating policy directories ... done.
installing site policies ... done.
generating cluster-layout.bro ... done.
generating local-networks.bro ... done.
generating broctl-config.bro ... done.
updating nodes ... done.
root@x:~# broctl start
starting manager ...
starting proxy ...
starting Dell-tracker10-eth1-1 ...
starting Dell-tracker10-eth1-10 ...
starting Dell-tracker10-eth1-11 ...
starting Dell-tracker10-eth1-12 ...
starting Dell-tracker10-eth1-13 ...
starting Dell-tracker10-eth1-14 ...
starting Dell-tracker10-eth1-15 ...
starting Dell-tracker10-eth1-16 ...
starting Dell-tracker10-eth1-17 ...
starting Dell-tracker10-eth1-18 ...
starting Dell-tracker10-eth1-19 ...
starting Dell-tracker10-eth1-2 ...
starting Dell-tracker10-eth1-20 ...
starting Dell-tracker10-eth1-21 ...
starting Dell-tracker10-eth1-22 ...
starting Dell-tracker10-eth1-23 ...
starting Dell-tracker10-eth1-24 ...
starting Dell-tracker10-eth1-25 ...
starting Dell-tracker10-eth1-26 ...
starting Dell-tracker10-eth1-3 ...
starting Dell-tracker10-eth1-4 ...
starting Dell-tracker10-eth1-5 ...
starting Dell-tracker10-eth1-6 ...
starting Dell-tracker10-eth1-7 ...
starting Dell-tracker10-eth1-8 ...
starting Dell-tracker10-eth1-9 ...
root@x:~# broctl status
Name Type Host Status Pid Peers Started
manager manager x.x.x.x running 21030 27 08 Oct 10:45:32
proxy proxy x.x.x.x running 21067 ??? 08 Oct 10:45:34
x-eth1-1 worker x.x.x.x running 21432 2 08 Oct 10:45:36
x-eth1-10 worker x.x.x.x running 21435 2 08 Oct 10:45:36
x-eth1-11 worker x.x.x.x running 21437 2 08 Oct 10:45:36
x-eth1-12 worker x.x.x.x running 21439 2 08 Oct 10:45:36
x-eth1-13 worker x.x.x.x running 21441 2 08 Oct 10:45:36
x-eth1-14 worker x.x.x.x running 21443 2 08 Oct 10:45:36
x-eth1-15 worker x.x.x.x running 21445 2 08 Oct 10:45:36
x-eth1-16 worker x.x.x.x running 21446 2 08 Oct 10:45:36
x-eth1-17 worker x.x.x.x running 21448 2 08 Oct 10:45:36
x-eth1-18 worker x.x.x.x running 21450 2 08 Oct 10:45:36
x-eth1-19 worker x.x.x.x running 21452 2 08 Oct 10:45:36
x-eth1-2 worker x.x.x.x running 21454 2 08 Oct 10:45:36
x-eth1-20 worker x.x.x.x running 21456 2 08 Oct 10:45:36
x-eth1-21 worker x.x.x.x running 21459 2 08 Oct 10:45:36
x-eth1-22 worker x.x.x.x running 21461 2 08 Oct 10:45:36
x-eth1-23 worker x.x.x.x running 21463 2 08 Oct 10:45:36
x-eth1-24 worker x.x.x.x running 21465 2 08 Oct 10:45:36
x-eth1-25 worker x.x.x.x running 21466 2 08 Oct 10:45:36
x-eth1-26 worker x.x.x.x running 21469 2 08 Oct 10:45:36
x-eth1-3 worker x.x.x.x running 21470 2 08 Oct 10:45:36
x-eth1-4 worker x.x.x.x running 21472 2 08 Oct 10:45:36
x-eth1-5 worker x.x.x.x running 21475 2 08 Oct 10:45:36
x-eth1-6 worker x.x.x.x running 21476 2 08 Oct 10:45:36
x-eth1-7 worker x.x.x.x running 21477 2 08 Oct 10:45:36
x-eth1-8 worker x.x.x.x running 21478 2 08 Oct 10:45:36
x-eth1-9 worker x.x.x.x running 21479 2 08 Oct 10:45:36
Doug Burks
If you're still having problems with the proxy crashing, please try
reducing your number of workers as I mentioned in the email I sent a
few minutes ago:
Please try reducing your Bro workers as follows:
sudo broctl stop
sudo broctl install
# Start Bro
sudo broctl start
bmp
I stopped bro, edited the number of workers back to 15, ran install and then started bro back up.
root@x:~# broctl status
Name Type Host Status Pid Peers Started
manager manager x.x.x.x running 26058 16 08 Oct 10:50:34
proxy proxy x.x.x.x running 26097 ??? 08 Oct 10:50:36
x-eth1-1 worker x.x.x.x running 26318 2 08 Oct 10:50:39
x-eth1-10 worker x.x.x.x running 26321 2 08 Oct 10:50:39
x-eth1-11 worker x.x.x.x running 26320 2 08 Oct 10:50:39
x-eth1-12 worker x.x.x.x running 26323 2 08 Oct 10:50:39
x-eth1-13 worker x.x.x.x running 26324 2 08 Oct 10:50:39
x-eth1-14 worker x.x.x.x running 26327 2 08 Oct 10:50:39
x-eth1-15 worker x.x.x.x running 26328 2 08 Oct 10:50:39
x-eth1-2 worker x.x.x.x running 26333 2 08 Oct 10:50:39
x-eth1-3 worker x.x.x.x running 26335 2 08 Oct 10:50:39
x-eth1-4 worker x.x.x.x running 26334 2 08 Oct 10:50:39
x-eth1-5 worker x.x.x.x running 26338 2 08 Oct 10:50:39
x-eth1-6 worker x.x.x.x running 26336 2 08 Oct 10:50:39
x-eth1-7 worker x.x.x.x running 26340 2 08 Oct 10:50:39
x-eth1-8 worker x.x.x.x running 26339 2 08 Oct 10:50:39
x-eth1-9 worker x.x.x.x running 26341 2 08 Oct 10:50:39
Peer value for proxy still showing up as ??? . No crash reports yet though.
And so far no drops on bro netstats. I'll let it run for a while - my peak traffic is later during the day. Will continue to monitor it.
-Benson
bmp
root@x:~# sudo broctl status
Name Type Host Status Pid Peers Started
manager manager x.x.x.x running 30100 15 08 Oct 10:54:16
proxy proxy x.x.x.x crashed
x-eth1-1 worker x.x.x.x running 30358 1 08 Oct 10:54:21
x-eth1-10 worker x.x.x.x running 30360 1 08 Oct 10:54:21
x-eth1-11 worker x.x.x.x running 30362 1 08 Oct 10:54:21
x-eth1-12 worker x.x.x.x running 30363 1 08 Oct 10:54:21
x-eth1-13 worker x.x.x.x running 30366 1 08 Oct 10:54:21
x-eth1-14 worker x.x.x.x running 30368 1 08 Oct 10:54:21
x-eth1-15 worker x.x.x.x running 30370 1 08 Oct 10:54:21
x-eth1-2 worker x.x.x.x running 30371 1 08 Oct 10:54:21
x-eth1-3 worker x.x.x.x running 30373 1 08 Oct 10:54:21
x-eth1-4 worker x.x.x.x running 30376 1 08 Oct 10:54:21
x-eth1-5 worker x.x.x.x running 30378 1 08 Oct 10:54:21
x-eth1-6 worker x.x.x.x running 30380 1 08 Oct 10:54:21
x-eth1-7 worker x.x.x.x running 30381 1 08 Oct 10:54:21
x-eth1-8 worker x.x.x.x running 30382 1 08 Oct 10:54:21
x-eth1-9 worker x.x.x.x running 30383 1 08 Oct 10:54:21
more /nsm/bro/spool/proxy/stderr.log
/opt/bro/share/bro/securityonion/./bpfconf.bro, line 81: BPFConf filename set: /etc/nsm/x-eth1/bpf-bro.conf (proxy)
internal error: unknown msg type 115 in Poll()
/opt/bro/share/broctl/scripts/run-bro: line 85: 30138 Aborted (core dumped) nohup $mybro "$@"
Seth Hall
On Oct 8, 2014, at 11:13 AM, bmp <benson....@gmail.com> wrote:
> Proxy has crashed again.
Are you loading the default configuration or have you added your own scripts? It's definitely possible that you could be running some script that is over zealously trying to synchronize data.
.Seth
--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/
bmp
Tried commenting Shellshock on local.bro and testing again. Same problem.
Doug Burks
https://code.google.com/p/security-onion/wiki/Hardware#RAM
You'll probably want to increase your RAM and/or consider adding a
second box and splitting the traffic between the two.
On Wed, Oct 8, 2014 at 1:25 PM, bmp <benson....@gmail.com> wrote:
> I recently added the Shellshock script. Other than that its only running everything that comes with it by default.
>
> Tried commenting Shellshock on local.bro and testing again. Same problem.
>
Seth Hall
On Oct 10, 2014, at 8:08 AM, Doug Burks <doug....@gmail.com> wrote:
> 96GB RAM is probably not enough for monitoring 7Gbps of traffic:
Without having a closer look at the box I'm not really sure what the problem could be. It's very possible that some specific traffic could be leading to an overabundance of communication though.
Doug Burks
>
> On Oct 10, 2014, at 8:08 AM, Doug Burks <doug....@gmail.com> wrote:
>
>> 96GB RAM is probably not enough for monitoring 7Gbps of traffic:
>
> The problem that Bro is exhibiting is the typical communication overload issue we've been dealing with for a long time, it doesn't seem to be memory exhaustion (on Bro's part at least).
>
> Without having a closer look at the box I'm not really sure what the problem could be. It's very possible that some specific traffic could be leading to an overabundance of communication though.
its own proxy.)