Re: [security-onion] BRO: Waiting for Lock.... Cant get lock

[Eric Ooi]

It looks like whatever it is, it's made more than just bro angry. Have you tried restarting nsm services or rebooting the machine?


On May 8, 2013, at 1:13 PM, Ross Warren <ro...@woodhome.com> wrote:

> sudo broctl status
> waiting for lock ..................................cannot get lock
>
> Something seems to have made bro angry.
>
> Where can I look to troubleshoot?
>
> Files in /nsm/bro/logs/current are not changing
> Files in /var/log/nsm are also not changing.
>
> ps aux | egrep bro
> root 5361 0.0 0.0 65640 10664 ? S 16:59 0:00 /usr/bin/python /opt/bro/bin/broctl start
> root 5384 0.0 0.0 17864 1516 ? S 16:59 0:00 bash /opt/bro/share/broctl/scripts/helpers/start /nsm/bro/spool/manager -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto
>
> Requisite sostat attached (note: snorby is disabled)
>
> Thanks,
> Ross Warren
> CISSP, GCIH, GSEC
>
>
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
> <sostat-bro.txt>

[Ross Warren]

Sorry.. Didnt mention.. This condition is surviving multiple reboots :(

-- Ross Warren

[Eric Ooi]

And each time all services show as "FAIL" like in the sostat?

[Seth Hall]

On May 8, 2013, at 1:13 PM, Ross Warren <ro...@woodhome.com> wrote:

> root 5361 0.0 0.0 65640 10664 ? S 16:59 0:00 /usr/bin/python /opt/bro/bin/broctl start

Hm, for some reason the start command isn't finishing. Did you run this or was this an SO start up script that ran it?

.Seth

--
Seth Hall
International Computer Science Institute
(Bro) because everyone has a network
http://www.bro.org/

[Ross Warren]

Seth,

It is from a script/reboot.. But i get the same results doing it manually.

Where should I look?

Thanks,

Ross Warren

-- Ross Warren

[Ross Warren]

Eric,

On reboot everyone is "FAIL" but then it looks like services are restarted ok manually.

*BUT* just checked and they are screaming red at me :)

Ross W

-- Ross Warren

[Doug Burks]

Hi Ross,

The startup script has a 60-second pause to ensure that all network
interfaces have negotiated link properly. Could that be what you're
seeing?

Please send the output of the following:
sudo service nsm stop
sudo service nsm start
sudo service nsm status

Thanks,
Doug

Doug Burks
http://securityonion.blogspot.com

[Ross Warren]

Doug,

I have attached the requested outputs.

Is there a log/debug i can look into.. I know how to troubleshoot myself, but havent found anything obvious yet.

Thanks

-- Ross Warren

[Doug Burks]

Please send the output of the following:

sudo broctl start
sudo broctl diag

Thanks,
Doug

[Ross Warren]

Doug,

As requested:

sudo broctl start

waiting for lock ..................................cannot get lock

logmanager@anteater:~$ sudo broctl diag

waiting for lock ..................................cannot get lock

-- Ross Warren

[Doug Burks]

I was hoping the "broctl diag" would show more detail.

Are there any running Bro processes?
pgrep -lf bro

Have you made any changes to the Bro configuration or any other files
in /opt/bro/?

Thanks,
Doug

[Ross Warren]

pgrep -lf bro

5361 /usr/bin/python /opt/bro/bin/broctl start

5384 bash /opt/bro/share/broctl/scripts/helpers/start /nsm/bro/spool/manager -U .status -p broctl -p broctl-live -p local -p manager local.bro broctl base/frameworks/cluster local-manager.bro broctl/auto

Last thing i was doing was to enable file extraction with the instructions from Liam.  I believe that I have reversed all those changes.

Thanks Doug.

-Ross Warren

-- Ross Warren

[Doug Burks]

Please try the following:

# Kill bro processes
sudo pkill -9 -f bro

# Verify they got killed
pgrep -lf bro

# Try bro again
sudo broctl check

sudo broctl start
sudo broctl diag

[Ross Warren]

sudo broctl check

manager failed.

   /opt/bro/share/broctl/scripts/check-config: line 14: /set-bro-path: No such file or directory

   /opt/bro/share/broctl/scripts/check-config: line 28: -U: command not found

Repeats for each instance.

line in /opt/bro/share/broctl/scripts/check-config:

source ${scriptsdir}/set-bro-path

I did not touch this.. :)

Ross Warren

-- Ross Warren

[Doug Burks]

Here are some things you can try:

Double-check the changes you made for file extraction and make sure
you have fully reversed them.

Check your Bro config for any changes to the scriptsdir setting.

Compare your broken config to a known good installation (perhaps in a VM).

Doug

[Ross Warren]

Installed SO on a VM.

Discovered that:

/nsm/bro/spool/broctl-config.sh was blank in.  So broctl check couldnt find any settings.. 

I looked at my command history and never touched that file..

so after a 

broctl check

broctl install

broctl start

everyone looks happy again. 

files in /nsm/bro/logs/current show activity again..

Thanks Doug!

-- Ross Warren