Re: [security-onion] 100% CPU utilization on new sensor install

[Liam Randall]

How much traffic are you feeding this thing?  Basic hardware configuration- memory, disk, cpu?

I assume the system was responsive before you ran the setup wizard?

I would stop the various components one at a time until you can run a sostat.

You can stop various components for you initial testing with 

sudo nsm_sensor_ps-stop

    --sensor-name=<name>             Define specific sensor <name> to process

    --only-barnyard2                 Only process barnyard2

    --only-snort-alert               Only process snort alert

    --only-pcap                      Only process packet logger

    --only-argus                     Only process argus

    --only-prads                     Only process prads

    --only-bro                       Only process bro

    --only-pcap-agent                Only process pcap_agent

    --only-sancp-agent               Only process sancp_agent

    --only-snort-agent               Only process snort_agent

    --only-http-agent                Only process http_agent

    --only-pads-agent                Only process pads_agent

    --only-ossec-agent               Only process ossec_agent

    --skip-barnyard2                 Skip processing of barnyard2

    --skip-snort-alert               Skip processing of snort alert

    --skip-pcap                      Skip processing of packet logger

    --skip-argus                     Skip processing of argus

    --skip-prads                     Skip processing of prads

    --skip-bro                       Skip processing of bro

    --skip-pcap-agent                Skip processing of pcap_agent

    --skip-sancp-agent               Skip processing of sancp_agent

    --skip-snort-agent               Skip processing of snort_agent

    --skip-http-agent                Skip processing of http_agent

    --skip-pads-agent                Skip processing of pads_agent

    --skip-ossec-agent               Skip processing of ossec_agent

    --dialog                         Same as -d

    --force-yes                      Same as -y

    --version                        Same as -V

    --help                           Same as -?

On Sat, Jan 26, 2013 at 1:34 PM, Jake Sallee <elcra...@gmail.com> wrote:

Please bear with me, I am new to SO and do not know all of its ins-and-outs yet.

I demoed SO as a basic setup and it worked excellently, however now that I am trying to deploy a sensor and server as my next test I am running into trouble.

My sensor is running 100% cpu at all times, so much so that the keyboard and mouse are not responsive.  If I move the mouse it may take 2-3 min for that movement to show up on the screen.

I thought this was because of the settings I chose during the sensor deployment.  So I re-ran the deployment tool choosing only a single instance of Snort and Bro which should be fine with my 4 way processor.

But still the system is unresponsive.

I completely unplugged ALL network connections to try to make sure the default rule set in snort wasn't just killing my box ( I know i need to tune the box but i can't get that far yet >:o ) but still the CPU is maxed out.

There is an error during the sosetup script the second time it asks me for the SSH credentials, something about not being able to get the FQDN on the server.  Bit it goes to quick to see clearly and i have not yet been able to find the logs.

IF I can get the system to cooperate I will post the output of sostat here.

However any ideas would be greatly appreciated.

--
You received this message because you are subscribed to the Google Groups "security-onion" group.
To post to this group, send email to securit...@googlegroups.com.
To unsubscribe from this group, send email to security-onio...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion?hl=en-US.

[Heine Lysemose]

Hi Jake

Could you tell a bit about your system specs while we wait for an sostat.

CPU/cores
RAM
Disk

Did you use the ISO or added the PPA manually?

/Lysemose

[Scott Runnels]

Did you update MySQL?  We have a known issue where the recent MySQL issue can put your machine at load.

https://code.google.com/p/security-onion/wiki/MySQLUpdates

Scott Runnels

To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.

To post to this group, send email to securit...@googlegroups.com.

[Doug Burks]

Hi Harvii,

Did you update MySQL recently? Have you seen our recommendations for
MySQL updates?
https://code.google.com/p/security-onion/wiki/MySQLUpdates

Hope that helps!

Thanks,
Doug

On Tue, Mar 19, 2013 at 8:41 AM, Harvii Dent <harvi...@gmail.com> wrote:
> On Monday, January 28, 2013 11:35:47 PM UTC+3, Jake Sallee wrote:
>> UPDATE:
>>
>> After carefully reading the upgrade instructions I realize what mistake I made. The CPU utilization seems to have been caused by the bug that was posted on the SO blog.
>>
>> I could not get the system responsive enough to run the commands and eventually just re-installed the SO distro from the iso and ran ALL the updates BEFORE running the sosetup script.
>>
>> Worked like a charm.
>>
>> I am monitoring the box carefully now but all indications are good.
>>
>> Thank you to everyone who helped.
>
> Hello, I'm seeing this same behavior on two of my four sensors, you mentioned that it's caused by a bug, which bug is that?
>
> Thanks

>
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion?hl=en-US.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Doug Burks
http://securityonion.blogspot.com

[Doug Burks]

Please send the output of the following (redacting sensitive info as necessary):
sudo sostat

Thanks,
Doug

On Tue, Mar 19, 2013 at 8:57 AM, Harvii Dent <harvi...@gmail.com> wrote:
> Hi Doug,
>
> Yes, I've followed the MySQL update recommendations.
>
> The weird thing is that it's happening on only two sensors, the other two are fine.
> I'm not sure where to start troubleshooting it, any suggestions are welcome.

[Doug Burks]

I don't see 100% CPU utilization in your sostat output.
Doug

On Tue, Mar 19, 2013 at 9:41 AM, Harvii Dent <harvi...@gmail.com> wrote:

> Here it is attached.

[Doug Burks]

I'm not aware of any CPU issues with the cron jobs.
Doug

On Tue, Mar 19, 2013 at 9:53 AM, Harvii Dent <harvi...@gmail.com> wrote:
> I had to force reboot the sensor in order to be able to run the command as it wasn't responding, it's stable at the moment.
>
> I've been looking at the CPU utilization in VMware and the issue seems to start around midnight, I'll be able to confirm this tomorrow; could one of the cron jobs be the cause?

[Doug Burks]

It just creates high CPU usage until the box is rebooted. Since
you've rebooted, the issue should be resolved.
Doug

On Tue, Mar 19, 2013 at 10:21 AM, Harvii Dent <harvi...@gmail.com> wrote:
> Hi Doug,
>
> It seems your initial suggestion was correct; I just went over my command history and APT installation history and found that I didn't follow the MySQL update procedure exactly as described on the sensors that have the problem, I issued:
> sudo apt-get update && sudo apt-get dist-upgrade
>
> when it should have been:
> sudo apt-get update && sudo apt-get install mysql-server mysql-server-core-5.5 mysql-server-5.5
>
> Is there a way to fix this or am I looking at a complete reinstall? :)
>
> Thanks

[Doug Burks]

On Wed, Mar 20, 2013 at 4:19 AM, Harvii Dent <harvi...@gmail.com> wrote:
> Hello again,

Hi Harvii,

> The high CPU usage issue is still happening so now I'm considering reinstalling the sensor, even though I'd prefer to fix it (if anyone knows how to do that then please let me know).

We can try to help you fix it if you provide further information.
What exactly is "high CPU usage"? What process(es) are using the CPU?
Now that the high CPU usage has started again, please send your "sudo
sostat" output again (redacting sensitive info as necessary).

> A couple of questions:
> 1. Is it safe to just run the SO setup again or should I format the machine and start fresh?

You should be able to just re-run sosetup, but re-installing the whole
OS shouldn't take much longer, so it's up to you.

> 2. What exactly would be lost in this process? Doug mentioned that "Pcaps, Bro logs, argus data, and raw OSSEC logs" remain on the sensor, is there anything else?

If you re-run sosetup on the sensor, it will wipe *all* NSM config AND
data (including pcaps, Bro logs, argus data, raw OSSEC logs, etc.).

Doug

> Thanks
>
>
> On Tuesday, March 19, 2013 6:06:15 PM UTC+3, Harvii Dent wrote:
>> Unfortunately this isn't a one-time thing, this maybe the 4th time that I had to force reboot the sensor.
>>
>> I'll post again tomorrow if it still happens.
>>
>> Thanks for your help Doug.

[Doug Burks]

You might want to include a timestamp so you can see exactly when the
problem starts.
Doug

On Wed, Mar 20, 2013 at 11:48 AM, Harvii Dent <harvi...@gmail.com> wrote:
> Hi Doug,
>

> Unfortunately when the problem happens the whole machine becomes unresponsive so I can't run "sostat" or even "top".
>
> I was thinking to maybe log the running processes and their CPU usage to a file so that when the problem happens at least I could have an idea of what was running, maybe something like this as a cron job:
> ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 >> logfile.txt
>
> what do you think?
>
> Thanks

[Martin Holste]

Yes, that's probably the culprit.  If you see a lot of Perl running around, it's probably my fault.  Issue killall -9 perl to quickly get the system back.  Is this still happening on the latest SO package?

On Sat, Mar 23, 2013 at 7:19 AM, Harvii Dent <harvi...@gmail.com> wrote:

Hi Doug,

Here is the log file; everything was fine until "Thu Mar 21 01:31:47 UTC 2013" when 9 out of the top 10 where 'perl' processes, after that even the logging became inconsistent until "Thu Mar 21 12:40:01 UTC 2013" when the machine was rebooted.

Could the incorrect method when updating MySQL be causing this to ELSA (the perl processes)?

Thanks

[Martin Holste]

Is this on an SO box?  I'm not sure that these updates are compatible with SO yet, though I'm interested in your experience.  There are a few additional Perl modules that would need to be installed.

Reload would fire off new Perl scripts as syslog-ng starts back up, so that's surely the trigger, but the new code shouldn't fire up the livetail forks, which have historically been the issue.  It's possible that your config is in a broken state, though.  So, shut down syslog-ng entirely, and look to see if any Perl processes are running.  Kill all you find.  To test ELSA, go to /opt/elsa/node and issue echo "" | perl elsa.pl -o and see if it runs to completion without an error.  Warnings may show up, but that's usually expected for this type of run. 

On Mon, Mar 25, 2013 at 8:16 AM, Harvii Dent <harvi...@gmail.com> wrote:

Hello Martin,

I've installed all the available updates but that did not fix the problem, however I think I've figured out what triggers it; it seems to be the command "/etc/init.d/syslog-ng reload" in the "sensor-newday" cron job.

Something breaks when syslog-ng reloads, any idea what it could be?

Thanks

On Saturday, March 23, 2013 9:23:55 PM UTC+3, Harvii Dent wrote:
> Hi Martin,
>
>
>
> I'm not sure at the moment if the system is running the latest packages, if not I'll update them and see if it still happens and report back.
>
>
>
> Thanks

>
>
>
> On Saturday, March 23, 2013 7:19:00 PM UTC+3, Martin wrote:
>
> > Yes, that's probably the culprit.  If you see a lot of Perl running around, it's probably my fault.  Issue killall -9 perl to quickly get the system back.  Is this still happening on the latest SO package?

[Martin Holste]

Ok, that directory should be valid, then (no errors, just warnings).  So after you verify that no Perl processes are running go ahead and start syslog-ng with service syslog-ng start.  On a happier note, I've commited my latest code upstream in ELSA so eventually these kinds of things will be history as SO catches up.

On Tue, Mar 26, 2013 at 2:15 AM, Harvii Dent <harvi...@gmail.com> wrote:

Hi Martin,

Yes, this is a SO box, and the updates that I installed are only the ones that are available through the package manager, nothing more.

These are the results of running the command 'echo "" | perl elsa.pl -o':

Validating directory...
Use of uninitialized value $db_size in concatenation (.) or string at /opt/elsa/node/Indexer.pm line 138.
Use of uninitialized value in numeric gt (>) at /opt/elsa/node/Indexer.pm line 532.
Use of uninitialized value $db_size in concatenation (.) or string at /opt/elsa/node/Indexer.pm line 191.
Use of uninitialized value $db_size in addition (+) at /opt/elsa/node/Indexer.pm line 211.
Running once
Use of uninitialized value $line[3] in string eq at /opt/elsa/node/Reader.pm line 204, <STDIN> line 1.

Thanks

[Doug Burks]

Hi Harvii,

The syslog-ng reload is there to pick up the new OSSEC archives.log
file after it gets rotated at midnight. If you don't care about
getting your OSSEC raw logs into ELSA, you can leave it disabled (but
be aware that it will be re-enabled whenever we update the
securityonion-nsmnow-admin-scripts package).

Thanks,
Doug

On Wed, Mar 27, 2013 at 8:59 AM, Harvii Dent <harvi...@gmail.com> wrote:
> Thanks Martin, it seems to be running fine now.
>
> Can I leave the cron job that reloads syslog-ng disabled until the changes are available for SO or does it have a purpose other than rotating the logs?

Doug Burks
http://securityonion.blogspot.com