CPU 100% with multipule PERL processes
Shawn
top - 20:11:40 up 1:54, 1 user, load average: 68.54, 68.55, 67.74
Tasks: 681 total, 69 running, 612 sleeping, 0 stopped, 0 zombie
Cpu(s): 76.2%us, 23.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65937644k total, 15113404k used, 50824240k free, 207984k buffers
Swap: 91562096k total, 0k used, 91562096k free, 6092020k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9178 root 20 0 287m 89m 5644 R 50 0.1 60:53.94 perl
10255 root 20 0 287m 89m 5644 R 50 0.1 53:32.50 perl
14936 root 20 0 287m 89m 5644 R 50 0.1 26:17.14 perl
15385 root 20 0 287m 89m 5644 R 50 0.1 26:21.64 perl
9063 root 20 0 287m 89m 5644 R 50 0.1 61:55.43 perl
9492 root 20 0 287m 89m 5644 R 50 0.1 59:26.97 perl
9595 root 20 0 287m 89m 5640 R 50 0.1 57:37.93 perl
11988 root 20 0 287m 89m 5644 R 50 0.1 40:22.50 perl
12215 root 20 0 287m 89m 5644 R 50 0.1 38:57.10 perl
13839 root 20 0 287m 89m 5644 R 50 0.1 27:58.07 perl
15438 root 20 0 287m 89m 5644 R 50 0.1 26:02.65 perl
8528 root 20 0 287m 89m 5644 R 50 0.1 69:31.55 perl
8944 root 20 0 287m 89m 5640 R 50 0.1 63:40.57 perl
10468 root 20 0 287m 89m 5644 R 50 0.1 51:37.12 perl
12907 root 20 0 287m 89m 5644 R 50 0.1 35:22.41 perl
13434 root 20 0 287m 89m 5644 R 49 0.1 31:43.28 perl
15749 root 20 0 287m 89m 5644 R 49 0.1 23:56.58 perl
9707 root 20 0 287m 89m 5644 R 49 0.1 56:57.07 perl
12108 root 20 0 287m 89m 5644 R 49 0.1 39:37.18 perl
12428 root 20 0 287m 89m 5644 R 49 0.1 37:15.21 perl
8234 root 20 0 287m 89m 5644 R 48 0.1 73:37.26 perl
8575 root 20 0 287m 89m 5644 R 48 0.1 68:27.26 perl
8815 root 20 0 287m 89m 5644 R 48 0.1 65:27.51 perl
Im also seeing this when I run a sostat
eth3 Link encap:Ethernet HWaddr ac:16:2d:79:60:5f
UP BROADCAST RUNNING NOARP PROMISC MULTICAST MTU:1500 Metric:1
--More--ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
ERROR 1040 (HY000): Too many connections
=========================================================================
CPU Usage
=========================================================================
Load average for the last 1, 5, and 15 minutes:
68.70 68.58 67.78
Processing units: 32
If load average is higher than processing units,
then tune until load average is lower than processing units.
top - 20:12:04 up 1:55, 1 user, load average: 68.70, 68.58, 67.78
Tasks: 687 total, 69 running, 618 sleeping, 0 stopped, 0 zombie
Cpu(s): 64.9%us, 20.7%sy, 0.0%ni, 14.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 65937644k total, 15174424k used, 50763220k free, 208120k buffers
Swap: 91562096k total, 0k used, 91562096k free, 6096372k cached
%CPU %MEM COMMAND
68.1 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
67.7 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
67.0 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
67.0 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
66.8 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
66.5 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
66.0 0.1 perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf
Doug Burks
Have you tried rebooting? Do the processes come back after a reboot?
Are you running any ELSA archive queries?
Have you configured any ELSA alerts (scheduled queries)?
> *The information contained in or attached to this email is strictly
> confidential. If you are not the intended recipient, please notify us
> immediately by telephone and return the message to us.*
>
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com
Shawn
> Hi Shawn,
>
> Have you tried rebooting? Do the processes come back after a reboot?
>
> Are you running any ELSA archive queries?
>
> Have you configured any ELSA alerts (scheduled queries)?
>
> > Good afternoon has anyone seen a SO that is configured as a server spin up multiple Perl instances until it maxed the CPU?
About an hour ago I updated the server and rebooted. The problem came back within a few minutes of the reboot.
The problem seemed to start after I ran an archive query yesterday afternoon. The query never finished.
I do not have any ELSA alerts created.
Thanks.
Doug Burks
> About an hour ago I updated the server and rebooted. The problem came back within a few minutes of the reboot.
> The problem seemed to start after I ran an archive query yesterday afternoon. The query never finished.
interface (ELSA --> Active Queries)?
Shawn Wiley.ext
--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/xqMrSmnBng4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/d/optout.
Shawn Wiley | Production Information Security Engineer| ULLINK | T: +1 646 565 6603 | M: +1 347 759 1750
|F: +1 212 883 9440| 11 Times Square, 31st fl. | New York, NY 10036 | shawn.w...@ullink.com | http://www.ullink.com
Doug Burks
https://groups.google.com/d/topic/security-onion/xF9v3RfNnaM/discussion
> "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.
--
Shawn W
Seems to be back to normal now. How did a large/vague query hang up perl like that?
Thanks,
Shawn
Shawn W
Doug Burks
to serve the request. I'd try going through the troubleshooting steps
again and see if you can find anything else in your database or logs
that is causing this to re-occur.
On Wed, Jul 29, 2015 at 6:42 PM, Shawn W <slw...@gmail.com> wrote:
> It looks like the problem resurfaced fairly quickly. I have noticed a few other issues. First Siguil is showing up as failed. and ELSA shows this when trying to get to the main elsa query page. Please see attachments.
>
Magnus Wild
I did try to run the perl-script with the parameters described manually though, and found that we are getting some error output. I have attached the resulting output below. I ran "perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf".
Doug Burks
> Hello! We are also seeing similar problems after an upgrade yesterday. I have verified that we have the pid-column, as described in https://groups.google.com/forum/#!topic/security-onion/xF9v3RfNnaM/discussion, and we are still seeing the issue after rebooting and bouncing the services multiple times. I have disabled the elsa-crob-job as described in the previous discussion as well to avoid having the perl-processes spawned indefinetly.
>
> I did try to run the perl-script with the parameters described manually though, and found that we are getting some error output. I have attached the resulting output below. I ran "perl /opt/elsa/web/cron.pl -c /etc/elsa_web.conf".
Please start a separate thread to troubleshoot your issues.
Thanks!
Magnus Wild
All right, i will create a new thread describing our issues. Thanks!
Shawn
As long as I leave the elsa line commented out in the cron.d file I run error free. Once I uncomment the line the perl processes start backing up and it brings the box to a crawl.
Doug Burks
leave it disabled permanently.
Please take a look at the logs in /nsm/elsa/data/elsa/log/ for
additional clues so that you can resolve the root issue and re-enable
the cron job.
For example, please see Magnus's resolution here:
https://groups.google.com/d/topic/security-onion/mDRRgcfxJA0/discussion
On Fri, Jul 31, 2015 at 5:58 PM, Shawn <shawn.w...@ullink.com> wrote:
> UPDATE
> As long as I leave the elsa line commented out in the cron.d file I run error free. Once I uncomment the line the perl processes start backing up and it brings the box to a crawl.
> *The information contained in or attached to this email is strictly
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/d/optout.