Disappearing ELSA nodes with "500 - Internal Server Error" returned from APIs

492 views
Skip to first unread message

Pietro Delsante

unread,
Jan 16, 2014, 7:02:30 AM1/16/14
to securit...@googlegroups.com
Hello,

In the last few days I've noticed some problems with remote ELSA nodes (peers). I'm using the latest Elsa packages from SO's repos and I regularly update them with soup (no other updates available ATM).

My current setup has one SO server and one remote sensor which is connected to my home network through an OpenVPN link using TCP (used only for admin traffic, that is: me connecting to the sensor via SSH) and via SO's AutoSSH connection, which uses my server's public IP (so, it does not pass through OpenVPN). I also tried configuring AutoSSH to use the server's private IP inside the VPN tunnel and I did not notice changes, so I switched back to the public IP to avoid nesting TCP tunnels.

The problem is that, once every few hours, Elsa's GUI reports that it only has 1 active node with no archived logs at all.

I've set up a Nagios check to monitor the remote sensor's Elsa API port (50000) published locally on the server via AutoSSH, like this:

date=$(date +%s); /usr/lib/nagios/plugins/check_http -H localhost -p 50000 -u "/API/stats" -k "Authorization: ApiKey elsa:$date:$(echo -n ${date}<my_api_key> | sha512sum | cut -d ' ' -f 1)"

It basically performs a call to /API/stats with the correct Authorization header. I noticed that, when the node disappears from Elsa's GUI, the Nagios check is returning a "500 - Internal Server Error".

The matter is that I still don't have a clue about what's causing this error:

- /var/log/apache2/error.log is empty, apart from the normal shutdown/restart logs

- /nsm/elsa/data/elsa/log/web.log only contains this:

#######################################
* ERROR [2014/01/16 11:05:17] /opt/elsa/web/lib/Utils.pm (679) Utils::__ANON__ 19374 [undef]
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "Internal Server Erro...") at /opt/elsa/web/lib/Utils.pm line 674.

Header: $VAR1 = {
'URL' => 'http://127.0.0.1:3154/API/local_info',
'connection' => 'keep-alive',
'Status' => '500',
'HTTPVersion' => '1.1',
'date' => 'Thu, 16 Jan 2014 11:05:17 GMT',
'content-length' => '21',
'Reason' => 'Internal Server Error',
'content-type' => 'text/plain'
};

body: $VAR1 = 'Internal Server Error';
#######################################

- the other logs in /nsm/elsa/data/elsa/log/ do not show any error messages and I really don't know how to find out any other info...

Regards,
Pietro

Martin Paszkiewicz

unread,
Jan 16, 2014, 11:20:32 AM1/16/14
to securit...@googlegroups.com
Hi Pietro,
I wanted to chime in since I was having similar issues however my environment setup is different. I have a standalone installation opposed to server/sensor setup.

Did you check the following topic?

-----

https://groups.google.com/forum/#!topic/security-onion/PhXd7WrQoZU

-----

Doug assisted me by having me adjust the query_timeout value in /etc/elsa_web.conf

Have you tried adjusting that value to something higher? For me setting it to 3000 resolved the issues for me. Your miles may vary...

Kind regards,
Martin Paszkiewicz

Pietro Delsante

unread,
Jan 17, 2014, 4:08:26 AM1/17/14
to securit...@googlegroups.com
Hello Martin,

Thanks for your suggestion, I hadn't thought about the timeout but you're right this might be somehow involved. I had checked out your thread but I thougt it was a different problem.

I've raised the query_timeout value with the amount you suggested, let's see what happens :-)

Regards,
Pietro Delsante


2014/1/16 Martin Paszkiewicz <zulun...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/Wu0sdQfqMk0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
To post to this group, send email to securit...@googlegroups.com.
Visit this group at http://groups.google.com/group/security-onion.
For more options, visit https://groups.google.com/groups/opt_out.

Pietro Delsante

unread,
Feb 8, 2014, 11:06:16 AM2/8/14
to securit...@googlegroups.com
On Friday, January 17, 2014 10:08:26 AM UTC+1, Pietro Delsante wrote:
> Hello Martin,
>
> Thanks for your suggestion, I hadn't thought about the timeout but you're right this might be somehow involved. I had checked out your thread but I thougt it was a different problem.
>
>
> I've raised the query_timeout value with the amount you suggested, let's see what happens :-)
>
> Regards,
> Pietro Delsante
>
>
>
>
> 2014/1/16 Martin Paszkiewicz
>
> Hi Pietro,
>
>       I wanted to chime in since I was having similar issues however my environment setup is different.  I have a standalone installation opposed to server/sensor setup.
>
>
>
> Did you check the following topic?
>
>
>
> -----
>
>
>
> https://groups.google.com/forum/#!topic/security-onion/PhXd7WrQoZU
>
>
>
> -----
>
>
>
> Doug assisted me by having me adjust the query_timeout value in /etc/elsa_web.conf
>
>
>
> Have you tried adjusting that value to something higher? For me setting it to 3000 resolved the issues for me.  Your miles may vary...
>
>
>
> Kind regards,
>
> Martin Paszkiewicz
>
>
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "security-onion" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/security-onion/Wu0sdQfqMk0/unsubscribe.
>
> To unsubscribe from this group and all its topics, send an email to security-onio...@googlegroups.com.
>
> To post to this group, send email to securit...@googlegroups.com.
>
> Visit this group at http://groups.google.com/group/security-onion.
>
> For more options, visit https://groups.google.com/groups/opt_out.

Hello,

Raising the timeout does not help. The culprit seems to be starman on the sensor, see also this post: https://groups.google.com/forum/#!topic/security-onion/VSGCRPleGLQ

My boxes are under Nagios monitoring and the strange things are:

1. The number of starman worker processes does not change when the error appears, so I don't think it's the linux kernel killing them;
2. There are no memory issues registered on Nagios when the problem occurrs, the box never swaps and there's still a good amount of free ram;

Restarting starman always brings the node back to life in Elsa.

The attached file contains the output of sostat-redacted taken from my sensor. Starman's error log does not seem to contain anything strange either.

Regards,
Pietro
so_sensor.txt

Pietro Delsante

unread,
Feb 8, 2014, 11:13:41 AM2/8/14
to securit...@googlegroups.com
By the way, Starman's error log does not seem to be rotated, wouldn't it be better to add it to /etc/logrotate.d/elsa or to give it its own logrotate conf file?

Doug Burks

unread,
Feb 8, 2014, 1:48:52 PM2/8/14
to securit...@googlegroups.com
On Sat, Feb 8, 2014 at 11:06 AM, Pietro Delsante
<pietro....@gmail.com> wrote:
> Raising the timeout does not help.

Just to confirm, did you increase the query_timeout value in
/etc/elsa_web.conf on the server AND all sensors running starman?

Did you then restart Apache/starman on all boxes?

What did you set the query_timeout to?


--
Doug Burks

Pietro Delsante

unread,
Feb 9, 2014, 6:13:26 AM2/9/14
to securit...@googlegroups.com
On Saturday, February 8, 2014 7:48:52 PM UTC+1, Doug Burks wrote:
>
> Just to confirm, did you increase the query_timeout value in
>
> /etc/elsa_web.conf on the server AND all sensors running starman?
>
> Did you then restart Apache/starman on all boxes?
>
> What did you set the query_timeout to?


This setup only has one sensor and one server, and now both of them have:

"query_timeout": 12000

However, this problem does not seem to involve the server's settings at all, since the "500 - Internal Server Error" message is shown even when I check the APIs status through Nagios, which does not involve Elsa on the collector as it queries "/API/stats", as shown below:

date=$(date +%s); /usr/lib/nagios/plugins/check_http -H localhost -p 50000 -u "/API/stats" -k "Authorization: ApiKey elsa:$date:$(echo -n ${date}<my_api_key> | sha512sum | cut -d ' ' -f 1)"

This Nagios check runs on the server, but I have the same result if I run it on the sensor on localhost:3154.

Regards,
Pietro

Doug Burks

unread,
Feb 10, 2014, 6:39:43 AM2/10/14
to securit...@googlegroups.com
I haven't experienced this starman issue on my production deployments.
How often does Nagios check the status? Is it possible that Nagios
is overwhelming starman?
> --
> You received this message because you are subscribed to the Google Groups "security-onion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to security-onio...@googlegroups.com.
> To post to this group, send email to securit...@googlegroups.com.
> Visit this group at http://groups.google.com/group/security-onion.
> For more options, visit https://groups.google.com/groups/opt_out.



--
Doug Burks

Pietro Delsante

unread,
Feb 10, 2014, 9:29:47 AM2/10/14
to securit...@googlegroups.com
On Monday, February 10, 2014 12:39:43 PM UTC+1, Doug Burks wrote:s.

> How often does Nagios check the status? Is it possible that Nagios
> is overwhelming starman?

I doubt so, Nagios check period defaults to 5 minutes which seems fair enough; moreover, I'm querying the stats api, which is quite fast and should not have much impact on performances. In addition, the problem started showing up way before I added this check to Nagios.

By the way, I found out that, when the issue shows up, it's because one of the starman workers becomes unresponsive. This is what I did today, when I found out that the APIs were returning a 500:

First, I manually ran the check and saw it was returning an error:

###############################
root@sensor-XXXXXXX:~# date=$(date +%s); /usr/lib/nagios/plugins/check_http -H localhost -p 3154 -u "/API/stats" -k "Authorization: ApiKey elsa:$date:$(echo -n ${date}<APIKEY> | sha512sum | cut -d ' ' -f 1)"
HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 161 bytes in 0.002 second response time |time=0.001879s;;;0.000000 size=161B;;;0
root@sensor-XXXXXXX:~#
###############################

I then checked starman's processes, and I found them all running:

###############################
root@sensor-XXXXXXX:~# ps aux | grep starman
root 2223 0.0 0.0 8116 952 pts/1 S+ 12:09 0:00 grep --color=auto starman
www-data 9658 0.0 0.3 441948 232896 ? S Feb09 0:24 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 12756 6.7 0.2 375340 166400 ? R Feb09 68:28 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 15595 0.0 0.5 560428 351720 ? S Feb09 0:28 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 17486 0.0 0.1 335796 126960 ? S Feb09 0:19 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 25570 0.0 0.3 418096 208588 ? S Feb09 0:24 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 26897 0.0 0.0 55420 9644 ? Ss Feb07 0:05 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
root@sensor-XXXXXXX:~#
###############################

After that, I tried to restart starman:

###############################
root@sensor-XXXXXXX:~# service starman restart
Restarting Starman:
root@sensor-XXXXXXX:~#
###############################

Tried to run the check again and this time I got a timeout, so starman hadn't come up correctly:

###############################
root@sensor-XXXXXXX:~# date=$(date +%s); /usr/lib/nagios/plugins/check_http -H localhost -p 3154 -u "/API/stats" -k "Authorization: ApiKey elsa:$date:$(echo -n ${date}d049a56006d9e15b0070ddb874147ae4 | sha512sum | cut -d ' ' -f 1)"
CRITICAL - Socket timeout after 10 seconds
root@sensor-XXXXXXX:~#
###############################

Checked starman's processes again and I found out that one process hand't died:

###############################
root@sensor-XXXXXXX:~# ps aux | grep starman
root 3509 0.0 0.0 8116 952 pts/1 S+ 12:10 0:00 grep --color=auto starman
www-data 12756 6.7 0.2 375340 166400 ? R Feb09 69:00 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
root@sensor-XXXXXXX:~#
###############################

Tried to stop starman using the init script but it did not do the trick, probably because Starman's main process had already died, while the hung one was a worker:

###############################
root@sensor-XXXXXXX:~# service starman stop
Stopping Starman: /etc/init.d/starman: line 43: kill: (26897) - No such process

root@sensor-XXXXXXX:~# ps aux | grep starman
root 3523 0.0 0.0 8116 952 pts/1 S+ 12:10 0:00 grep --color=auto starman
www-data 12756 6.8 0.2 375340 166400 ? R Feb09 69:18 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
###############################

Manually killing the remaining process and starting starman did the trick:

###############################
root@sensor-XXXXXXX:~# kill 12756
root@sensor-XXXXXXX:~# ps aux | grep starman
root 3525 0.0 0.0 8116 952 pts/1 S+ 12:10 0:00 grep --color=auto starman
root@sensor-XXXXXXX:~# service starman start
Starting Starman: Pid_file "/var/run/starman.pid" already exists. Overwriting!

root@sensor-XXXXXXX:~# ps aux | grep starman
www-data 3530 0.0 0.0 55420 9624 ? Ss 12:10 0:00 starman master -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 3531 68.0 0.1 315240 107924 ? S 12:10 0:02 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 3532 68.5 0.1 315240 107928 ? S 12:10 0:02 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 3533 69.5 0.1 315140 107928 ? S 12:10 0:02 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 3534 68.0 0.1 315140 107928 ? S 12:10 0:02 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
www-data 3535 68.5 0.1 315240 107928 ? S 12:10 0:02 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi
root 3556 0.0 0.0 8120 952 pts/1 S+ 12:11 0:00 grep --color=auto starman
root@sensor-XXXXXXX:~#
###############################

So, the problem seems to be one of the starman processes becoming unresponsive. Is there anything I can do to further debug this? Is there any reason (performances or whatever) why the APIs are using starman instead of Apache2?

Regards,
Pietro

Doug Burks

unread,
Feb 10, 2014, 9:36:13 AM2/10/14
to securit...@googlegroups.com
On Mon, Feb 10, 2014 at 9:29 AM, Pietro Delsante
<pietro....@gmail.com> wrote:
> After that, I tried to restart starman:
>
> ###############################
> root@sensor-XXXXXXX:~# service starman restart
> Restarting Starman:
> root@sensor-XXXXXXX:~#
> ###############################
>
> Tried to run the check again and this time I got a timeout, so starman hadn't come up correctly:

I think there is a known issue with the starman "restart" function.
As you can see when we released the ELSA 1.5 packages, we recommended
that folks do a "stop" and then do a "start" instead of doing a
"restart":
http://blog.securityonion.net/2013/12/bro-22-and-elsa-15-packages-now.html

stop/start (NOT restart) starman on each sensor:
sudo service starman stop
sudo service starman start

> So, the problem seems to be one of the starman processes becoming unresponsive. Is there anything I can do to further debug this? Is there any reason (performances or whatever) why the APIs are using starman instead of Apache2?

Scott and Martin may be able to provide more background here, but I
think the main goal of using starman was to limit the resource usage
on sensors since starman should be more lightweight than Apache.


--
Doug Burks

BBCan177

unread,
Feb 10, 2014, 10:03:53 AM2/10/14
to securit...@googlegroups.com
On Monday, February 10, 2014 9:36:13 AM UTC-5, Doug Burks wrote:
>
> I think there is a known issue with the starman "restart" function.
> As you can see when we released the ELSA 1.5 packages, we recommended
> that folks do a "stop" and then do a "start" instead of doing a
> "restart":
>
> http://blog.securityonion.net/2013/12/bro-22-and-elsa-15-packages-now.html
>
>
>
> stop/start (NOT restart) starman on each sensor:
>
> sudo service starman stop
>
> sudo service starman start
>
>
>
> > So, the problem seems to be one of the starman processes becoming unresponsive. Is there anything I can do to further debug this? Is there any reason (performances or whatever) why the APIs are using starman instead of Apache2?
>
>
>
> Scott and Martin may be able to provide more background here, but I
> think the main goal of using starman was to limit the resource usage
> on sensors since starman should be more lightweight than Apache.

My Sensors Starman crashed again. Restarting Starman brings it back.


With the new SOSTAT, this is the output of the new ELSA section


=========================================================================
ELSA
=========================================================================
Syslog-ng
Checking for process:
1500 supervising syslog-ng
1501 /usr/sbin/syslog-ng -p /var/run/syslog-ng.pid
Checking for connection:
Connection to localhost 514 port [tcp/shell] succeeded!

MySQL
Checking for process:
1554 /usr/sbin/mysqld
Checking for connection:
Connection to localhost 50000 port [tcp/*] succeeded!

Sphinx
Checking for process:
1542 su -s /bin/sh -c exec "$0" "$@" sphinxsearch -- /usr/bin/searchd --nodetach
Checking for connection:
Connection to localhost 9306 port [tcp/*] succeeded!

ELSA Buffers in Queue:
-rw-r--r-- 1 root root 1442 Feb 10 14:58 /nsm/elsa/data/elsa/tmp/buffers/1392044312.60106
-rw-r--r-- 1 root root 67274 Feb 10 14:58 /nsm/elsa/data/elsa/tmp/buffers/1392044252.51847
-rw-r--r-- 1 root root 36701 Feb 10 14:57 /nsm/elsa/data/elsa/tmp/buffers/1392044192.35206
-rw-r--r-- 1 root root 51591 Feb 10 14:56 /nsm/elsa/data/elsa/tmp/buffers/1392044132.29418
-rw-r--r-- 1 root root 19210 Feb 10 14:55 /nsm/elsa/data/elsa/tmp/buffers/1392044072.23614
-rw-r--r-- 1 root root 52615 Feb 10 14:54 /nsm/elsa/data/elsa/tmp/buffers/1392044012.15217
-rw-r--r-- 1 root root 51842 Feb 10 14:53 /nsm/elsa/data/elsa/tmp/buffers/1392043952.07836
-rw-r--r-- 1 root root 71387 Feb 10 14:52 /nsm/elsa/data/elsa/tmp/buffers/1392043892.01457
-rw-r--r-- 1 root root 127967 Feb 10 14:51 /nsm/elsa/data/elsa/tmp/buffers/1392043831.93361
-rw-r--r-- 1 root root 92275 Feb 10 14:50 /nsm/elsa/data/elsa/tmp/buffers/1392043771.85841
-rw-r--r-- 1 root root 54227 Feb 10 14:49 /nsm/elsa/data/elsa/tmp/buffers/1392043711.77573
-rw-r--r-- 1 root root 40269 Feb 10 14:48 /nsm/elsa/data/elsa/tmp/buffers/1392043651.67633
-rw-r--r-- 1 root root 19989 Feb 10 14:47 /nsm/elsa/data/elsa/tmp/buffers/1392043591.57791
-rw-r--r-- 1 root root 20620 Feb 10 14:46 /nsm/elsa/data/elsa/tmp/buffers/1392043531.54707
-rw-r--r-- 1 root root 37139 Feb 10 14:45 /nsm/elsa/data/elsa/tmp/buffers/1392043471.51896
-rw-r--r-- 1 root root 39326 Feb 10 14:44 /nsm/elsa/data/elsa/tmp/buffers/1392043411.44503
-rw-r--r-- 1 root root 27219 Feb 10 14:43 /nsm/elsa/data/elsa/tmp/buffers/1392043351.33881
-rw-r--r-- 1 root root 38400 Feb 10 14:42 /nsm/elsa/data/elsa/tmp/buffers/1392043291.26458
-rw-r--r-- 1 root root 18493 Feb 10 14:41 /nsm/elsa/data/elsa/tmp/buffers/1392043231.21598
-rw-r--r-- 1 root root 25776 Feb 10 14:40 /nsm/elsa/data/elsa/tmp/buffers/1392043171.14947
-rw-r--r-- 1 root root 69609 Feb 10 14:39 /nsm/elsa/data/elsa/tmp/buffers/1392043111.07486
-rw-r--r-- 1 root root 129647 Feb 10 14:38 /nsm/elsa/data/elsa/tmp/buffers/1392043051.00874
-rw-r--r-- 1 root root 165648 Feb 10 14:37 /nsm/elsa/data/elsa/tmp/buffers/1392042990.87574
-rw-r--r-- 1 root root 39973 Feb 10 14:36 /nsm/elsa/data/elsa/tmp/buffers/1392042930.81002
-rw-r--r-- 1 root root 40141 Feb 10 14:35 /nsm/elsa/data/elsa/tmp/buffers/1392042870.76817
-rw-r--r-- 1 root root 38516 Feb 10 14:34 /nsm/elsa/data/elsa/tmp/buffers/1392042810.70266
-rw-r--r-- 1 root root 25717 Feb 10 14:33 /nsm/elsa/data/elsa/tmp/buffers/1392042750.64565
-rw-r--r-- 1 root root 36279 Feb 10 14:32 /nsm/elsa/data/elsa/tmp/buffers/1392042690.57976
-rw-r--r-- 1 root root 22675 Feb 10 14:31 /nsm/elsa/data/elsa/tmp/buffers/1392042630.54796
-rw-r--r-- 1 root root 32813 Feb 10 14:30 /nsm/elsa/data/elsa/tmp/buffers/1392042570.44487
-rw-r--r-- 1 root root 61696 Feb 10 14:29 /nsm/elsa/data/elsa/tmp/buffers/1392042510.4204
-rw-r--r-- 1 root root 87409 Feb 10 14:28 /nsm/elsa/data/elsa/tmp/buffers/1392042450.35985
-rw-r--r-- 1 root root 36575 Feb 10 14:27 /nsm/elsa/data/elsa/tmp/buffers/1392042390.2858
-rw-r--r-- 1 root root 34066 Feb 10 14:26 /nsm/elsa/data/elsa/tmp/buffers/1392042330.22812
-rw-r--r-- 1 root root 26676 Feb 10 14:25 /nsm/elsa/data/elsa/tmp/buffers/1392042270.1618
-rw-r--r-- 1 root root 26965 Feb 10 14:24 /nsm/elsa/data/elsa/tmp/buffers/1392042210.10466
-rw-r--r-- 1 root root 34239 Feb 10 14:23 /nsm/elsa/data/elsa/tmp/buffers/1392042150.03899
-rw-r--r-- 1 root root 61295 Feb 10 14:22 /nsm/elsa/data/elsa/tmp/buffers/1392042089.99724
-rw-r--r-- 1 root root 51049 Feb 10 14:21 /nsm/elsa/data/elsa/tmp/buffers/1392042029.95555
-rw-r--r-- 1 root root 49701 Feb 10 14:20 /nsm/elsa/data/elsa/tmp/buffers/1392041969.88716
-rw-r--r-- 1 root root 89491 Feb 10 14:19 /nsm/elsa/data/elsa/tmp/buffers/1392041909.83615
-rw-r--r-- 1 root root 66820 Feb 10 14:18 /nsm/elsa/data/elsa/tmp/buffers/1392041849.80248
-rw-r--r-- 1 root root 23969 Feb 10 14:17 /nsm/elsa/data/elsa/tmp/buffers/1392041789.76178
-rw-r--r-- 1 root root 28853 Feb 10 14:16 /nsm/elsa/data/elsa/tmp/buffers/1392041729.67074
-rw-r--r-- 1 root root 22932 Feb 10 14:15 /nsm/elsa/data/elsa/tmp/buffers/1392041669.60016
-rw-r--r-- 1 root root 33708 Feb 10 14:14 /nsm/elsa/data/elsa/tmp/buffers/1392041609.54237
-rw-r--r-- 1 root root 18626 Feb 10 14:13 /nsm/elsa/data/elsa/tmp/buffers/1392041549.50333
-rw-r--r-- 1 root root 28487 Feb 10 14:12 /nsm/elsa/data/elsa/tmp/buffers/1392041489.42931
-rw-r--r-- 1 root root 24861 Feb 10 14:11 /nsm/elsa/data/elsa/tmp/buffers/1392041429.35552
-rw-r--r-- 1 root root 15935 Feb 10 14:10 /nsm/elsa/data/elsa/tmp/buffers/1392041369.31337
-rw-r--r-- 1 root root 42423 Feb 10 14:09 /nsm/elsa/data/elsa/tmp/buffers/1392041309.24693
-rw-r--r-- 1 root root 44042 Feb 10 14:08 /nsm/elsa/data/elsa/tmp/buffers/1392041249.1795
-rw-r--r-- 1 root root 34309 Feb 10 14:07 /nsm/elsa/data/elsa/tmp/buffers/1392041189.1224
-rw-r--r-- 1 root root 239706 Feb 10 14:06 /nsm/elsa/data/elsa/tmp/buffers/1392041129.03924
-rw-r--r-- 1 root root 80557 Feb 10 14:05 /nsm/elsa/data/elsa/tmp/buffers/1392041068.98031
-rw-r--r-- 1 root root 46709 Feb 10 14:04 /nsm/elsa/data/elsa/tmp/buffers/1392041008.93115
-rw-r--r-- 1 root root 223788 Feb 10 14:03 /nsm/elsa/data/elsa/tmp/buffers/1392040948.82346
-rw-r--r-- 1 root root 43601 Feb 10 14:02 /nsm/elsa/data/elsa/tmp/buffers/1392040888.74808
-rw-r--r-- 1 root root 78597 Feb 10 14:01 /nsm/elsa/data/elsa/tmp/buffers/1392040828.65634
-rw-r--r-- 1 root root 126873 Feb 10 14:00 /nsm/elsa/data/elsa/tmp/buffers/1392040768.59235
-rw-r--r-- 1 root root 41199 Feb 10 13:59 /nsm/elsa/data/elsa/tmp/buffers/1392040708.49314
-rw-r--r-- 1 root root 28784 Feb 10 13:58 /nsm/elsa/data/elsa/tmp/buffers/1392040648.4025
-rw-r--r-- 1 root root 20863 Feb 10 13:57 /nsm/elsa/data/elsa/tmp/buffers/1392040588.32045
-rw-r--r-- 1 root root 35926 Feb 10 13:56 /nsm/elsa/data/elsa/tmp/buffers/1392040528.22908
-rw-r--r-- 1 root root 34652 Feb 10 13:55 /nsm/elsa/data/elsa/tmp/buffers/1392040468.13713
-rw-r--r-- 1 root root 114373 Feb 10 13:54 /nsm/elsa/data/elsa/tmp/buffers/1392040408.10408
-rw-r--r-- 1 root root 34364 Feb 10 13:53 /nsm/elsa/data/elsa/tmp/buffers/1392040348.03054
-rw-r--r-- 1 root root 26015 Feb 10 13:52 /nsm/elsa/data/elsa/tmp/buffers/1392040287.95661
-rw-r--r-- 1 root root 19803 Feb 10 13:51 /nsm/elsa/data/elsa/tmp/buffers/1392040227.87563
-rw-r--r-- 1 root root 33838 Feb 10 13:50 /nsm/elsa/data/elsa/tmp/buffers/1392040167.7991
-rw-r--r-- 1 root root 51817 Feb 10 13:49 /nsm/elsa/data/elsa/tmp/buffers/1392040107.70819
-rw-r--r-- 1 root root 66020 Feb 10 13:48 /nsm/elsa/data/elsa/tmp/buffers/1392040047.63331
-rw-r--r-- 1 root root 32979 Feb 10 13:47 /nsm/elsa/data/elsa/tmp/buffers/1392039987.55217
-rw-r--r-- 1 root root 26714 Feb 10 13:46 /nsm/elsa/data/elsa/tmp/buffers/1392039927.49466
-rw-r--r-- 1 root root 33595 Feb 10 13:45 /nsm/elsa/data/elsa/tmp/buffers/1392039867.45748
-rw-r--r-- 1 root root 40729 Feb 10 13:44 /nsm/elsa/data/elsa/tmp/buffers/1392039807.39172
-rw-r--r-- 1 root root 21422 Feb 10 13:43 /nsm/elsa/data/elsa/tmp/buffers/1392039747.2509
-rw-r--r-- 1 root root 35704 Feb 10 13:42 /nsm/elsa/data/elsa/tmp/buffers/1392039687.1936
-rw-r--r-- 1 root root 28677 Feb 10 13:41 /nsm/elsa/data/elsa/tmp/buffers/1392039627.14463
-rw-r--r-- 1 root root 38064 Feb 10 13:40 /nsm/elsa/data/elsa/tmp/buffers/1392039567.06924
-rw-r--r-- 1 root root 38216 Feb 10 13:39 /nsm/elsa/data/elsa/tmp/buffers/1392039506.99475
-rw-r--r-- 1 root root 34659 Feb 10 13:38 /nsm/elsa/data/elsa/tmp/buffers/1392039446.95575
-rw-r--r-- 1 root root 28965 Feb 10 13:37 /nsm/elsa/data/elsa/tmp/buffers/1392039386.89001
-rw-r--r-- 1 root root 25127 Feb 10 13:36 /nsm/elsa/data/elsa/tmp/buffers/1392039326.82444
-rw-r--r-- 1 root root 29299 Feb 10 13:35 /nsm/elsa/data/elsa/tmp/buffers/1392039266.75748
-rw-r--r-- 1 root root 57170 Feb 10 13:34 /nsm/elsa/data/elsa/tmp/buffers/1392039206.68288
-rw-r--r-- 1 root root 29401 Feb 10 13:33 /nsm/elsa/data/elsa/tmp/buffers/1392039146.62562
-rw-r--r-- 1 root root 42959 Feb 10 13:32 /nsm/elsa/data/elsa/tmp/buffers/1392039086.57642
-rw-r--r-- 1 root root 46544 Feb 10 13:31 /nsm/elsa/data/elsa/tmp/buffers/1392039026.52633
-rw-r--r-- 1 root root 29668 Feb 10 13:30 /nsm/elsa/data/elsa/tmp/buffers/1392038966.4504
-rw-r--r-- 1 root root 68984 Feb 10 13:29 /nsm/elsa/data/elsa/tmp/buffers/1392038906.38389
-rw-r--r-- 1 root root 98598 Feb 10 13:28 /nsm/elsa/data/elsa/tmp/buffers/1392038846.3069
-rw-r--r-- 1 root root 43432 Feb 10 13:27 /nsm/elsa/data/elsa/tmp/buffers/1392038786.27422
-rw-r--r-- 1 root root 24080 Feb 10 13:26 /nsm/elsa/data/elsa/tmp/buffers/1392038726.22526
-rw-r--r-- 1 root root 38129 Feb 10 13:25 /nsm/elsa/data/elsa/tmp/buffers/1392038666.19165
-rw-r--r-- 1 root root 58224 Feb 10 13:24 /nsm/elsa/data/elsa/tmp/buffers/1392038606.07541
-rw-r--r-- 1 root root 37339 Feb 10 13:23 /nsm/elsa/data/elsa/tmp/buffers/1392038545.97549
-rw-r--r-- 1 root root 82829 Feb 10 13:22 /nsm/elsa/data/elsa/tmp/buffers/1392038485.93259
-rw-r--r-- 1 root root 20415 Feb 10 13:21 /nsm/elsa/data/elsa/tmp/buffers/1392038425.85002
-rw-r--r-- 1 root root 45444 Feb 10 13:20 /nsm/elsa/data/elsa/tmp/buffers/1392038365.80835
-rw-r--r-- 1 root root 65082 Feb 10 13:19 /nsm/elsa/data/elsa/tmp/buffers/1392038305.77427
-rw-r--r-- 1 root root 15644 Feb 10 13:18 /nsm/elsa/data/elsa/tmp/buffers/1392038245.71742
-rw-r--r-- 1 root root 28127 Feb 10 13:17 /nsm/elsa/data/elsa/tmp/buffers/1392038185.67141
-rw-r--r-- 1 root root 14370 Feb 10 13:16 /nsm/elsa/data/elsa/tmp/buffers/1392038125.60207
-rw-r--r-- 1 root root 13922 Feb 10 13:15 /nsm/elsa/data/elsa/tmp/buffers/1392038065.53361
-rw-r--r-- 1 root root 26606 Feb 10 13:14 /nsm/elsa/data/elsa/tmp/buffers/1392038005.49366
-rw-r--r-- 1 root root 11927 Feb 10 13:13 /nsm/elsa/data/elsa/tmp/buffers/1392037945.41996
-rw-r--r-- 1 root root 19662 Feb 10 13:12 /nsm/elsa/data/elsa/tmp/buffers/1392037885.38792
-rw-r--r-- 1 root root 13200 Feb 10 13:11 /nsm/elsa/data/elsa/tmp/buffers/1392037825.33071
-rw-r--r-- 1 root root 20672 Feb 10 13:10 /nsm/elsa/data/elsa/tmp/buffers/1392037765.27295
-rw-r--r-- 1 root root 40428 Feb 10 13:09 /nsm/elsa/data/elsa/tmp/buffers/1392037705.23975
-rw-r--r-- 1 root root 46911 Feb 10 13:08 /nsm/elsa/data/elsa/tmp/buffers/1392037645.14864
-rw-r--r-- 1 root root 32174 Feb 10 13:07 /nsm/elsa/data/elsa/tmp/buffers/1392037585.09917
-rw-r--r-- 1 root root 70071 Feb 10 13:06 /nsm/elsa/data/elsa/tmp/buffers/1392037525.07435
-rw-r--r-- 1 root root 120854 Feb 10 13:05 /nsm/elsa/data/elsa/tmp/buffers/1392037464.99776
-rw-r--r-- 1 root root 39302 Feb 10 13:04 /nsm/elsa/data/elsa/tmp/buffers/1392037404.96545
-rw-r--r-- 1 root root 15946 Feb 10 13:03 /nsm/elsa/data/elsa/tmp/buffers/1392037344.93377
-rw-r--r-- 1 root root 21530 Feb 10 13:02 /nsm/elsa/data/elsa/tmp/buffers/1392037284.81953
-rw-r--r-- 1 root root 33931 Feb 10 13:01 /nsm/elsa/data/elsa/tmp/buffers/1392037224.74498
-rw-r--r-- 1 root root 90556 Feb 10 13:00 /nsm/elsa/data/elsa/tmp/buffers/1392037164.68192
-rw-r--r-- 1 root root 54328 Feb 10 12:59 /nsm/elsa/data/elsa/tmp/buffers/1392037104.63246
-rw-r--r-- 1 root root 21068 Feb 10 12:58 /nsm/elsa/data/elsa/tmp/buffers/1392037044.60042
-rw-r--r-- 1 root root 26224 Feb 10 12:57 /nsm/elsa/data/elsa/tmp/buffers/1392036984.52673
-rw-r--r-- 1 root root 15219 Feb 10 12:56 /nsm/elsa/data/elsa/tmp/buffers/1392036924.453
-rw-r--r-- 1 root root 79294 Feb 10 12:55 /nsm/elsa/data/elsa/tmp/buffers/1392036864.3772
-rw-r--r-- 1 root root 121274 Feb 10 12:54 /nsm/elsa/data/elsa/tmp/buffers/1392036804.34414
-rw-r--r-- 1 root root 32036 Feb 10 12:53 /nsm/elsa/data/elsa/tmp/buffers/1392036744.26213
-rw-r--r-- 1 root root 21952 Feb 10 12:52 /nsm/elsa/data/elsa/tmp/buffers/1392036684.21282
-rw-r--r-- 1 root root 62159 Feb 10 12:51 /nsm/elsa/data/elsa/tmp/buffers/1392036624.15429
-rw-r--r-- 1 root root 203359 Feb 10 12:50 /nsm/elsa/data/elsa/tmp/buffers/1392036564.07107
-rw-r--r-- 1 root root 28176 Feb 10 12:49 /nsm/elsa/data/elsa/tmp/buffers/1392036504.01411
-rw-r--r-- 1 root root 20841 Feb 10 12:48 /nsm/elsa/data/elsa/tmp/buffers/1392036443.9163
-rw-r--r-- 1 root root 14667 Feb 10 12:47 /nsm/elsa/data/elsa/tmp/buffers/1392036383.82699
-rw-r--r-- 1 root root 29014 Feb 10 12:46 /nsm/elsa/data/elsa/tmp/buffers/1392036323.74378
-rw-r--r-- 1 root root 18086 Feb 10 12:45 /nsm/elsa/data/elsa/tmp/buffers/1392036263.69234
-rw-r--r-- 1 root root 35974 Feb 10 12:44 /nsm/elsa/data/elsa/tmp/buffers/1392036203.61023
-rw-r--r-- 1 root root 46307 Feb 10 12:43 /nsm/elsa/data/elsa/tmp/buffers/1392036143.53595
-rw-r--r-- 1 root root 36625 Feb 10 12:42 /nsm/elsa/data/elsa/tmp/buffers/1392036083.42829
-rw-r--r-- 1 root root 17985 Feb 10 12:41 /nsm/elsa/data/elsa/tmp/buffers/1392036023.37169
-rw-r--r-- 1 root root 14676 Feb 10 12:40 /nsm/elsa/data/elsa/tmp/buffers/1392035963.29719
-rw-r--r-- 1 root root 39273 Feb 10 12:39 /nsm/elsa/data/elsa/tmp/buffers/1392035903.22381
-rw-r--r-- 1 root root 25341 Feb 10 12:38 /nsm/elsa/data/elsa/tmp/buffers/1392035843.15044
-rw-r--r-- 1 root root 38407 Feb 10 12:37 /nsm/elsa/data/elsa/tmp/buffers/1392035783.08505
-rw-r--r-- 1 root root 32648 Feb 10 12:36 /nsm/elsa/data/elsa/tmp/buffers/1392035723.01108
-rw-r--r-- 1 root root 48465 Feb 10 12:35 /nsm/elsa/data/elsa/tmp/buffers/1392035662.92791
-rw-r--r-- 1 root root 37720 Feb 10 12:34 /nsm/elsa/data/elsa/tmp/buffers/1392035602.85409
-rw-r--r-- 1 root root 11925 Feb 10 12:33 /nsm/elsa/data/elsa/tmp/buffers/1392035542.73232
-rw-r--r-- 1 root root 22987 Feb 10 12:32 /nsm/elsa/data/elsa/tmp/buffers/1392035482.64134
-rw-r--r-- 1 root root 20490 Feb 10 12:31 /nsm/elsa/data/elsa/tmp/buffers/1392035422.55069
-rw-r--r-- 1 root root 21876 Feb 10 12:30 /nsm/elsa/data/elsa/tmp/buffers/1392035362.48263
-rw-r--r-- 1 root root 69206 Feb 10 12:29 /nsm/elsa/data/elsa/tmp/buffers/1392035302.40849
-rw-r--r-- 1 root root 26803 Feb 10 12:28 /nsm/elsa/data/elsa/tmp/buffers/1392035242.31853
-rw-r--r-- 1 root root 16009 Feb 10 12:27 /nsm/elsa/data/elsa/tmp/buffers/1392035182.24179
-rw-r--r-- 1 root root 21491 Feb 10 12:26 /nsm/elsa/data/elsa/tmp/buffers/1392035122.15107
-rw-r--r-- 1 root root 23300 Feb 10 12:25 /nsm/elsa/data/elsa/tmp/buffers/1392035062.05842
-rw-r--r-- 1 root root 28772 Feb 10 12:24 /nsm/elsa/data/elsa/tmp/buffers/1392035001.95895
-rw-r--r-- 1 root root 23995 Feb 10 12:23 /nsm/elsa/data/elsa/tmp/buffers/1392034941.86842
-rw-r--r-- 1 root root 18368 Feb 10 12:22 /nsm/elsa/data/elsa/tmp/buffers/1392034881.77767
-rw-r--r-- 1 root root 25819 Feb 10 12:21 /nsm/elsa/data/elsa/tmp/buffers/1392034821.71993
-rw-r--r-- 1 root root 25673 Feb 10 12:20 /nsm/elsa/data/elsa/tmp/buffers/1392034761.66093
-rw-r--r-- 1 root root 25827 Feb 10 12:19 /nsm/elsa/data/elsa/tmp/buffers/1392034701.57055
-rw-r--r-- 1 root root 13814 Feb 10 12:18 /nsm/elsa/data/elsa/tmp/buffers/1392034641.48835
-rw-r--r-- 1 root root 14616 Feb 10 12:17 /nsm/elsa/data/elsa/tmp/buffers/1392034581.41403
-rw-r--r-- 1 root root 14330 Feb 10 12:16 /nsm/elsa/data/elsa/tmp/buffers/1392034521.33958
-rw-r--r-- 1 root root 17263 Feb 10 12:15 /nsm/elsa/data/elsa/tmp/buffers/1392034461.26283
-rw-r--r-- 1 root root 32965 Feb 10 12:14 /nsm/elsa/data/elsa/tmp/buffers/1392034401.18014
-rw-r--r-- 1 root root 24280 Feb 10 12:13 /nsm/elsa/data/elsa/tmp/buffers/1392034341.13084
-rw-r--r-- 1 root root 15728 Feb 10 12:12 /nsm/elsa/data/elsa/tmp/buffers/1392034281.04095
-rw-r--r-- 1 root root 12406 Feb 10 12:11 /nsm/elsa/data/elsa/tmp/buffers/1392034220.95036
-rw-r--r-- 1 root root 16460 Feb 10 12:10 /nsm/elsa/data/elsa/tmp/buffers/1392034160.90032
-rw-r--r-- 1 root root 39043 Feb 10 12:09 /nsm/elsa/data/elsa/tmp/buffers/1392034100.85071
-rw-r--r-- 1 root root 25053 Feb 10 12:08 /nsm/elsa/data/elsa/tmp/buffers/1392034040.79336
-rw-r--r-- 1 root root 15001 Feb 10 12:07 /nsm/elsa/data/elsa/tmp/buffers/1392033980.73614
-rw-r--r-- 1 root root 10180 Feb 10 12:06 /nsm/elsa/data/elsa/tmp/buffers/1392033920.68835
-rw-r--r-- 1 root root 15806 Feb 10 12:05 /nsm/elsa/data/elsa/tmp/buffers/1392033860.61248
-rw-r--r-- 1 root root 27267 Feb 10 12:04 /nsm/elsa/data/elsa/tmp/buffers/1392033800.55497
-rw-r--r-- 1 root root 35556 Feb 10 12:03 /nsm/elsa/data/elsa/tmp/buffers/1392033740.48831
-rw-r--r-- 1 root root 256208 Feb 10 12:02 /nsm/elsa/data/elsa/tmp/buffers/1392033680.43264
-rw-r--r-- 1 root root 18859 Feb 10 12:01 /nsm/elsa/data/elsa/tmp/buffers/1392033620.36583
-rw-r--r-- 1 root root 26847 Feb 10 12:00 /nsm/elsa/data/elsa/tmp/buffers/1392033560.30505
-rw-r--r-- 1 root root 26850 Feb 10 11:59 /nsm/elsa/data/elsa/tmp/buffers/1392033500.24007
-rw-r--r-- 1 root root 16580 Feb 10 11:58 /nsm/elsa/data/elsa/tmp/buffers/1392033440.18325
-rw-r--r-- 1 root root 16863 Feb 10 11:57 /nsm/elsa/data/elsa/tmp/buffers/1392033380.12643
-rw-r--r-- 1 root root 16450 Feb 10 11:56 /nsm/elsa/data/elsa/tmp/buffers/1392033320.07764
-rw-r--r-- 1 root root 14089 Feb 10 11:55 /nsm/elsa/data/elsa/tmp/buffers/1392033260.00594
-rw-r--r-- 1 root root 26278 Feb 10 11:54 /nsm/elsa/data/elsa/tmp/buffers/1392033199.9464
-rw-r--r-- 1 root root 32842 Feb 10 11:53 /nsm/elsa/data/elsa/tmp/buffers/1392033139.74887
-rw-r--r-- 1 root root 17481 Feb 10 11:52 /nsm/elsa/data/elsa/tmp/buffers/1392033079.68325
-rw-r--r-- 1 root root 15832 Feb 10 11:51 /nsm/elsa/data/elsa/tmp/buffers/1392033019.62666
-rw-r--r-- 1 root root 17126 Feb 10 11:50 /nsm/elsa/data/elsa/tmp/buffers/1392032959.5687
-rw-r--r-- 1 root root 24919 Feb 10 11:49 /nsm/elsa/data/elsa/tmp/buffers/1392032899.52822
-rw-r--r-- 1 root root 14461 Feb 10 11:48 /nsm/elsa/data/elsa/tmp/buffers/1392032839.46331
-rw-r--r-- 1 root root 17726 Feb 10 11:47 /nsm/elsa/data/elsa/tmp/buffers/1392032779.39777
-rw-r--r-- 1 root root 14972 Feb 10 11:46 /nsm/elsa/data/elsa/tmp/buffers/1392032719.34643
-rw-r--r-- 1 root root 16630 Feb 10 11:45 /nsm/elsa/data/elsa/tmp/buffers/1392032659.32124
-rw-r--r-- 1 root root 25020 Feb 10 11:44 /nsm/elsa/data/elsa/tmp/buffers/1392032599.28955
-rw-r--r-- 1 root root 12014 Feb 10 11:43 /nsm/elsa/data/elsa/tmp/buffers/1392032539.22433
-rw-r--r-- 1 root root 18570 Feb 10 11:42 /nsm/elsa/data/elsa/tmp/buffers/1392032479.19233
-rw-r--r-- 1 root root 16535 Feb 10 11:41 /nsm/elsa/data/elsa/tmp/buffers/1392032419.13494
-rw-r--r-- 1 root root 16145 Feb 10 11:40 /nsm/elsa/data/elsa/tmp/buffers/1392032359.10161
-rw-r--r-- 1 root root 27018 Feb 10 11:39 /nsm/elsa/data/elsa/tmp/buffers/1392032299.07007
-rw-r--r-- 1 root root 29552 Feb 10 11:38 /nsm/elsa/data/elsa/tmp/buffers/1392032238.96357
-rw-r--r-- 1 root root 12518 Feb 10 11:37 /nsm/elsa/data/elsa/tmp/buffers/1392032178.91546
-rw-r--r-- 1 root root 88349 Feb 10 11:36 /nsm/elsa/data/elsa/tmp/buffers/1392032118.84917
-rw-r--r-- 1 root root 29581 Feb 10 11:35 /nsm/elsa/data/elsa/tmp/buffers/1392032058.81573
-rw-r--r-- 1 root root 26568 Feb 10 11:34 /nsm/elsa/data/elsa/tmp/buffers/1392031998.71694
-rw-r--r-- 1 root root 12034 Feb 10 11:33 /nsm/elsa/data/elsa/tmp/buffers/1392031938.66855
-rw-r--r-- 1 root root 12861 Feb 10 11:32 /nsm/elsa/data/elsa/tmp/buffers/1392031878.63737
-rw-r--r-- 1 root root 13151 Feb 10 11:31 /nsm/elsa/data/elsa/tmp/buffers/1392031818.57123
-rw-r--r-- 1 root root 17975 Feb 10 11:30 /nsm/elsa/data/elsa/tmp/buffers/1392031758.50116
-rw-r--r-- 1 root root 31495 Feb 10 11:29 /nsm/elsa/data/elsa/tmp/buffers/1392031698.39391
-rw-r--r-- 1 root root 30915 Feb 10 11:28 /nsm/elsa/data/elsa/tmp/buffers/1392031638.31084
-rw-r--r-- 1 root root 83750 Feb 10 11:27 /nsm/elsa/data/elsa/tmp/buffers/1392031578.24466
-rw-r--r-- 1 root root 16506 Feb 10 11:26 /nsm/elsa/data/elsa/tmp/buffers/1392031518.1627
-rw-r--r-- 1 root root 16288 Feb 10 11:25 /nsm/elsa/data/elsa/tmp/buffers/1392031458.07902
-rw-r--r-- 1 root root 31685 Feb 10 11:24 /nsm/elsa/data/elsa/tmp/buffers/1392031398.01384
-rw-r--r-- 1 root root 20982 Feb 10 11:23 /nsm/elsa/data/elsa/tmp/buffers/1392031337.92336
-rw-r--r-- 1 root root 13873 Feb 10 11:22 /nsm/elsa/data/elsa/tmp/buffers/1392031277.84124
-rw-r--r-- 1 root root 16033 Feb 10 11:21 /nsm/elsa/data/elsa/tmp/buffers/1392031217.7505
-rw-r--r-- 1 root root 25033 Feb 10 11:20 /nsm/elsa/data/elsa/tmp/buffers/1392031157.65018
-rw-r--r-- 1 root root 33907 Feb 10 11:19 /nsm/elsa/data/elsa/tmp/buffers/1392031097.59269
-rw-r--r-- 1 root root 27356 Feb 10 11:18 /nsm/elsa/data/elsa/tmp/buffers/1392031037.47699
-rw-r--r-- 1 root root 13944 Feb 10 11:17 /nsm/elsa/data/elsa/tmp/buffers/1392030977.42949
-rw-r--r-- 1 root root 14891 Feb 10 11:16 /nsm/elsa/data/elsa/tmp/buffers/1392030917.38013
-rw-r--r-- 1 root root 18925 Feb 10 11:15 /nsm/elsa/data/elsa/tmp/buffers/1392030857.29482
-rw-r--r-- 1 root root 25830 Feb 10 11:14 /nsm/elsa/data/elsa/tmp/buffers/1392030797.23041
-rw-r--r-- 1 root root 24264 Feb 10 11:13 /nsm/elsa/data/elsa/tmp/buffers/1392030737.17332
-rw-r--r-- 1 root root 16160 Feb 10 11:12 /nsm/elsa/data/elsa/tmp/buffers/1392030677.09991
-rw-r--r-- 1 root root 47464 Feb 10 11:11 /nsm/elsa/data/elsa/tmp/buffers/1392030617.00086
-rw-r--r-- 1 root root 20093 Feb 10 11:10 /nsm/elsa/data/elsa/tmp/buffers/1392030556.90827
-rw-r--r-- 1 root root 29798 Feb 10 11:09 /nsm/elsa/data/elsa/tmp/buffers/1392030496.86724
-rw-r--r-- 1 root root 18568 Feb 10 11:08 /nsm/elsa/data/elsa/tmp/buffers/1392030436.76819
-rw-r--r-- 1 root root 34485 Feb 10 11:07 /nsm/elsa/data/elsa/tmp/buffers/1392030376.67601
-rw-r--r-- 1 root root 22970 Feb 10 11:06 /nsm/elsa/data/elsa/tmp/buffers/1392030316.59378
-rw-r--r-- 1 root root 17757 Feb 10 11:05 /nsm/elsa/data/elsa/tmp/buffers/1392030256.52746
-rw-r--r-- 1 root root 24674 Feb 10 11:04 /nsm/elsa/data/elsa/tmp/buffers/1392030196.437
-rw-r--r-- 1 root root 11873 Feb 10 11:03 /nsm/elsa/data/elsa/tmp/buffers/1392030136.34715
-rw-r--r-- 1 root root 13653 Feb 10 11:02 /nsm/elsa/data/elsa/tmp/buffers/1392030076.27339
-rw-r--r-- 1 root root 21855 Feb 10 11:01 /nsm/elsa/data/elsa/tmp/buffers/1392030016.18992
-rw-r--r-- 1 root root 29026 Feb 10 11:00 /nsm/elsa/data/elsa/tmp/buffers/1392029956.14588
-rw-r--r-- 1 root root 39598 Feb 10 10:59 /nsm/elsa/data/elsa/tmp/buffers/1392029896.0727
-rw-r--r-- 1 root root 14716 Feb 10 10:58 /nsm/elsa/data/elsa/tmp/buffers/1392029835.97426
-rw-r--r-- 1 root root 24082 Feb 10 10:57 /nsm/elsa/data/elsa/tmp/buffers/1392029775.89229
-rw-r--r-- 1 root root 33057 Feb 10 10:56 /nsm/elsa/data/elsa/tmp/buffers/1392029715.80997
-rw-r--r-- 1 root root 14172 Feb 10 10:55 /nsm/elsa/data/elsa/tmp/buffers/1392029655.74499
-rw-r--r-- 1 root root 49133 Feb 10 10:54 /nsm/elsa/data/elsa/tmp/buffers/1392029595.68782
-rw-r--r-- 1 root root 17261 Feb 10 10:53 /nsm/elsa/data/elsa/tmp/buffers/1392029535.6062
-rw-r--r-- 1 root root 15992 Feb 10 10:52 /nsm/elsa/data/elsa/tmp/buffers/1392029475.5326
-rw-r--r-- 1 root root 12657 Feb 10 10:51 /nsm/elsa/data/elsa/tmp/buffers/1392029415.47669
-rw-r--r-- 1 root root 17345 Feb 10 10:50 /nsm/elsa/data/elsa/tmp/buffers/1392029355.41879
-rw-r--r-- 1 root root 33128 Feb 10 10:49 /nsm/elsa/data/elsa/tmp/buffers/1392029295.33653
-rw-r--r-- 1 root root 47994 Feb 10 10:48 /nsm/elsa/data/elsa/tmp/buffers/1392029235.28822
-rw-r--r-- 1 root root 10506 Feb 10 10:47 /nsm/elsa/data/elsa/tmp/buffers/1392029175.20836
-rw-r--r-- 1 root root 13492 Feb 10 10:46 /nsm/elsa/data/elsa/tmp/buffers/1392029115.10074
-rw-r--r-- 1 root root 17396 Feb 10 10:45 /nsm/elsa/data/elsa/tmp/buffers/1392029054.99952
-rw-r--r-- 1 root root 112874 Feb 10 10:44 /nsm/elsa/data/elsa/tmp/buffers/1392028994.9585
-rw-r--r-- 1 root root 24739 Feb 10 10:43 /nsm/elsa/data/elsa/tmp/buffers/1392028934.90202
-rw-r--r-- 1 root root 21560 Feb 10 10:42 /nsm/elsa/data/elsa/tmp/buffers/1392028874.82823
-rw-r--r-- 1 root root 40328 Feb 10 10:41 /nsm/elsa/data/elsa/tmp/buffers/1392028814.77967
-rw-r--r-- 1 root root 15761 Feb 10 10:40 /nsm/elsa/data/elsa/tmp/buffers/1392028754.722
-rw-r--r-- 1 root root 24722 Feb 10 10:39 /nsm/elsa/data/elsa/tmp/buffers/1392028694.65724
-rw-r--r-- 1 root root 11124 Feb 10 10:38 /nsm/elsa/data/elsa/tmp/buffers/1392028634.5774
-rw-r--r-- 1 root root 22879 Feb 10 10:37 /nsm/elsa/data/elsa/tmp/buffers/1392028574.51246
-rw-r--r-- 1 root root 19915 Feb 10 10:36 /nsm/elsa/data/elsa/tmp/buffers/1392028514.43914
-rw-r--r-- 1 root root 18730 Feb 10 10:35 /nsm/elsa/data/elsa/tmp/buffers/1392028454.38953
-rw-r--r-- 1 root root 26229 Feb 10 10:34 /nsm/elsa/data/elsa/tmp/buffers/1392028394.32453
-rw-r--r-- 1 root root 10574 Feb 10 10:33 /nsm/elsa/data/elsa/tmp/buffers/1392028334.28555
-rw-r--r-- 1 root root 13633 Feb 10 10:32 /nsm/elsa/data/elsa/tmp/buffers/1392028274.21367
-rw-r--r-- 1 root root 14975 Feb 10 10:31 /nsm/elsa/data/elsa/tmp/buffers/1392028214.15577
-rw-r--r-- 1 root root 12836 Feb 10 10:30 /nsm/elsa/data/elsa/tmp/buffers/1392028154.11248
-rw-r--r-- 1 root root 32684 Feb 10 10:29 /nsm/elsa/data/elsa/tmp/buffers/1392028094.05782
-rw-r--r-- 1 root root 12908 Feb 10 10:28 /nsm/elsa/data/elsa/tmp/buffers/1392028033.99306
-rw-r--r-- 1 root root 11095 Feb 10 10:27 /nsm/elsa/data/elsa/tmp/buffers/1392027973.94572
-rw-r--r-- 1 root root 17607 Feb 10 10:26 /nsm/elsa/data/elsa/tmp/buffers/1392027913.89756
-rw-r--r-- 1 root root 13785 Feb 10 10:25 /nsm/elsa/data/elsa/tmp/buffers/1392027853.80714
-rw-r--r-- 1 root root 27418 Feb 10 10:24 /nsm/elsa/data/elsa/tmp/buffers/1392027793.74339
-rw-r--r-- 1 root root 17577 Feb 10 10:23 /nsm/elsa/data/elsa/tmp/buffers/1392027733.68712
-rw-r--r-- 1 root root 19202 Feb 10 10:22 /nsm/elsa/data/elsa/tmp/buffers/1392027673.614
-rw-r--r-- 1 root root 15078 Feb 10 10:21 /nsm/elsa/data/elsa/tmp/buffers/1392027613.54976
-rw-r--r-- 1 root root 25115 Feb 10 10:20 /nsm/elsa/data/elsa/tmp/buffers/1392027553.43323
-rw-r--r-- 1 root root 28871 Feb 10 10:19 /nsm/elsa/data/elsa/tmp/buffers/1392027493.38654
-rw-r--r-- 1 root root 17939 Feb 10 10:18 /nsm/elsa/data/elsa/tmp/buffers/1392027433.33427
-rw-r--r-- 1 root root 15972 Feb 10 10:17 /nsm/elsa/data/elsa/tmp/buffers/1392027373.30247
-rw-r--r-- 1 root root 14773 Feb 10 10:16 /nsm/elsa/data/elsa/tmp/buffers/1392027313.23546
-rw-r--r-- 1 root root 13666 Feb 10 10:15 /nsm/elsa/data/elsa/tmp/buffers/1392027253.15183
-rw-r--r-- 1 root root 25135 Feb 10 10:14 /nsm/elsa/data/elsa/tmp/buffers/1392027193.07851
-rw-r--r-- 1 root root 10810 Feb 10 10:13 /nsm/elsa/data/elsa/tmp/buffers/1392027133.04663
-rw-r--r-- 1 root root 17596 Feb 10 10:12 /nsm/elsa/data/elsa/tmp/buffers/1392027073.01502
-rw-r--r-- 1 root root 24098 Feb 10 10:11 /nsm/elsa/data/elsa/tmp/buffers/1392027012.95697
-rw-r--r-- 1 root root 36637 Feb 10 10:10 /nsm/elsa/data/elsa/tmp/buffers/1392026952.86463
-rw-r--r-- 1 root root 42524 Feb 10 10:09 /nsm/elsa/data/elsa/tmp/buffers/1392026892.7823
-rw-r--r-- 1 root root 14668 Feb 10 10:08 /nsm/elsa/data/elsa/tmp/buffers/1392026832.72482
-rw-r--r-- 1 root root 14118 Feb 10 10:07 /nsm/elsa/data/elsa/tmp/buffers/1392026772.61928
-rw-r--r-- 1 root root 9746 Feb 10 10:06 /nsm/elsa/data/elsa/tmp/buffers/1392026712.56397
-rw-r--r-- 1 root root 17578 Feb 10 10:05 /nsm/elsa/data/elsa/tmp/buffers/1392026652.47944
-rw-r--r-- 1 root root 24235 Feb 10 10:04 /nsm/elsa/data/elsa/tmp/buffers/1392026592.39503
-rw-r--r-- 1 root root 17239 Feb 10 10:03 /nsm/elsa/data/elsa/tmp/buffers/1392026532.29192
-rw-r--r-- 1 root root 12729 Feb 10 10:02 /nsm/elsa/data/elsa/tmp/buffers/1392026472.21076
-rw-r--r-- 1 root root 18270 Feb 10 10:01 /nsm/elsa/data/elsa/tmp/buffers/1392026412.16864
-rw-r--r-- 1 root root 22633 Feb 10 10:00 /nsm/elsa/data/elsa/tmp/buffers/1392026352.09141
-rw-r--r-- 1 root root 24640 Feb 10 09:59 /nsm/elsa/data/elsa/tmp/buffers/1392026291.98776
-rw-r--r-- 1 root root 17772 Feb 10 09:58 /nsm/elsa/data/elsa/tmp/buffers/1392026231.92343
-rw-r--r-- 1 root root 70 Feb 10 09:57 /nsm/elsa/data/elsa/tmp/buffers/host_stats.tsv
-rw-r--r-- 1 root root 13051 Feb 10 09:57 /nsm/elsa/data/elsa/tmp/buffers/1392026171.88511
-rw-r--r-- 1 root root 0 Feb 10 03:06 /nsm/elsa/data/elsa/tmp/buffers/1392001608.52631
-rw-r--r-- 1 root root 9064 Feb 10 03:06 /nsm/elsa/data/elsa/tmp/buffers/1392001556.02193

ELSA Directory Sizes:
1.8G /nsm/elsa/data
7.8M /var/lib/mysql/syslog
4.3G /var/lib/mysql/syslog_data

autossh
Checking for process:
2523 /usr/lib/autossh/autossh -M 0 -q -N -o ServerAliveInterval 60 -o ServerAliveCountMax 3 -i /root/.ssh/securityonion -L 3306:127.0.0.1:3306 -R 50000:localhost:3154 xxxx...@x.x.x.x

Checking APIKEY:
APIKEY matches server.

starman
Checking for processes:
2084 starman worker -I/opt/elsa/web/lib --user=www-data --listen :3154 --daemonize --pid /var/run/starman.pid --error-log /var/log/starman.log /opt/elsa/web/lib/Web.psgi

Doug Burks

unread,
Feb 10, 2014, 10:14:07 AM2/10/14
to securit...@googlegroups.com
"ELSA Buffers in Queue" seems high.

Can you provide the full sostat-redacted output so that we can see
what the rest of the system looks like?

Pietro Delsante

unread,
Feb 10, 2014, 10:16:55 AM2/10/14
to securit...@googlegroups.com
On Monday, February 10, 2014 3:36:13 PM UTC+1, Doug Burks wrote:
> I think there is a known issue with the starman "restart" function.
> As you can see when we released the ELSA 1.5 packages, we recommended
> that folks do a "stop" and then do a "start" instead of doing a
> "restart":
>
> http://blog.securityonion.net/2013/12/bro-22-and-elsa-15-packages-now.html
>
> stop/start (NOT restart) starman on each sensor:
>
> sudo service starman stop
> sudo service starman start

Thank you, Doug, I will keep this in mind next time.

However, I'm still left without a clue about why Starman is periodically failing... :-(

BBCan177

unread,
Feb 10, 2014, 10:33:56 AM2/10/14
to securit...@googlegroups.com
On Monday, February 10, 2014 10:14:07 AM UTC-5, Doug Burks wrote:
> "ELSA Buffers in Queue" seems high.

> Can you provide the full sostat-redacted output so that we can see
> what the rest of the system looks like?

When I ran sostat starman was down.
See second attachment for sostat after stop/start of Starman.

I know this box is short on memory. I plan on upgrading soon but its 14hrs away
from me. When i was using Snort, it did not use any of the swap memory.


sudo service starman stop
Stopping Starman: /etc/init.d/starman: line 43: kill: (2074) - No such process

sudo service starman start

Sensor sostat.txt
Sensor sostat after Starman Restart.txt

Doug Burks

unread,
Feb 10, 2014, 10:46:32 AM2/10/14
to securit...@googlegroups.com
On Mon, Feb 10, 2014 at 10:33 AM, BBCan177 <bbca...@gmail.com> wrote:
> I know this box is short on memory. I plan on upgrading soon but its 14hrs away
> from me. When i was using Snort, it did not use any of the swap memory.

2GB RAM is REALLY low for a production sensor running that many
services. There's no way we can guarantee reliable performance with
only 2GB RAM.

--
Doug Burks

Pietro Delsante

unread,
Feb 10, 2014, 12:03:01 PM2/10/14
to securit...@googlegroups.com
On Monday, February 10, 2014 4:46:32 PM UTC+1, Doug Burks wrote:

> On Mon, Feb 10, 2014 at 10:33 AM, BBCan177 wrote:
>
> > I know this box is short on memory. I plan on upgrading soon but its 14hrs away
>
> > from me. When i was using Snort, it did not use any of the swap memory.
>
>
>
> 2GB RAM is REALLY low for a production sensor running that many
>
> services. There's no way we can guarantee reliable performance with
>
> only 2GB RAM.
>
>
>
> --
>
> Doug Burks

My sensor is running with 64Gb of RAM and starman is showing the very same behaviour, so I think this issue might not be related to how much RAM you have... :-(

Doug Burks

unread,
Feb 10, 2014, 1:19:09 PM2/10/14
to securit...@googlegroups.com
On Mon, Feb 10, 2014 at 12:03 PM, Pietro Delsante
<pietro....@gmail.com> wrote:
> My sensor is running with 64Gb of RAM and starman is showing the very same behaviour, so I think this issue might not be related to how much RAM you have... :-(

Please send the output of the following:
grep -v "Use of uninitialized value" /var/log/starman.log |grep -v
"attempted on invalid"


--
Doug Burks

BBCan177

unread,
Feb 10, 2014, 1:28:36 PM2/10/14
to securit...@googlegroups.com
starman.log

Doug Burks

unread,
Feb 10, 2014, 1:35:06 PM2/10/14
to securit...@googlegroups.com
Hi BBCan177,

As I mentioned in my previous response to you, 2GB RAM is REALLY low
for a production sensor running that many services and there's no way
we can guarantee reliable performance with only 2GB RAM. I was asking
for starman.log from Pietro since he has sufficient RAM. If you
continue to have this issue after upgrading your RAM, then please
start a new thread and we'll troubleshoot there.

Pietro Delsante

unread,
Feb 11, 2014, 5:52:59 AM2/11/14
to securit...@googlegroups.com
On Monday, February 10, 2014 7:19:09 PM UTC+1, Doug Burks wrote:

> On Mon, Feb 10, 2014 at 12:03 PM, Pietro Delsante wrote:
>
> > My sensor is running with 64Gb of RAM and starman is showing the very same behaviour, so I think this issue might not be related to how much RAM you have... :-(
>
>
>
> Please send the output of the following:
>
> grep -v "Use of uninitialized value" /var/log/starman.log |grep -v
>
> "attempted on invalid"
>
>
>
>
>
> --
>
> Doug Burks

Here you go, as I told you there's not much useful there in there, and it does not even have timestamps (except for the startup)...:

*************************
Process Backgrounded
2014/02/11-10:45:36 Starman::Server (type Net::Server::PreFork) starting! pid(2141)
Using default listen value of 128
Binding to TCP port 3154 on host *
Setting gid to "0 0"
Setting uid to "33"
*************************

Regards,
Pietro

Pietro Delsante

unread,
Feb 11, 2014, 5:59:52 AM2/11/14
to securit...@googlegroups.com

Woah, sorry, my bad... I had rotated the log.

Looking at old logs, they're full of this:

*************************
Cannot write to '/nsm/elsa/data/elsa/log/web.log': No space left on device at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
EV: error in callback (ignoring): Cannot write to '/nsm/elsa/data/elsa/log/web.log': No space left on device at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': No space left on device at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': No space left on device at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Cannot write to '/nsm/elsa/data/elsa/log/web.log': at /usr/share/perl5/Log/Log4perl/Appender/File.pm line 245.
Process Backgrounded
2014/02/07-16:21:57 Starman::Server (type Net::Server::PreFork) starting! pid(26718)


Using default listen value of 128
Binding to TCP port 3154 on host *

2014/02/07-16:21:57 Can't connect to TCP port 3154 on * [Address already in use]
at line 92 in file /usr/share/perl5/Net/Server/Proto/TCP.pm
2014/02/07-16:21:57 Server closing!
Process Backgrounded
2014/02/07-16:23:20 Starman::Server (type Net::Server::PreFork) starting! pid(26897)


Using default listen value of 128
Binding to TCP port 3154 on host *

Setting gid to "0 0 0"


Setting uid to "33"
Process Backgrounded
2014/02/11-10:45:36 Starman::Server (type Net::Server::PreFork) starting! pid(2141)
Using default listen value of 128
Binding to TCP port 3154 on host *
Setting gid to "0 0"
Setting uid to "33"
*************************

So, I imagine that the problem is caused by a full partition... We should receive a new disk for that host in a few days, I'll let you know if the problem persists after the upgrade.

Thanks a lot for your help!

Pietro

Reply all
Reply to author
Forward
0 new messages