[Wikitech-l] rsync on scap/sync reporting 'no space left on device' for a lot of hosts

[Arthur Richards]

I just ran scap and saw the following for a lot of hosts:

srv285: rsync: write failed on
"/usr/local/apache/common-local/php-1.19/cache/l10n/l10n_cache-ab.cdb": No
space left on device (28)

srv285: rsync error: error in file IO (code 11) at receiver.c(302)
[receiver=3.0.7]

srv285: rsync: connection unexpectedly closed (2051 bytes received so far)
[generator]

srv285: rsync error: error in rsync protocol data stream (code 12) at
io.c(601) [generator=3.0.7]

Also, on configchange:

mw21: rsync: write failed on
"/apache/common-local/wmf-config/CommonSettings.php": No space left on
device (28)

mw21: rsync error: error in file IO (code 11) at receiver.c(302)
[receiver=3.0.7]

mw21: rsync: connection unexpectedly closed (37 bytes received so far)
[generator]

mw21: rsync error: error in rsync protocol data stream (code 12) at
io.c(601) [generator=3.0.7]

Not sure if this is a problem and/or if others are aware/working on it, but
thought I'd mention it.

--
Arthur Richards
Software Engineer, Mobile
[[User:Awjrichards]]
IRC: awjr
+1-415-839-6885 x6687
_______________________________________________
Wikitech-l mailing list
Wikit...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Leslie Carr]

It appears that there are a few issues

#1 the cleanup cron is only looking for php* and there are many older
files named timeline*

On some machines /tmp is a separate partition, but / is still filled
up. This appears to be due to a huge amount of php errors filling the
logs

example below :
Apr 2 20:32:46 mw21 apache2[17566]: PHP Warning: readdir() expects
parameter 1 to be resource, boolean given in
/usr/local/apache/common-local/php-1.19/extensions/ConfirmEdit/FancyCaptcha.class.php
on line 100

Can someone check out the php errors?

--
Leslie Carr
Wikimedia Foundation
AS 14907, 43821

[Antoine Musso]

Le 03/04/12 01:30, Leslie Carr a écrit :

> On some machines /tmp is a separate partition, but / is still filled
> up. This appears to be due to a huge amount of php errors filling the
> logs

Either we should stop Apache logging entirely (since logs are already
sent on a central host anyway) or at least have /var/log/apache mounted
on a dedicated partition.

--
Antoine "hashar" Musso

[Tim Starling]

On 03/04/12 15:58, Antoine Musso wrote:
> Le 03/04/12 01:30, Leslie Carr a écrit :
>> On some machines /tmp is a separate partition, but / is still filled
>> up. This appears to be due to a huge amount of php errors filling the
>> logs
>
> Either we should stop Apache logging entirely (since logs are already
> sent on a central host anyway) or at least have /var/log/apache mounted
> on a dedicated partition.

I thought so until I read about the output channel feature in rsyslog,
then I decided to try using that instead. I deployed an rsyslog
configuration change which splits off the apache log to a separate
file from /var/log/syslog and limits it to 500MB including archives.
Remote logging will continue to work as normal.

https://gerrit.wikimedia.org/r/#change,4149

-- Tim Starling

[Antoine Musso]

Le 03/04/12 08:22, Tim Starling a écrit :
<snip>

> I thought so until I read about the output channel feature in rsyslog,
> then I decided to try using that instead. I deployed an rsyslog
> configuration change which splits off the apache log to a separate
> file from /var/log/syslog and limits it to 500MB including archives.
> Remote logging will continue to work as normal.
>
> https://gerrit.wikimedia.org/r/#change,4149

I did not know that rsyslog could track file size. I thought about using
logrotate instead, but it is probably not going to guarantee the log
files are under a given threshold.

*gives yet another kudos to Tim*

--
Antoine "hashar" Musso