Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Why syslog is not rotating?

928 views
Skip to first unread message

Itay

unread,
Nov 2, 2013, 4:10:01 PM11/2/13
to
Hi,

Can someone help me, please, to understand why syslog is not rotating?

# ls -gh /var/log/syslog*
-rw-r----- 1 adm 219M Nov 2 21:50 syslog
-rw-r----- 1 adm 2.5K Jun 5 2010 syslog.1
-rw-r----- 1 adm 0 Nov 1 07:50 syslog.1.gz
-rw-r----- 1 adm 661 Jun 5 2010 syslog.2.gz
-rw-r----- 1 adm 709 Jun 4 2010 syslog.3.gz
-rw-r----- 1 adm 689 Jun 3 2010 syslog.4.gz
-rw-r----- 1 adm 678 Jun 2 2010 syslog.5.gz
-rw-r----- 1 adm 675 Jun 1 2010 syslog.6.gz
-rw-r----- 1 adm 664 May 31 2010 syslog.7.gz

In case it is needed, here is /etc/logrotate.d/rsyslog:

--------------[BEGIN]--------------
/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
--------------[END]--------------

Thanks in advance,
Itay



--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Sven Hartge

unread,
Nov 2, 2013, 4:50:01 PM11/2/13
to
Itay <deb...@itayf.fastmail.fm> wrote:

> Can someone help me, please, to understand why syslog is not rotating?

The system ist not running at the time when cron.daily is scheduled to
run? --> anacron takes care of that.

Or somehow there is an error and logrotate refuses to run. In that case
run logrotate manually with the debug-switch "-d" and see if anything
strange appears.

Grüße,
Sven.

--
Sigmentation fault. Core dumped.


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1a66d7...@mids.svenhartge.de

Itay

unread,
Nov 3, 2013, 2:10:02 AM11/3/13
to
On Sat, 2 Nov 2013, Sven Hartge wrote:

> Date: Sat, 2 Nov 2013 21:47:11 +0100
> From: Sven Hartge <sv...@svenhartge.de>
> To: debia...@lists.debian.org
> Subject: Re: Why syslog is not rotating?
> Resent-Date: Sat, 2 Nov 2013 20:48:34 +0000 (UTC)
> Resent-From: debia...@lists.debian.org
>
> Itay <deb...@itayf.fastmail.fm> wrote:
>
>> Can someone help me, please, to understand why syslog is not rotating?
>
> The system ist not running at the time when cron.daily is scheduled to
> run? --> anacron takes care of that.

Sven,

Thank you for your reply.

My system was continuously on except for very short random periods and
3 weeks on Aug 2013. In contrast, the listing below shows (I believe)
that syslog stopped rotating at 2010.

# ls -gh /var/log/syslog*
-rw-r----- 1 adm 219M Nov 2 21:50 syslog
-rw-r----- 1 adm 2.5K Jun 5 2010 syslog.1
-rw-r----- 1 adm 0 Nov 1 07:50 syslog.1.gz
-rw-r----- 1 adm 661 Jun 5 2010 syslog.2.gz

However:
I checked /etc/cron.daily and did not find entry for rsyslog.
Maybe that's the cause?

> Or somehow there is an error and logrotate refuses to run. In that case
> run logrotate manually with the debug-switch "-d" and see if anything
> strange appears.
>

Is logrotate responsible for rotating syslog files?
Information on the net [1] suggests it's not.
-----------------------------------
[1] https://wiki.debian.org/Rsyslog

I am confused...

Finally, I should point out that, I have just completed dist-upgrade
from squeeze to wheezy with no major problems.
This issue came up as I was cleaning-up after the upgrade.

Many thanks,
Itay

> Grüße,
> Sven.
>
>


Shawn Wilson

unread,
Nov 3, 2013, 4:10:02 AM11/3/13
to

Can syslog rotate logs? I just use logrotate.



Reco

unread,
Nov 3, 2013, 4:30:02 AM11/3/13
to
Hi.

On Sun, 3 Nov 2013 09:04:36 +0200 (IST)
Itay <deb...@itayf.fastmail.fm> wrote:

> On Sat, 2 Nov 2013, Sven Hartge wrote:
> Thank you for your reply.
>
> My system was continuously on except for very short random periods and
> 3 weeks on Aug 2013. In contrast, the listing below shows (I believe)
> that syslog stopped rotating at 2010.
>
> # ls -gh /var/log/syslog*
> -rw-r----- 1 adm 219M Nov 2 21:50 syslog
> -rw-r----- 1 adm 2.5K Jun 5 2010 syslog.1
> -rw-r----- 1 adm 0 Nov 1 07:50 syslog.1.gz
> -rw-r----- 1 adm 661 Jun 5 2010 syslog.2.gz
>
> However:
> I checked /etc/cron.daily and did not find entry for rsyslog.
> Maybe that's the cause?

On a stock Debian system logrotate is used to rotate rsyslog
logfiles. This is configured in /etc/logrotate.d/rsyslog.
Logrotate is invoked at /etc/cron.daily/logrotate.

Now, that listing shows that someone (possibly logrotate) DID create an
empty syslog.1.gz file (on 1st Nov 2013), and that suggests that
logrotate is misconfigured somehow.

Can you please post a contents of /etc/logrotate.d/rsyslog?

Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131103132051.55ec...@gmail.com

Itay

unread,
Nov 3, 2013, 4:40:02 AM11/3/13
to
On Sun, 3 Nov 2013, Reco wrote:

> Date: Sun, 3 Nov 2013 13:20:51 +0400
> From: Reco <recov...@gmail.com>
> To: debia...@lists.debian.org
> Cc: Itay <deb...@itayf.fastmail.fm>
> Subject: Re: Why syslog is not rotating?
> Resent-Date: Sun, 3 Nov 2013 09:21:12 +0000 (UTC)
> Resent-From: debia...@lists.debian.org
>
> On Sun, 3 Nov 2013 09:04:36 +0200 (IST)
> Itay <deb...@itayf.fastmail.fm> wrote:
>
>> On Sat, 2 Nov 2013, Sven Hartge wrote:
>> Thank you for your reply.
>>
>> My system was continuously on except for very short random periods and
>> 3 weeks on Aug 2013. In contrast, the listing below shows (I believe)
>> that syslog stopped rotating at 2010.
>>
>> # ls -gh /var/log/syslog*
>> -rw-r----- 1 adm 219M Nov 2 21:50 syslog
>> -rw-r----- 1 adm 2.5K Jun 5 2010 syslog.1
>> -rw-r----- 1 adm 0 Nov 1 07:50 syslog.1.gz
>> -rw-r----- 1 adm 661 Jun 5 2010 syslog.2.gz
>>
>> However:
>> I checked /etc/cron.daily and did not find entry for rsyslog.
>> Maybe that's the cause?
>
> On a stock Debian system logrotate is used to rotate rsyslog
> logfiles. This is configured in /etc/logrotate.d/rsyslog.
> Logrotate is invoked at /etc/cron.daily/logrotate.
>
> Now, that listing shows that someone (possibly logrotate) DID create an
> empty syslog.1.gz file (on 1st Nov 2013), and that suggests that
> logrotate is misconfigured somehow.
>
> Can you please post a contents of /etc/logrotate.d/rsyslog?
>
> Reco
>

Reposting:

----------[Begin: /etc/logrotate.d/rsyslog]----------
----------[End: /etc/logrotate.d/rsyslog]----------

I appreciate the help.
Itay




--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Itay

unread,
Nov 3, 2013, 4:40:02 AM11/3/13
to

On Sun, 3 Nov 2013, Shawn Wilson wrote:

> Date: Sun, 03 Nov 2013 04:07:45 -0500
> From: Shawn Wilson <ag4v...@gmail.com>
> To: Itay <deb...@itayf.fastmail.fm>, Itay <deb...@itayf.fastmail.fm>,
> debia...@lists.debian.org
> Subject: Re: Why syslog is not rotating?
> Resent-Date: Sun, 3 Nov 2013 09:08:46 +0000 (UTC)
> Resent-From: debia...@lists.debian.org
>
> Can syslog rotate logs? I just use logrotate.

Syslog cannot rotate logs.
I misunderstood the information on the internet.

It's better is someone authorative would explain who rotates who...;-)

Thanks,
Itay





--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Reco

unread,
Nov 3, 2013, 4:50:02 AM11/3/13
to
Looks that's a stock one.
Try it like this:

1) Invoke as a root:

/usr/sbin/logrotate /etc/logrotate.conf


2) If it doesn't help, add 'size' stanza to
the /etc/logrotate.d/rsyslog like this:

/var/log/syslog
{
rotate 7
daily
missingok
notifempty
delaycompress
compress
size 1024k
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}

and invoke logrotate once more:

/usr/sbin/logrotate /etc/logrotate.conf

Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131103134842.12ce...@gmail.com

Itay

unread,
Nov 3, 2013, 7:30:01 AM11/3/13
to
On Sun, 3 Nov 2013, Reco wrote:

> Date: Sun, 3 Nov 2013 13:48:42 +0400
> Subject: Re: Why syslog is not rotating?
> Resent-Date: Sun, 3 Nov 2013 09:49:15 +0000 (UTC)
> Resent-From: debia...@lists.debian.org
>
> Looks that's a stock one.
> Try it like this:
>
> 1) Invoke as a root:
>
> /usr/sbin/logrotate /etc/logrotate.conf

Didn't work.

> 2) If it doesn't help, add 'size' stanza to
> the /etc/logrotate.d/rsyslog like this:
>
> /var/log/syslog
> {
> rotate 7
> daily
> missingok
> notifempty
> delaycompress
> compress
> size 1024k
> postrotate
> invoke-rc.d rsyslog rotate > /dev/null
> endscript
> }
>
> and invoke logrotate once more:
>
> /usr/sbin/logrotate /etc/logrotate.conf

Not yet, but we have some progress...

Trial 1:

# /usr/sbin/logrotate /etc/logrotate.conf
error: error creating output file /var/log/syslog.1.gz: File exists

Trial 2:
# rm /var/log/syslog.1.gz
# /usr/sbin/logrotate /etc/logrotate.conf
gzip: stdin: Input/output error
error: failed to compress log /var/log/syslog.1

Should I backup syslog, delete it, and watch how things evolve?

Many thanks,
Itay

> Reco
>
>
>


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Reco

unread,
Nov 3, 2013, 8:40:02 AM11/3/13
to
On Sun, 3 Nov 2013 14:25:38 +0200 (IST)
Itay <deb...@itayf.fastmail.fm> wrote:

> On Sun, 3 Nov 2013, Reco wrote:
>
> Not yet, but we have some progress...
>
> Trial 1:
>
> # /usr/sbin/logrotate /etc/logrotate.conf
> error: error creating output file /var/log/syslog.1.gz: File exists
>
> Trial 2:
> # rm /var/log/syslog.1.gz
> # /usr/sbin/logrotate /etc/logrotate.conf
> gzip: stdin: Input/output error
> error: failed to compress log /var/log/syslog.1
>
> Should I backup syslog, delete it, and watch how things evolve?
>

Now that's interesting. Is there anything similar to this messages
in /var/log/cron.log?
Does, say, 'md5sum /var/log/syslog' runs to the completion?
What about 'cat /var/log/syslog > /dev/null'?
Can you run fsck on the filesystem containing /var/log/syslog?
What does smartctl --all shows on the partition with this filesystem?

Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131103173017.4122...@gmail.com

Itay

unread,
Nov 3, 2013, 10:20:02 AM11/3/13
to
On Sun, 3 Nov 2013, Reco wrote:

> Date: Sun, 3 Nov 2013 17:30:17 +0400
> From: Reco <recov...@gmail.com>
> To: debia...@lists.debian.org
> Subject: Re: Why syslog is not rotating?
> Resent-Date: Sun, 3 Nov 2013 13:30:47 +0000 (UTC)
> Resent-From: debia...@lists.debian.org
>
> On Sun, 3 Nov 2013 14:25:38 +0200 (IST)
> Itay <deb...@itayf.fastmail.fm> wrote:
>
>> On Sun, 3 Nov 2013, Reco wrote:
>>
>> Not yet, but we have some progress...
>>
>> Trial 1:
>>
>> # /usr/sbin/logrotate /etc/logrotate.conf
>> error: error creating output file /var/log/syslog.1.gz: File exists
>>
>> Trial 2:
>> # rm /var/log/syslog.1.gz
>> # /usr/sbin/logrotate /etc/logrotate.conf
>> gzip: stdin: Input/output error
>> error: failed to compress log /var/log/syslog.1
>>
>> Should I backup syslog, delete it, and watch how things evolve?
>>
>
> Now that's interesting. Is there anything similar to this messages
> in /var/log/cron.log?

Hmmmm... there is no /var/log/cron.log !!

> Does, say, 'md5sum /var/log/syslog' runs to the completion?

Yes. Without warnings/errors.

> What about 'cat /var/log/syslog > /dev/null'?

Yes. Without warnings/errors.

> Can you run fsck on the filesystem containing /var/log/syslog?

I have to unmount /var for that; right?
So I need to use Live CD for that; right?

> What does smartctl --all shows on the partition with this filesystem?

I never used smartctl (installed it now following-up your question).
In my system /var resides on a logical volume.
So I am not sure how to proceed.

> Reco

Many thanks for the help
Itay


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Reco

unread,
Nov 3, 2013, 11:30:02 AM11/3/13
to
On Sun, 3 Nov 2013 17:16:02 +0200 (IST)
Itay <deb...@itayf.fastmail.fm> wrote:

> On Sun, 3 Nov 2013, Reco wrote:
> > Now that's interesting. Is there anything similar to this messages
> > in /var/log/cron.log?
>
> Hmmmm... there is no /var/log/cron.log !!

Sorry, my mistake. I have an old installation, /var/log/cron.log is a
leftover of etch's setup in my case. Do you have any MTA
(exim, sendmail or postfix) installed? Anytime cron job puts anything
to the stderr cron should send mail to the local root (default
settings). Is there anything suspicious in the root mailbox?
And, is there anything unusual in /var/log/kern.log at the time you
had this error?


> > Does, say, 'md5sum /var/log/syslog' runs to the completion?
>
> Yes. Without warnings/errors.
>
> > What about 'cat /var/log/syslog > /dev/null'?
>
> Yes. Without warnings/errors.

Ok. What about 'cat /var/log/syslog | gzip -c > /dev/null'?
And, while we're at that, what about:

cat /var/log/syslog | gzip -c > /var/log/syslog.test.gz

If error shows early, can you also post contents of (/tmp/gzip):

strace -fo /tmp/gzip cat /var/log/syslog | gzip -c > /dev/null


> > Can you run fsck on the filesystem containing /var/log/syslog?
>
> I have to unmount /var for that; right?

Yes.


> So I need to use Live CD for that; right?

Sure, that's possible to do with livecd. But, you can also do it from a
single-user (i.e. init 1; unmount /var; run fsck on a logical volume;
reboot).


> > What does smartctl --all shows on the partition with this filesystem?
>
> I never used smartctl (installed it now following-up your question).
> In my system /var resides on a logical volume.
> So I am not sure how to proceed.

Find a physical volume corresponding to the /var logical volume.
Run smartctl --all on the disk that's containing that physical volume.
In case you have RAID (be it mdadm or dm-mirror) - run smartctl on all
disks that are part of said RAID.

While we're on it, also run smartctl -t long on said disk, wait for a
while (smartctl should say you, how much), and run smartctl --all on
the same disk again.

Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131103202659.268f...@gmail.com

Itay

unread,
Nov 4, 2013, 4:00:02 AM11/4/13
to
On Sun, 3 Nov 2013, Reco wrote:

> On Sun, 3 Nov 2013 17:16:02 +0200 (IST)
> Itay <deb...@itayf.fastmail.fm> wrote:
>
>> On Sun, 3 Nov 2013, Reco wrote:

> [...] Is there anything suspicious in the root mailbox?

root mail box has daily messages like this starting at june 2010
(yes, I know, bad me)

/etc/cron.daily/logrotate:

gzip: stdin: Input/output error
error: failed to compress log /var/log/syslog.1
run-parts: /etc/cron.daily/logrotate exited with return code 1

> And, is there anything unusual in /var/log/kern.log at the time you
> had this error?

Multiple messages like those two:

...
Oct 31 07:59:35 gandalf kernel: [4627180.405646] ata3.00: exception
Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Oct 31 07:59:35 gandalf kernel: [4627180.405650] ata3.00: irq_stat
0x40000008
Oct 31 07:59:35 gandalf kernel: [4627180.405653] ata3.00: failed
command: READ FPDMA QUEUED
Oct 31 07:59:35 gandalf kernel: [4627180.405659] ata3.00: cmd
60/08:00:cb:05:a9/00:00:05:00:00/40 tag 0 ncq 4096 in
Oct 31 07:59:35 gandalf kernel: [4627180.405661] res
41/40:00:cd:05:a9/00:00:05:00:00/40 Emask 0x409 (media error) <F>
Oct 31 07:59:35 gandalf kernel: [4627180.405664] ata3.00: status: {
DRDY ERR }
Oct 31 07:59:35 gandalf kernel: [4627180.405666] ata3.00: error: { UNC
}
Oct 31 07:59:35 gandalf kernel: [4627180.407143] ata3.00: configured
for UDMA/133
Oct 31 07:59:35 gandalf kernel: [4627180.407153] sd 2:0:0:0: [sda]
Unhandled sense code
Oct 31 07:59:35 gandalf kernel: [4627180.407155] sd 2:0:0:0: [sda]
Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 31 07:59:35 gandalf kernel: [4627180.407158] sd 2:0:0:0: [sda]
Sense Key : Medium Error [current] [descriptor]
Oct 31 07:59:35 gandalf kernel: [4627180.407163] Descriptor sense data
with sense descriptors (in hex):
Oct 31 07:59:35 gandalf kernel: [4627180.407165] 72 03 11 04
00 00 00 0c 00 0a 80 00 00 00 00 00
Oct 31 07:59:35 gandalf kernel: [4627180.407173] 05 a9 05 cd
Oct 31 07:59:35 gandalf kernel: [4627180.407176] sd 2:0:0:0: [sda]
Add. Sense: Unrecovered read error - auto reallocate failed
Oct 31 07:59:35 gandalf kernel: [4627180.407181] sd 2:0:0:0: [sda]
CDB: Read(10): 28 00 05 a9 05 cb 00 00 08 00
Oct 31 07:59:35 gandalf kernel: [4627180.407188] end_request: I/O
error, dev sda, sector 94963149
Oct 31 07:59:35 gandalf kernel: [4627180.407208] ata3: EH complete
...
Nov 1 07:50:21 gandalf kernel: [4713026.178488] ata3.00: exception
Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Nov 1 07:50:21 gandalf kernel: [4713026.178492] ata3.00: irq_stat
0x40000008
Nov 1 07:50:21 gandalf kernel: [4713026.178496] ata3.00: failed
command: READ FPDMA QUEUED
Nov 1 07:50:21 gandalf kernel: [4713026.178502] ata3.00: cmd
60/08:00:cb:05:a9/00:00:05:00:00/40 tag 0 ncq 4096 in
Nov 1 07:50:21 gandalf kernel: [4713026.178503] res
41/40:00:cd:05:a9/00:00:05:00:00/40 Emask 0x409 (media error) <F>
Nov 1 07:50:21 gandalf kernel: [4713026.178506] ata3.00: status: {
DRDY ERR }
Nov 1 07:50:21 gandalf kernel: [4713026.178509] ata3.00: error: { UNC
}
Nov 1 07:50:21 gandalf kernel: [4713026.179984] ata3.00: configured
for UDMA/133
Nov 1 07:50:21 gandalf kernel: [4713026.179992] ata3: EH complete
...

>>> Does, say, 'md5sum /var/log/syslog' runs to the completion?
>>
>> Yes. Without warnings/errors.
>>
>>> What about 'cat /var/log/syslog > /dev/null'?
>>
>> Yes. Without warnings/errors.
>
> Ok. What about 'cat /var/log/syslog | gzip -c > /dev/null'?
> And, while we're at that, what about:
>
> cat /var/log/syslog | gzip -c > /var/log/syslog.test.gz

Both commands finished without warnings/errors.

> If error shows early, can you also post contents of (/tmp/gzip):
>
> strace -fo /tmp/gzip cat /var/log/syslog | gzip -c > /dev/null

Didn't try since there were no errors.

>>> Can you run fsck on the filesystem containing /var/log/syslog?

[snip]

File system was found clean. No errors were reported.

>>> What does smartctl --all shows on the partition with this filesystem?
>>
>> I never used smartctl (installed it now following-up your question).
>> In my system /var resides on a logical volume.
>> So I am not sure how to proceed.
>
> Find a physical volume corresponding to the /var logical volume.
> Run smartctl --all on the disk that's containing that physical volume.
> In case you have RAID (be it mdadm or dm-mirror) - run smartctl on all
> disks that are part of said RAID.
>
> While we're on it, also run smartctl -t long on said disk, wait for a
> while (smartctl should say you, how much), and run smartctl --all on
> the same disk again.

Output of 'smartctl --all' (after running 'smartctl -t long'):

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Blue Serial ATA
Device Model: WDC WD1600AAJS-00L7A0
Serial Number: WD-WCAV34031063
LU WWN Device Id: 5 0014ee 15756c0f2
Firmware Version: 01.03E01
User Capacity: 160,041,885,696 bytes [160 GB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Mon Nov 4 10:42:48 2013 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 121) The previous self-test completed having
the read element of the test failed.
Total time to complete Offline
data collection: ( 3000) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 39) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 10434
3 Spin_Up_Time 0x0027 135 130 021 Pre-fail Always - 4241
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 119
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 060 060 000 Old_age Always - 29269
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 117
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 52
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 119
194 Temperature_Celsius 0x0022 100 093 000 Old_age Always - 43
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 1
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed: read failure 90% 29267 94963149

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

==========================================================
End of 'smartctl --all' output.

Many thanks for the help and the patience!
Itay

> Reco
>
>
>


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Shawn Wilson

unread,
Nov 4, 2013, 5:00:01 AM11/4/13
to
I can't see how a default config would do this, but do you have SELinux or AppArmor enabled? What does fstab and mount show?

If possible, copy the system off and write ones and then zeros to the disk (and look for speed drops). Looking at the disk might've been a good call.
Archive: http://lists.debian.org/ec494183-9aa0-410f...@email.android.com

Reco

unread,
Nov 4, 2013, 5:40:01 AM11/4/13
to
Well, I have good news and bad news.


On Mon, 4 Nov 2013 10:57:18 +0200 (IST)
Itay <deb...@itayf.fastmail.fm> wrote:

> > [...] Is there anything suspicious in the root mailbox?
>
> root mail box has daily messages like this starting at june 2010
> (yes, I know, bad me)
>
> /etc/cron.daily/logrotate:
>
> gzip: stdin: Input/output error
> error: failed to compress log /var/log/syslog.1
> run-parts: /etc/cron.daily/logrotate exited with return code 1

The good news are - both cron and logrotate are working as intended on
your system. At least, they try their best.


> > And, is there anything unusual in /var/log/kern.log at the time you
> > had this error?
>
> Multiple messages like those two:
>
> ...
> Oct 31 07:59:35 gandalf kernel: [4627180.407176] sd 2:0:0:0: [sda]
> Add. Sense: Unrecovered read error - auto reallocate failed

And the bad news are - your drive is failing. And you've already lost
some data (best scenario - some contents of /var/log/syslog).


> Output of 'smartctl --all' (after running 'smartctl -t long'):
<skip>
> 9 Power_On_Hours 0x0032 060 060 000 Old_age Always - 29269

That's an old WD harddrive, and it run for about 3 years continuously.
These things aren't get better with age.


<skip>
> 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1
> 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 1

And these show that you've already lost one 512 byte sector on that
disk irrecoverably.


>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
> # 1 Extended offline Completed: read failure 90% 29267 94963149

This shows the same, with an address of first failing sector.


Shawn already suggested you to replace your harddrive ASAP, I second
this suggestion. In fact, buy two harddrives and do a RAID1 then forget
about the thing for a next few years.

Considering that fsck showed you no errors that means
that /var filesystem metadata is consistent. That's good as it means
you can just copy all files to the new harddrive and filesystem state
won't prevent you to do so. That, sadly, speaks nothing about an
integrity of data itself.

Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131104143226.1407...@gmail.com

Itay

unread,
Nov 4, 2013, 6:00:02 AM11/4/13
to
On Mon, 4 Nov 2013, Reco wrote:

[skip]

> The good news are - both cron and logrotate are working as intended on
> your system. At least, they try their best.

[skip]

> And the bad news are - your drive is failing. And you've already lost
> some data (best scenario - some contents of /var/log/syslog).

[skip detailed instructive explanation of smartctl]

> Shawn already suggested you to replace your harddrive ASAP, I second
> this suggestion. In fact, buy two harddrives and do a RAID1 then forget
> about the thing for a next few years.

I will follow your advice immediately.
(Sorry Shawn, I somehow missed your advice.)

> Considering that fsck showed you no errors that means
> that /var filesystem metadata is consistent. That's good as it means
> you can just copy all files to the new harddrive and filesystem state
> won't prevent you to do so. That, sadly, speaks nothing about an
> integrity of data itself.

Thanks a lot for the help, Reco.
Also to Shawn and Sven for responding.

I'll mark this thread as solved.

Itay

> Reco


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Itay

unread,
Nov 4, 2013, 6:20:01 AM11/4/13
to
On Mon, 4 Nov 2013, Shawn Wilson wrote:

> Date: Mon, 04 Nov 2013 04:46:48 -0500

For some reason your messages ended up in a differenet mailbox,
therefore I saw them only now. Sorry about that.

As was recommended I will replace the harddrive that holds /var.

Nevertheless, respectfully, I will answer your response.

> I can't see how a default config would do this, but do you have SELinux or AppArmor enabled? What does fstab and mount show?

I didn't enable SElinux or AppArmor.
How do I double-check?

mount output:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=473761,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=380288k,mode=755)
/dev/disk/by-uuid/4712a4c6-be99-43bf-8628-0edd641d5262 on / type ext3 (rw,relatime,errors=remount-ro,barrier=1,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=2323680k)
/dev/sda1 on /boot type ext2 (rw,relatime,errors=continue)
/dev/mapper/vg-cache on /cache type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-home on /home type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-opt on /opt type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-tmp on /tmp type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-usr on /usr type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-usrlocal on /usr/local type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/mapper/vg-var on /var type ext3 (rw,relatime,errors=continue,barrier=1,data=ordered)
/dev/sdb8 on /legacy type ext3 (rw,nosuid,nodev,noexec,relatime,errors=continue,barrier=1,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

And /etc/fstab:

# /etc/fstab: static file system information.

# !! Note !!
# UUID is a property of the residing file system -- not of the device.
# Any file system reformatting will result in change to UUID.

# <file system> <mount point> <type> <options> <dump> <pass>

####################### Static Devices #######################

proc /proc proc defaults 0 0

# Master Hard Drive Partitions

# 1: boot (purposefully an ext2 partition)
UUID=81d775af-9f80-4081-be71-c8d379c05944 /boot ext2 defaults 0 2
# 2: root file system
UUID=4712a4c6-be99-43bf-8628-0edd641d5262 / ext3 errors=remount-ro 0 1
# 3: swap partition
UUID=73035ad5-bba9-4d8e-8cfd-546ab09ac456 none swap sw 0 0

# LVM
/dev/mapper/vg-cache /cache ext3 defaults 0 2
/dev/mapper/vg-home /home ext3 defaults 0 2
/dev/mapper/vg-opt /opt ext3 defaults 0 2
/dev/mapper/vg-tmp /tmp ext3 defaults 0 2
/dev/mapper/vg-usr /usr ext3 defaults 0 2
/dev/mapper/vg-usrlocal /usr/local ext3 defaults 0 2
/dev/mapper/vg-var /var ext3 defaults 0 2


###################### Removable Media ######################

/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
#/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0

# USB: Gandalf Attached Storage.
UUID=6b01200d-446d-46da-b6ac-405f1ec5b1bf /legacy ext3 user 0 0
UUID=0866a19d-3661-422b-818c-410d96868139 /media/gandalfBACK ext3 noauto,user 0 0
# LABEL="gandalfBACK2" Gandalf's secondary backup (kept at my parents' house)
UUID=14a0b647-02e0-4681-b37c-cea165004e24 /media/gandalfBACK2 ext3 noauto 0 0

# USB: Portable disk.
UUID=ac334401-5833-4946-bc7b-4d93622ea929 /media/palm/legacy ext3 noauto,user 0 0
UUID=807e9825-1838-4d5c-bada-e54a7d94e2ff /media/palm/lennyBACK ext3 noauto,user 0 0
UUID=3A39-B1B2 /media/palm/reut12 vfat rw,noauto,user 0 0

# USB: Ora's portable disk.
UUID=57e7b504-442c-396e-8c77-dcdd9e6b54d6 /media/usbora hfsplus rw,noauto,user 0 0

# USB: Nikon D300, 8, 4, & 2, GB cards
UUID=4DCC-F3FA /media/D300-8 vfat ro,noauto,user 0 0
UUID=888E-E0E0 /media/D300-4 vfat ro,noauto,user 0 0
UUID=C496-1651 /media/D300-2 vfat ro,noauto,user 0 0
# USB: Card reader
LABEL=CF /media/D300 vfat ro,noauto,user 0 0

# USB: bilbo's backup
LABEL=bilboBACK /media/usbdisk ext3 rw,noauto,user 0 0

# USB: My good old stick
LABEL=PUBLIC /media/PUBLIC vfat rw,noauto,user 0 0

# USB: Little cyan cruize blade (LABEL='ItayF sky')
UUID=C042-86D6 /media/sky vfat rw,noauto,user 0 0
# USB: Little red cruize blade (LABEL='ItayF fire')
UUID=782C-9B8B /media/fire vfat rw,noauto,user 0 0


> If possible, copy the system off and write ones and then zeros to the disk (and look for speed drops). Looking at the disk might've been a good call.

I don't understand the procedure you suggest, but it's not necessary
after deciding to replace the HDD anyhow -- I guess.

Many thanks,
Itay


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

shawn wilson

unread,
Nov 4, 2013, 8:30:02 AM11/4/13
to
Raco already basically said what the issue is. I didn't notice it in
the prior messages you posted. If you wanted to confirm, after you
migrate your data, do a:
dd if=/dev/zero of=<old disk>
and then looking at something like:
iostat -xtc 5 /dev/<old disk>

And, if you don't see an error there, do
dd if=<(tr '\000' '\377' < /dev/zero) of=<old disk>
and look at the iostat for that.

Even if the disk won't tell the kernel it's breaking, chances are dd
will error. But, even if that doesn't happen, you'd notice a big slow
down with iostat.

As it is, disks are cheap - get another one and be done with the old one.
Archive: http://lists.debian.org/CAH_OBif-DVW1L6nyVzkt_ft7...@mail.gmail.com

Paul E Condon

unread,
Nov 4, 2013, 10:40:02 AM11/4/13
to
On 20131104_105718, Itay wrote:
> On Sun, 3 Nov 2013, Reco wrote:
>
> >On Sun, 3 Nov 2013 17:16:02 +0200 (IST)
> >Itay <deb...@itayf.fastmail.fm> wrote:
> >
> >>On Sun, 3 Nov 2013, Reco wrote:
>
> >[...] Is there anything suspicious in the root mailbox?
>
> root mail box has daily messages like this starting at june 2010
> (yes, I know, bad me)

A lot has changed since I was skilled at Debian install, but I
remember that there was a way to automatically forward all emails
that are sent to root to some other mailbox, like your own user
mailbox. That way you would have become aware of your problem
back in june 2010. Whatever you do to fix logrotate problem, don't
forget to root mail forwarding to a place where you regularly
read your mail. Do this by adding a line to /etc/aliases (I think)




--
Paul E Condon
peco...@mesanetworks.net


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20131104131842.GA20170@big

Dan Ritter

unread,
Nov 5, 2013, 10:10:02 AM11/5/13
to
On Mon, Nov 04, 2013 at 06:18:42AM -0700, Paul E Condon wrote:
> On 20131104_105718, Itay wrote:
> > On Sun, 3 Nov 2013, Reco wrote:
> >
> > >On Sun, 3 Nov 2013 17:16:02 +0200 (IST)
> > >Itay <deb...@itayf.fastmail.fm> wrote:
> > >
> > >>On Sun, 3 Nov 2013, Reco wrote:
> >
> > >[...] Is there anything suspicious in the root mailbox?
> >
> > root mail box has daily messages like this starting at june 2010
> > (yes, I know, bad me)
>
> A lot has changed since I was skilled at Debian install, but I
> remember that there was a way to automatically forward all emails
> that are sent to root to some other mailbox, like your own user
> mailbox. That way you would have become aware of your problem
> back in june 2010. Whatever you do to fix logrotate problem, don't
> forget to root mail forwarding to a place where you regularly
> read your mail. Do this by adding a line to /etc/aliases (I think)

Yes. In /etc/aliases, make sure that there is a line like this:

root: deb...@itayf.fastmail.fm

(use whatever address is desired.)

and then, as root:

# newaliases

Then send an email to root to check that mail is going where you
want it to go.

This method works for machines running exim (Debian default),
postfix, and sendmail. If you run something else, you are
expected to be able to figure it out from the documentation.

-dsr-


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2013110514...@randomstring.org

Itay

unread,
Nov 6, 2013, 3:50:02 AM11/6/13
to
On Mon, 4 Nov 2013, Paul E Condon wrote:

> On 20131104_105718, Itay wrote:
>> On Sun, 3 Nov 2013, Reco wrote:
>>
>>> On Sun, 3 Nov 2013 17:16:02 +0200 (IST)
>>> Itay <deb...@itayf.fastmail.fm> wrote:
>>>
>>>> On Sun, 3 Nov 2013, Reco wrote:
>>
>>> [...] Is there anything suspicious in the root mailbox?
>>
>> root mail box has daily messages like this starting at june 2010
>> (yes, I know, bad me)
>
> A lot has changed since I was skilled at Debian install, but I
> remember that there was a way to automatically forward all emails
> that are sent to root to some other mailbox, like your own user
> mailbox. That way you would have become aware of your problem
> back in june 2010. Whatever you do to fix logrotate problem, don't
> forget to root mail forwarding to a place where you regularly
> read your mail. Do this by adding a line to /etc/aliases (I think)

Actually, I had this implemented (but forgot all about it...).
The problem was that my mail client wasn't monitoring my local
mailbox. (Yes, I know, emberassing... I'm not a skilled sys-admin.)
After the inquiries by Reco and Shawn I checked it and found the said
messages.

Following your suggestion I edited /etc/aliases such that root mail
will be sent to my personal mailbox, and a copy will be saved locally
in root's mailbox. Now the root entry looks like this

root: root,pers...@email.address.org

Hopefully I did it right.

Thanks for the pointer!

Itay




--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet

Itay

unread,
Nov 6, 2013, 4:30:03 AM11/6/13
to
On Tue, 5 Nov 2013, Dan Ritter wrote:

> On Mon, Nov 04, 2013 at 06:18:42AM -0700, Paul E Condon wrote:

>> On 20131104_105718, Itay wrote:

>>> root mail box has daily messages like this starting at june 2010
>>> (yes, I know, bad me)
>>
>> A lot has changed since I was skilled at Debian install, but I
>> remember that there was a way to automatically forward all emails
>> that are sent to root to some other mailbox, like your own user
>> mailbox. That way you would have become aware of your problem
>> back in june 2010. Whatever you do to fix logrotate problem, don't
>> forget to root mail forwarding to a place where you regularly
>> read your mail. Do this by adding a line to /etc/aliases (I think)
>
> Yes. In /etc/aliases, make sure that there is a line like this:
>
> root: deb...@itayf.fastmail.fm
>
> (use whatever address is desired.)
>
> and then, as root:
>
> # newaliases
>
> Then send an email to root to check that mail is going where you
> want it to go.

It seems that exim4 is not configured, on my machine, to deliver mail
to external domains. I will have to read how to do it.

Thanks.

Itay


> This method works for machines running exim (Debian default),
> postfix, and sendmail. If you run something else, you are
> expected to be able to figure it out from the documentation.
>
> -dsr-
>
>
>


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/alpine.DEB.2.02.1...@gandalf.furmanet
0 new messages