Help, HDD problem - western digital

[Stealth]

HI all,

Sometime, I lose my second drive (when I'm using rsync for example)

There are some messages like this:

Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] Unhandled error code
Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] Result: hostbyte=0x04 driverbyte=0x00
Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 86 91 05 10 00 00 08 00
Nov 23 15:26:24 DNS-323 user.err kernel: end_request: I/O error, dev sdb, sector 2257650960
Nov 23 15:26:24 DNS-323 user.crit kernel: EXT4-fs error (device sdb2): ext4_find_entry: inode #70516859: (comm rsync) reading directory lblock 0
Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] Unhandled error code
Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] Result: hostbyte=0x04 driverbyte=0x00
Nov 23 15:26:24 DNS-323 user.info kernel: sd 1:0:0:0: [sdb] CDB: cdb[0]=0x28: 28 00 86 91 05 20 00 00 08 00
Nov 23 15:26:24 DNS-323 user.err kernel: end_request: I/O error, dev sdb, sector 2257650976
Nov 23 15:26:24 DNS-323 user.crit kernel: EXT4-fs error (device sdb2): ext4_find_entry: inode #70516861: (comm rsync) reading directory lblock 0

I must reboot for temporary fixing (sometimes the nas was totally hang)

No problem with fsck.ext4, and I have not received mail from smartd daemon, it's like there are too much data between the two drives  

# smartctl --all /dev/sdb
smartctl 5.42 2011-10-20 r3458 [armv5tel-linux-2.6.35.14] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD20EZRX-00DC0B0
Serial Number:    WD-WMC1T0541071
LU WWN Device Id: 5 0014ee 0036e4845
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  ACS-2 (revision not indicated)
Local Time is:    Fri Nov 23 15:33:17 2012 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (26940) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 255) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x70b5)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   194   174   021    Pre-fail  Always       -       5266
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       10
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       110
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       7
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       5
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       11
194 Temperature_Celsius     0x0022   114   113   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       1
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   188   188   000    Old_age   Offline      -       3896

SMART Error Log Version: 1
ATA Error Count: 1
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 81 hours (3 days + 9 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  04 61 00 53 4f c2 00  Device Fault; Error: ABRT

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  b0 d8 00 01 4f c2 00 08   1d+03:31:19.721  SMART ENABLE OPERATIONS
  e5 00 00 00 00 00 00 08   1d+03:31:19.566  CHECK POWER MODE
  ec 00 01 00 00 00 00 08   1d+03:31:19.413  IDENTIFY DEVICE
  ec 00 01 00 00 00 00 08   1d+03:31:19.395  IDENTIFY DEVICE
  e5 00 00 00 00 00 00 08   1d+03:31:14.568  CHECK POWER MODE

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        98         -
# 2  Extended offline    Interrupted (host reset)      90%        17         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    1.8T    706.8G      1.0T  41% /
tmpfs                    30.3M      7.3M     22.9M  24% /rootmnt
aufs                      1.8T    706.8G      1.0T  41% /
/dev/loop0                5.5M      5.5M         0 100% /rootmnt/rootsq
tmpfs                   104.0M    508.0K    103.5M   0% /tmp
/dev/sda2                 1.8T    706.8G      1.0T  41% /mnt/sda2
/dev/sdb2                 1.8T    706.8G      1.0T  41% /mnt/sdb2

My disk is dead, or I misunderstand something ?

Thanks

Juste one another question, how I can see the swap usage?  With top there is only ram ?

[Stealth]

Another precision

more /etc/misc.conf
MAILTO=xx...@xx.fr
HDSLEEP_LEFT=10
HDPOWER_LEFT=1
HDPOWER_RIGHT=255
HDSLEEP_RIGHT=20
SMARTD_INTERVAL=1440
NTPD_DAEMON=no
NTPD_BOOT=yes


When the disk is "down"

# umount /dev/sdb2
umount: can't umount /dev/sdb2: Invalid argument
# umount -f /dev/sdb2
umount: can't forcibly umount /dev/sdb2: Invalid argument
# mount /dev/sdb2
mount: mounting /dev/sdb2 on /mnt/sdb2 failed: Invalid argument
#

when rsync run:

Top

Mem: 60908K used, 1048K free, 0K shrd, 21328K buff, 4488K cached
CPU:   2% usr  17% sys   0% nic   0% idle  77% io   0% irq   2% sirq
Load average: 1.52 1.45 0.89 2/57 791
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  788   787 root     R    17300  28%  18% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
   13     2 root     DW       0   0%   1% [kswapd0]
    9     2 root     SW       0   0%   1% [kblockd/0]
  791   709 root     R     1180   2%   1% top
  467     1 root     S      588   1%   0% sysctrl
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
  790   788 root     S    16184  26%   0% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
  787   744 root     S    15988  26%   0% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
  785     1 root     S    10176  16%   0% smbd -D
  789   785 root     S    10176  16%   0% smbd -D


Today no problem, I can't reproduce for the moment

[Joao Cardoso]

On Friday, November 23, 2012 2:16:51 PM UTC, Stealth wrote:

Another precision

more /etc/misc.conf
MAILTO=xx...@xx.fr
HDSLEEP_LEFT=10
HDPOWER_LEFT=1
HDPOWER_RIGHT=255
HDSLEEP_RIGHT=20
SMARTD_INTERVAL=1440
NTPD_DAEMON=no
NTPD_BOOT=yes


When the disk is "down"

# umount /dev/sdb2
umount: can't umount /dev/sdb2: Invalid argument

After the hardware error the disc disappeared from the system, as well 

as its partition, so sdb2 doesn't exists.

# umount -f /dev/sdb2
umount: can't forcibly umount /dev/sdb2: Invalid argument
# mount /dev/sdb2
mount: mounting /dev/sdb2 on /mnt/sdb2 failed: Invalid argument
#

when rsync run:

Top

Mem: 60908K used, 1048K free, 0K shrd, 21328K buff, 4488K cached
CPU:   2% usr  17% sys   0% nic   0% idle  77% io   0% irq   2% sirq
Load average: 1.52 1.45 0.89 2/57 791
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  788   787 root     R    17300  28%  18% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
   13     2 root     DW       0   0%   1% [kswapd0]
    9     2 root     SW       0   0%   1% [kblockd/0]
  791   709 root     R     1180   2%   1% top
  467     1 root     S      588   1%   0% sysctrl
    3     2 root     SW       0   0%   0% [ksoftirqd/0]
  790   788 root     S    16184  26%   0% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
  787   744 root     S    15988  26%   0% rsync -a --delete-after --stats /mnt/sda2/ /mnt/sdb2/
  785     1 root     S    10176  16%   0% smbd -D
  789   785 root     S    10176  16%   0% smbd -D


Today no problem, I can't reproduce for the moment

 It seems to be an intermittent problem, but there are hardware errors in

both the kernel logs and on the SMART log.

The disk seems to be new... I would keep and eye on it, running a small

SMART test everyday and a long one perhaps weekly  I would also activate

"Scans the drive every four hours for disk defects". This will test  

stress the drive. (Services->System->smart)

You can also try to remove the disks and open and air-blow  the box,

just in case there are some electrical bad contacts (there is a Topic

where someone found (real) bugs inside the box :-)

And, needless to say, do a backup, ;-)

[Stealth]

 It seems to be an intermittent problem, but there are hardware errors in

both the kernel logs and on the SMART log.

The disk seems to be new... I would keep and eye on it, running a small

SMART test everyday and a long one perhaps weekly  I would also activate

"Scans the drive every four hours for disk defects". This will test  

stress the drive. (Services->System->smart)

You can also try to remove the disks and open and air-blow  the box,

just in case there are some electrical bad contacts (there is a Topic

where someone found (real) bugs inside the box :-)

And, needless to say, do a backup, ;-)

 

Thank for your answer

Yes I had removed the disk I think there is no contact problem. It's like the HDD is exhausted ;-)
I will try with your advice about smartd

Maybe this is just a kind of memory limitation, there are only 1048K free in my previous post and it was a rsync with no files exchange - where I can see my swap usage ?

Also, just for my comprehension

HDSLEEP_LEFT=10 -> sleep after ten minutes of inactivity
HDPOWER_LEFT=1 -> I don't understand that ?
HDPOWER_RIGHT=255 > ?
HDSLEEP_RIGHT=20 -> sleep after 20 minutes of inactivity
 

[Joao Cardoso]

On Friday, November 23, 2012 3:08:25 PM UTC, Stealth wrote:

 It seems to be an intermittent problem, but there are hardware errors in

both the kernel logs and on the SMART log.

The disk seems to be new... I would keep and eye on it, running a small

SMART test everyday and a long one perhaps weekly  I would also activate

"Scans the drive every four hours for disk defects". This will test  

stress the drive. (Services->System->smart)

You can also try to remove the disks and open and air-blow  the box,

just in case there are some electrical bad contacts (there is a Topic

where someone found (real) bugs inside the box :-)

And, needless to say, do a backup, ;-)

 

Thank for your answer

Yes I had removed the disk I think there is no contact problem. It's like the HDD is exhausted ;-)
I will try with your advice about smartd

Maybe this is just a kind of memory limitation, there are only 1048K free in my previous post and it was a rsync with no files exchange - where I can see my swap usage ?

Use the 'free' command, or 'cat /proc/swaps'

 

Also, just for my comprehension  

HDSLEEP_LEFT=10 -> sleep after ten minutes of inactivity
HDPOWER_LEFT=1 -> I don't understand that ?

That has to do with disk power saving, as set in Disk->Utilities.

That is an old APM standard, and some new drives don't support it. It will affect spindown.

You can see the disk capabilities using 'hdparam -i /dev/sdx' (or '-I') and watch for (on my old disks)

AdvancedPM=yes: disabled (255)

AdvancedPM=no

[FredB]

That has to do with disk power saving, as set in Disk->Utilities.

That is an old APM standard, and some new drives don't support it. It will affect spindown.

You can see the disk capabilities using 'hdparam -i /dev/sdx' (or '-I') and watch for (on my old disks)

AdvancedPM=yes: disabled (255)

AdvancedPM=no

 

HDPOWER_RIGHT=255 > ?
HDSLEEP_RIGHT=20 -> sleep after 20 minutes of inactivity
 

--

hdparm -i /dev/sda

/dev/sda:
hdparm: ioctl 0x304 failed: Inappropriate ioctl for device

 Model=WDC WD20EZRX-00DC0B0                    , FwRev=80.00A80, SerialNo=     WD-WMC1T0568311
 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0
 BuffType=(0) unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?0?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: Unspecified:  ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7 ATA/ATAPI-8 ATA/ATAPI-9

 * current active mode

Same answer for sdb, so I should put HDPOWER_LEFT=255 ?

[Joao Cardoso]

Don't remember, use the web page and read the tooltips and/or the online help.

[FredB]

This morming the disk is dead, ok I will contact WD
support

Thank for your help