failing HDD, ddrescue says remaning time is 7104d

[ppr]

I would appreciate advice from the community about a failing hard drive.

When booting up, the computer complained about /dev/sdb, which is a ext4
HDD with data (not the computer main disk). dmesg shows `AE_NOT_FOUND`
and `failed command: READ FPDMA QUEUED` messages (full dmesg log at
https://hastebin.com/raw/jebelileru).

It has finally booted after trying unsuccessfully to start /dev/sdb.

I launched smartctl which shows hard drive failure.

---
# smartctl -H -i /dev/sdb
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-21-amd64] (local
build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke,
www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" DT01ACA... Desktop HDD
Device Model: TOSHIBA DT01ACA100
Serial Number: 663X1XGNS
LU WWN Device Id: 5 000039 fe9dad918
Firmware Version: MS2OA750
User Capacity: 1 000 204 886 016 bytes [1,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed Aug 31 13:56:34 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
2 Throughput_Performance 0x0005 037 037 054 Pre-fail
Offline FAILING_NOW 3774
5 Reallocated_Sector_Ct 0x0033 001 001 005 Pre-fail Always
FAILING_NOW 2004
---

I did not try to mount the HDD. I plugged an external HDD (ext4) and
launched ddrescue. After two days it has recovered 33GB of 1TB but the
speed are now so slow it will take 7104 days to complete.

# ddrescue -n /dev/sdb
/media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/image_HDD1.img
/media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/recup.log
GNU ddrescue 1.23
Press Ctrl-C to interrupt
ipos: 33992 MB, non-trimmed: 0 B, current rate: 636
B/s
opos: 33992 MB, non-scraped: 0 B, average rate: 188
kB/s
non-tried: 966212 MB, bad-sector: 0 B, error rate: 0
B/s
rescued: 33992 MB, bad areas: 0, run time: 2d 2h
6m
pct rescued: 3.39%, read errors: 0, remaining time: 7104d
20h
time since last successful read:
0s
Copying non-tried blocks... Pass 1 (forwards)^C

Should I wait hoping for a speeding? Should I pass different option to
ddrescue or use another tool?

[Charles Curley]

On Wed, 31 Aug 2022 15:25:36 +0200
ppr <p...@zaclys.net> wrote:

> Should I wait hoping for a speeding? Should I pass different option
> to ddrescue or use another tool?

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

[to...@tuxteam.de]

On Wed, Aug 31, 2022 at 03:25:36PM +0200, ppr wrote:
> I would appreciate advice from the community about a failing hard drive.
>

[...]

> I did not try to mount the HDD. I plugged an external HDD (ext4) and
> launched ddrescue. After two days it has recovered 33GB of 1TB but the speed
> are now so slow it will take 7104 days to complete.

External means an USB enclosure? Depending on the USB this might be the
bottleneck.

Cheers
--
t

[Cindy Sue Causey]

My experience is that a session's reboot freshness affects transfer
statistics, too. In addition to starting with a new reboot, I will
also rsync single directories. Doing so means the system is will be
churning away at, choking on less data while it grapples with copying
that data over to other media.

Since that method of attack leaves room for the human error of
accidentally skipping over directories, I'll run the entire setup one
last time at the end. Doing so does on occasion catch something I've
missed.

Cindy :)
--
Talking Rock, Pickens County, Georgia, USA
* runs with birdseed *

[Michael Stone]

On Wed, Aug 31, 2022 at 03:25:36PM +0200, ppr wrote:

>I did not try to mount the HDD. I plugged an external HDD (ext4) and
>launched ddrescue. After two days it has recovered 33GB of 1TB but the
>speed are now so slow it will take 7104 days to complete.

is the img file still growing? in general you're going to have issues
with error retries on external disks because the various layers don't
play well together (including the sata/usb hardware in the enclosure).
your best bet would be to try to hook the drive up internally, but if
the disk is really dead nothing is going to help.

[David Christensen]

Unless you have enterprise grade equipment designed for 100% duty cycle
for 48 hours, I would kill the ddresue job before your hardware is
destroyed.


Both the failed drive and the destination drive will be in heavy use
while you attempt to recover sectors. At 100 MB/s, transferring 1 TB
will take nearly 3 hours (!). Make sure everything has good power
supplies and good cooling. Use the best drive you have for the
destination; an SSD will expedite this process and steps that follow.


Ensure that the destination contains zeros for sectors not recovered.


Comment out the /etc/crypttab and/or /etc/fstab entries for the failed
drive. When you mount the drive, mount it read only.


The challenge is figuring out the right options and strategies for using
ddresue(1) to get as many good sectors as you can off the failing drive
before it dies completely. Fortunately or unfortunately, I have not
needed ddrescue(1) in many years; so, I would RFTM carefully and then
STFW for articles about using ddrescue(1) effectively. Consider doing
the work in chunks. You should already have sectors 0- 33 GB. Skip 33
GB and/or 34 GB. Do 35-100 GB. Then, 100-200 GB, 200-300 GB, 300-400
GB, etc.. Get the good sectors first. Do the problem sectors last.


Once you have an image file containing whatever sectors you could
recover, make the file read-only and back it up. Better yet, make two
backups and put one off-site.


To do the filesystem repair/ recovery work, make a copy of the image and
work on the copy. If you make a mistake, you can throw away the copy
and start over.


I find it very useful to install Debian onto a good quality USB 3.0
flash drive, to use for system administration, maintenance,
trouble-shooting, etc.. I prefer this approach over "live"
distributions because I have a full Debian system and can install
anything I want or need.


I find it very useful to have a spare computer for maintenance and
troubleshooting tasks.


I find it very useful to use a version control system for system
configuration files, system administration notes, etc..


I backup, archive, and image compulsively. I keep a supply of spare
parts on hand. Do not be afraid to spend money new an improved parts --
the last time I lost data when when I tried to "get by" with old and
inadequate parts.


David


https://toshiba.semicon-storage.com/us/storage/product/internal-specialty/pc/articles/dt01aca-series.html

https://linux.die.net/man/1/ddrescue

[David Wright]

On Wed 31 Aug 2022 at 15:25:36 (+0200), ppr wrote:
> Copying non-tried blocks... Pass 1 (forwards)^C
>
> Should I wait hoping for a speeding? Should I pass different option to
> ddrescue or use another tool?

I would look at two options you could try adding to your command line:

-K --skip-size could ascertain whether you've hit a really bad patch
that's holding everything up but can jump over it, or whether the
rest of the disk is just as bad.

-R --reverse will start an attempt from the end of the disk, and if
you're extremely lucky, it might copy most of the remaining 960-odd GB
of data. OTOH it might only confirm that the disk is closer to meeting
its maker than it was when you started the rescue.

Cheers,
David.

[David Wright]

I was under the impression that an internal drive was being rescued,
and the copy and mapfile were being written to the external one.

Or does the SMART information tell you something I've overlooked?

Cheers,
David.

[David Wright]

On Wed 31 Aug 2022 at 14:02:19 (-0700), David Christensen wrote:
> On 8/31/22 06:25, ppr wrote:
> > I would appreciate advice from the community about a failing hard drive.
> >
> > When booting up, the computer complained about /dev/sdb, which is
> > a ext4 HDD with data (not the computer main disk). dmesg shows
> > `AE_NOT_FOUND` and  `failed command: READ FPDMA QUEUED` messages
> > (full dmesg log at https://hastebin.com/raw/jebelileru).
> >
> > It has finally booted after trying unsuccessfully to start /dev/sdb.

> Comment out the /etc/crypttab and/or /etc/fstab entries for the failed
> drive. When you mount the drive, mount it read only.

I don't think it's wise to mount this disk at all, and certainly not
before everything that can be rescued from it has been obtained and
copied/archived.

> Consider doing the work in chunks. You
> should already have sectors 0- 33 GB. Skip 33 GB and/or 34 GB. Do
> 35-100 GB. Then, 100-200 GB, 200-300 GB, 300-400 GB, etc.. Get the
> good sectors first. Do the problem sectors last.

Agreed, though ddrescue should be able to do this more flexibly, and
automatically, with -K.

Cheers,
David.

[David Christensen]

On 8/31/22 15:35, David Wright wrote:
> On Wed 31 Aug 2022 at 14:02:19 (-0700), David Christensen wrote:
>> On 8/31/22 06:25, ppr wrote:
>>> I would appreciate advice from the community about a failing hard drive.
>>>
>>> When booting up, the computer complained about /dev/sdb, which is
>>> a ext4 HDD with data (not the computer main disk). dmesg shows
>>> `AE_NOT_FOUND` and  `failed command: READ FPDMA QUEUED` messages
>>> (full dmesg log at https://hastebin.com/raw/jebelileru).
>>>
>>> It has finally booted after trying unsuccessfully to start /dev/sdb.
>
>> Comment out the /etc/crypttab and/or /etc/fstab entries for the failed
>> drive. When you mount the drive, mount it read only.
>
> I don't think it's wise to mount this disk at all, and certainly not
> before everything that can be rescued from it has been obtained and
> copied/archived.

First sentence -- You don't want the OS to access the drive on the next
boot.


Second sentence -- I should have prefaced that with "after ddrescue has
finished".

>> Consider doing the work in chunks. You
>> should already have sectors 0- 33 GB. Skip 33 GB and/or 34 GB. Do
>> 35-100 GB. Then, 100-200 GB, 200-300 GB, 300-400 GB, etc.. Get the
>> good sectors first. Do the problem sectors last.
>
> Agreed, though ddrescue should be able to do this more flexibly, and
> automatically, with -K.

RTFM [1], I don't know if I would use -K. Take a look at the examples
given at the end of section "4 Algorithm" and in "10 A small tutorial
with examples" (examples 3 and 5 look relevant to the OP).


David


[1] https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html

[Stefan Monnier]

ppr [2022-08-31 15:25:36] wrote:
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: FAILED!
> Drive failure expected in less than 24 hours. SAVE ALL DATA.
> Failed Attributes:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
> WHEN_FAILED RAW_VALUE
> 2 Throughput_Performance 0x0005 037 037 054 Pre-fail Offline
> FAILING_NOW 3774
> 5 Reallocated_Sector_Ct 0x0033 001 001 005 Pre-fail Always
> FAILING_NOW 2004
> ---

Not looking very good, eh?

> I did not try to mount the HDD. I plugged an external HDD (ext4) and
> launched ddrescue. After two days it has recovered 33GB of 1TB but the speed
> are now so slow it will take 7104 days to complete.
>
> # ddrescue -n /dev/sdb
> /media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/image_HDD1.img
> /media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/recup.log
> GNU ddrescue 1.23
> Press Ctrl-C to interrupt
> ipos: 33992 MB, non-trimmed: 0 B, current rate: 636 B/s
> opos: 33992 MB, non-scraped: 0 B, average rate: 188 kB/s
> non-tried: 966212 MB, bad-sector: 0 B, error rate: 0 B/s
> rescued: 33992 MB, bad areas: 0, run time: 2d 2h 6m
> pct rescued: 3.39%, read errors: 0, remaining time: 7104d 20h
> time since last successful read: 0s
> Copying non-tried blocks... Pass 1 (forwards)^C

Hmm... I have no idea what's going on with your disk, but: I had a 2TB
2½" drive that failed with somewhat similar symptoms: the `dmesg` and
`smartctl` messages did not look anything like your drive, but just like
you show, `ddrescue` was able to read the drive without errors, just at
a dreadfully slow pace (tho I think it was more around 50kB/s).

While playing around with it, I ended up noticing that the performance
of `ddrescue` would sometimes briefly jump up to "several MB/s" and that
those jumps were somewhat(!?) linked to my running `lvscan`.

In the end, I ran a "while loop" that did `lvscan` repeatedly, and that
sped up `ddrescue` to an average of ~1MB/s which was sufficient for me
to copy the few partitions that actually mattered, within a tolerable
time. I also mounted a few other partitions to copy over some
specific subdirectories.

FWIW, I still have the drive, and last time I tried, it still
(mis)behaves the same. According to my web searches my problem was
probably linked to overheating which apparently can change the shape of
the platters enough that the head's positioning always fails-at-first
bringing access speed down to a crawl.

> Should I wait hoping for a speeding?

I'd prioritize the parts of the drive that matter.

> Should I pass different option to ddrescue or use another tool?

That did not seem to make any difference in my case, but obviously YMMV.


Stefan

[David Wright]

On Wed 31 Aug 2022 at 16:33:39 (-0700), David Christensen wrote:
> On 8/31/22 15:35, David Wright wrote:
> > On Wed 31 Aug 2022 at 14:02:19 (-0700), David Christensen wrote:
> > > On 8/31/22 06:25, ppr wrote:
> > > > I would appreciate advice from the community about a failing hard drive.
> > > >
> > > > When booting up, the computer complained about /dev/sdb, which is
> > > > a ext4 HDD with data (not the computer main disk). dmesg shows
> > > > `AE_NOT_FOUND` and  `failed command: READ FPDMA QUEUED` messages
> > > > (full dmesg log at https://hastebin.com/raw/jebelileru).
> > > >
> > > > It has finally booted after trying unsuccessfully to start /dev/sdb.
> >
> > > Comment out the /etc/crypttab and/or /etc/fstab entries for the failed
> > > drive. When you mount the drive, mount it read only.
> >
> > I don't think it's wise to mount this disk at all, and certainly not
> > before everything that can be rescued from it has been obtained and
> > copied/archived.
>
> First sentence -- You don't want the OS to access the drive on the
> next boot.
>
> Second sentence -- I should have prefaced that with "after ddrescue
> has finished".

Yes, I was misled by the order of paragraphs.

> > > Consider doing the work in chunks. You
> > > should already have sectors 0- 33 GB. Skip 33 GB and/or 34 GB. Do
> > > 35-100 GB. Then, 100-200 GB, 200-300 GB, 300-400 GB, etc.. Get the
> > > good sectors first. Do the problem sectors last.
> >
> > Agreed, though ddrescue should be able to do this more flexibly, and
> > automatically, with -K.
>
> RTFM [1], I don't know if I would use -K. Take a look at the examples
> given at the end of section "4 Algorithm" and in "10 A small tutorial
> with examples" (examples 3 and 5 look relevant to the OP).

I guess it would be worth setting -a --min-read-rate in addition,
as 636 bytes/sec is unsustainable. This will trigger the skipping.
But the most help could be just reversing the pass, to see whether
the "damage" is local or across the whole disk.

As for the examples, I didn't see a freeze reported (ex.3), nor any
read errors (ex.5), just a slowdown; I think ex.3 is for occasions
when reading particular sectors just locks up the disk.

Cheers,
David.

[Max Nikulin]

On 01/09/2022 06:33, David Christensen wrote:
>
> First sentence -- You don't want the OS to access the drive on the next
> boot.

I would say that while ddrescue is running any other disk operations,
especially causing seeks, may be undesired. While smartd may be
configured or stopped, udisksd is tightly integrated into desktop
environments and 10 min SMART polling interval is hardcoded for SATA
disks (however USB enclosures are ignored).
https://github.com/storaged-project/udisks/issues/407

In some cases it still might be better to monitor disk temperature.

Unfortunately I can not suggest anything concerning too low read speed,
I do not have such experience. It might be problem with particular area
on disk surface, so speed should increase for other block ranges, or it
might be firmware failure.

In the case of really precious data try to find a service with
appropriate equipment. Otherwise attempt of reading may cause so severe
damage that nobody will be able to recover any information.

[ppr]

Le 01/09/2022 à 00:24, David Wright a écrit :

> -R --reverse will start an attempt from the end of the disk, and if
> you're extremely lucky, it might copy most of the remaining 960-odd GB
> of data. OTOH it might only confirm that the disk is closer to meeting
> its maker than it was when you started the rescue.

Thank you all for yours anwsers.

Following the suggestion of Cindy I rebooted the computer to see if
session's reboot freshness affects transfer. It did not in my case.

Transfert rate is very slow but ddrescue make some progress. After 5
days and one interruption (for rebooting), here is the current status:

---

# ddrescue -n /dev/sdb
/media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/image_HDD1.img
/media/sara/2274a2da-1f02-4afd-a5c5-e8dcb1c02195/recup_HDD_sara/recup.log
GNU ddrescue 1.23
Press Ctrl-C to interrupt

Initial status (read from mapfile)
rescued: 33992 MB, tried: 0 B, bad-sector: 0 B, bad areas: 0

Current status
ipos: 72645 MB, non-trimmed: 26050 kB, current rate: 1638
B/s
opos: 72645 MB, non-scraped: 0 B, average rate: 55829
B/s
non-tried: 951012 MB, bad-sector: 0 B, error rate: 0
B/s
rescued: 49165 MB, bad areas: 0, run time: 3d 3h
29m
pct rescued: 4.91%, read errors: 490, remaining time: 5382d
14h

time since last successful read:
0s

Copying non-tried blocks... Pass 1 (forwards)

---

I am considering using -R option (as suggested by David Wright) to see
if ddrescue can have better transfert rate starting from the other side
of the HDD. My concern is whether or not ddrescue can resume a job
started, stopped, and relaunched with -R option added. ddrescue's manual
says only:

--reverse
Reverse the direction of all passes (copying, trimming, scraping,
and retrying). Every pass that is normally run forwards will now be run
backwards, and vice versa. '--reverse' does not modify the size of the
blocks copied during each phase, just the order in which they are tried.

If I stop ddrescue and add -R option, would the mapfile be able to
handle this (ddrescue started from the beginning and the end the the
drive)?

I also add some information related to other answers:

- The destination HDD is an external USB3 one. It is a physical RAID 1
with two 8To HDD 7200rpm. The commercial name is Lacie 2big raid. The
fact is image is growing and the speed seems to vary (even if never
fast) lets me think it is perhaps the HDD condition which is the cause
of the slow transfer rate.

- I commented out the failing HDD in /etc/fstab and disable
smartmontools tests for this drive. I did not try to mount the HDD.

Cheers.
ppr

[David Christensen]

I assume you are not seeing any problems with the destination drive (via
dmesg(1), /var/log/messages, etc.)?


Interpreting the ddrescue(1) output:

https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Invoking-ddrescue


At 7200 RPM, ddrescue(1) should be reading 120 blocks per second. At
512 bytes per block, that is 61,440 bytes per second. I interpret the
"current rate: 1638 B/s" and "error rate: 0 B/s" statistics to mean that
the drive read 3 blocks and returned 0 errors in the second prior to the
displayed output. So, the drive took an average of 40 internal reads
for each of those 3 blocks returned to ddrescue(1) (via the kernel).
While it is good that the drive is returning data, reading each block 40
times scares me. You want ddrescue to read and return the good blocks
once each, and then fight the bad blocks.


Looking at the command line options:

https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Invoking-ddrescue


I would interrupt (Ctrl+C) the current run and re-invoke ddrescue(1)
with the following options:

-a 30720

-d

-R

-T 10

I would not use "-K" option. I expect the GNU project carefully tuned
the initial and maximum skip size parameters. But, I do not know for
what size disk(s); yours is 1 TB. I would STFW for specific advice
before adjusting those parameters.


I would not use the "-n" option. The scrape phase is part of the design
of ddrescue; I would keep it.


The timeout option "-T 10" might stop ddrescue if it gets stuck when you
are not paying attention. If this happens, you can rethink the options
when restarting.


"--delay-slow=3", "--pause-on-error=3", and "--reset-slow=3" might be
useful for tuning "-a".


The "-i" option might be useful when restarting ddrescue. Leave it off
the first time you try "-R".


I sometimes watch disk I/O with nmon(1). But, the direct access "-d"
option might prevent nmon(1) from seeing disk I/O.


The verbose "-v" might be worth a try.


David

[David Wright]

On Sat 03 Sep 2022 at 17:46:39 (+0200), ppr wrote:
> Le 01/09/2022 à 00:24, David Wright a écrit :
>
> > -R --reverse will start an attempt from the end of the disk, and if
> > you're extremely lucky, it might copy most of the remaining 960-odd GB
> > of data. OTOH it might only confirm that the disk is closer to meeting
> > its maker than it was when you started the rescue.
>

> I am considering using -R option (as suggested by David Wright) to see
> if ddrescue can have better transfert rate starting from the other
> side of the HDD. My concern is whether or not ddrescue can resume a
> job started, stopped, and relaunched with -R option added. ddrescue's
> manual says only:
>
> --reverse
> Reverse the direction of all passes (copying, trimming, scraping,
> and retrying). Every pass that is normally run forwards will now be
> run backwards, and vice versa. '--reverse' does not modify the size of
> the blocks copied during each phase, just the order in which they are
> tried.
>
> If I stop ddrescue and add -R option, would the mapfile be able to
> handle this (ddrescue started from the beginning and the end the the
> drive)?

I don't know the answer to that. For safety, I would copy the mapfile
so far made before changing direction, just in case. After changing
direction, I would compare the backed up mapfile with the one that's
being written to see what's happening to it.

Note that the size of the output image may suddenly increase to the
total size of the disk being rescued, because ddrescue needs to start
writing to the end of it.

> I also add some information related to other answers:
>
> - The destination HDD is an external USB3 one. It is a physical RAID 1
> with two 8To HDD 7200rpm. The commercial name is Lacie 2big raid. The
> fact is image is growing and the speed seems to vary (even if never
> fast) lets me think it is perhaps the HDD condition which is the cause
> of the slow transfer rate.

So your diagnostic (image is growing) won't be usable after reversing.
But running, say, tail -f on the mapfile might show lines being
appended to it. Or repeated diffs between the mapfile and its backup
as ddrescue runs. (Your experience with ddrescue is running ahead of mine.)

> - I commented out the failing HDD in /etc/fstab and disable
> smartmontools tests for this drive. I did not try to mount the HDD.

Cheers,
David.