Orphaned Inode Problem

[Stephen P. Molnar]

I am running up to date Bookworm on my Debian platform:

Processor	AMD FX(tm)-8320 Eight-Core Processor
Memory	8026MB (5267MB used)
Machine Type	Desktop
Operating System	Debian GNU/Linux 12 (bookworm)

I have been plagued with orphaned inodes. Last night the problem cane to a head. When I reboot the computer, after an orphaned inode incident created stop, it got as far as the user login. After the return I got the Windows infamous blue screen. Restarting produced the same problem.

Fortunately, I have another SSD used to test Bookworm, before updating on the SSD that is having the problem. I can access the problem drive and am in the process of backing up files.

I ran sudo e2fsck -f/dev/sdc1 and got:

Script started on 2024-02-19 08:15:52-05:00 [TERM="xterm-256color" TTY="/dev/pts/0" COLUMNS="100" LINES="24"]
[?2004h(base) ]0;comp@AbNormal: ~ [01;32mcomp@AbNormal [00m: [01;34m~ [00m$ sudo e2fsck -f /dev/sdc1 lcaomo [K sudo e2fsck -f /dev/sdc1
[?2004l
[sudo] password for comp:
e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create<y>? yes
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdc1: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdc1: 7982363/121577472 files (0.3% non-contiguous), 421959365/486307328 blocks
[?2004h(base) ]0;comp@AbNormal: ~ [01;32mcomp@AbNormal [00m: [01;34m~ [00m$ [?2004l

Comments and suggestions will be appreciated.

Thanks in advance.

-- 
Stephen P. Molnar, Ph.D.            
https://insilicochemistry.net
(614)312-7528 (c)
Skype:  smolnar1

[to...@tuxteam.de]

This session doesn't show anything to worry about. As far as fsck
is concerned, the file system looks clean. Back up its contents as
quickly as you can and treat the disk with suspicion. There are
other candidate suspects for file system corruption (flaky power
supply, software doing silly things, kernel bugs, loose cables),
but the disk would be the pirmary.

Cheers
--
t

[Stephen P. Molnar]

Thanks for he reply. It's somewhat reassuring.

According to my logs the box had its' last major  last upgrade in 2014,
so I shouldn't be too surprised.

My backup is underweight and should be done sometime tomorrow.  I have a
2 TB HDD I'm going to use for the new install.

[to...@tuxteam.de]

On Mon, Feb 19, 2024 at 12:30:30PM -0500, Stephen P. Molnar wrote:

[...]

> Thanks for he reply. It's somewhat reassuring.
>
> According to my logs the box had its' last major  last upgrade in 2014, so I
> shouldn't be too surprised.
>
> My backup is underweight and should be done sometime tomorrow.  I have a 2
> TB HDD I'm going to use for the new install.

Fingers crossed...

Cheers
--
t

[Eike Lantzsch ZP5CGE / KY4PZ]

Just as an aside note:
The notorious red SATA cables - I threw them out long ago. The red
pigment eats up the fine copper threads, changing the impedance of the
cable and eventually making false contact before failing completely.
Of course this does not apply to NVME SSDs.

--
Eike Lantzsch KY4PZ / ZP5CGE

[Andy Smith]

Hi,

On Mon, Feb 19, 2024 at 04:12:44PM -0300, Eike Lantzsch ZP5CGE / KY4PZ wrote:
> The notorious red SATA cables - I threw them out long ago. The red
> pigment eats up the fine copper threads, changing the impedance of the
> cable and eventually making false contact before failing completely.

I've never heard of this. I did a bit of searching around and all I
can find is assertions that cable colour doesn't matter for SATA. I
can't seem to find anything about red pigment damaging the copper.
Have you got a reference so I can learn more?

Thanks,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting

[Felix Miata]

Andy Smith composed on 2024-02-20 00:48 (UTC):

> On Mon, Feb 19, 2024 at 04:12:44PM -0300, Eike Lantzsch ZP5CGE / KY4PZ wrote:

>> The notorious red SATA cables - I threw them out long ago. The red
>> pigment eats up the fine copper threads, changing the impedance of the
>> cable and eventually making false contact before failing completely.

> I've never heard of this. I did a bit of searching around and all I
> can find is assertions that cable colour doesn't matter for SATA. I
> can't seem to find anything about red pigment damaging the copper.
> Have you got a reference so I can learn more?

Don't you ever read Gene Heskett posts? I expect he'll be chiming in on this one
shortly.... While you wait, consider searching this very list's archives.
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.

Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata

[Andy Smith]

Hello,

On Mon, Feb 19, 2024 at 08:16:49PM -0500, Felix Miata wrote:
> > I've never heard of this. I did a bit of searching around and all I
> > can find is assertions that cable colour doesn't matter for SATA. I
> > can't seem to find anything about red pigment damaging the copper.
> > Have you got a reference so I can learn more?
>
> Don't you ever read Gene Heskett posts?

Ah I see:

https://lists.debian.org/debian-user/2023/06/msg00103.html

Stefan: Can you point to any evidence?

Gene: Just my own life [segue to story from 1970]

The usual story.

Yeah I skipped that thread the first time around owing to its
subject line containing "urban legends".

> consider searching this very list's archives.

Moments of my life I will never get back, and no more authoritative
sources unfortunately!

[Felix Miata]

Andy Smith composed on 2024-02-20 01:29 (UTC):

> On Mon, Feb 19, 2024 at 08:16:49PM -0500, Felix Miata wrote:

>>> I've never heard of this. I did a bit of searching around and all I
>>> can find is assertions that cable colour doesn't matter for SATA. I
>>> can't seem to find anything about red pigment damaging the copper.
>>> Have you got a reference so I can learn more?

>> Don't you ever read Gene Heskett posts?

> Ah I see:

> https://lists.debian.org/debian-user/2023/06/msg00103.html

> Stefan: Can you point to any evidence?

> Gene: Just my own life [segue to story from 1970]

> The usual story.

Gene's been around quite a while, working electronics longer than most of us have
lived, likely finished his schooling and went to work before Roosevelt died.

> Yeah I skipped that thread the first time around owing to its
> subject line containing "urban legends".

>> consider searching this very list's archives.

> Moments of my life I will never get back, and no more authoritative
> sources unfortunately!

My experience with that particular color cables matches Gene's. Cut one open, and
out comes a powdery substance instead of clean copper strands. I think most for
gen 1.0 SATA 2 decades ago, so there shouldn't be many still around bogging down
3.0 drives.

[gene heskett]

Andy, I am the source of that red cable story. Actually it is not
technically a red but a magenta that fluoresces reddish to get that
brightness. And my history with early failure of cables that used that
dye to color the insulation goes back to the 1970's when the majority of
the CB radios sold were from japan, not china. Microphone cables that
included a push to talk start failing quite rapidly, The hot red wire
was used for that about 99% of the time..Open up the plug or the
microphone, the red wire had come unsoldered or broken off, attempt to
strip it back to good wire wasn't possible. there was no good copper
left anyplace in the cable. Cut an inch of it off where there should
have been copper, grab it by the end with suture clamps and thump it
with a pencil over white copy paper and shake the copper out of it as a
reddish, face powder fine dust, the copper had been I assume made into
copper oxide. It took every good tech in the country to start returning
mike cables back to the makers as defective before they got the message
that that die was poison. That took about 9 months before we could order
replacement cable specifing that they would be returned for credit if we
found any 'hot" red in the cables they were selling us. The shortage at
the time forced them to ship whatever they had I guess. If you goto
Loews or any electrical supply where they have to sell NEC approved
cabling, you will NOT see that red on any wire on the shelf or in the
rack. Then about the time sata came out, they found a new market for
that plastic dye, and sure as heck, we had cabling problems out the yang
in about 3 years. If you have that hot red wire anyplace in you
computer, it will fail, order more cables. Tan, Black, Yellow, but not
hot red. And sleep better knowing that time bomb has gone out with the
trash.

Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
- Louis D. Brandeis

[gene heskett]

Andy, look at that CET after my name in the sig, that stands for
Certified Electronics Tachnician. We teach EE's which there are 1000's
of compared to us, what their profs didn't teach them before issuing
that sheepskin they hang on the office wall. I'm also a 1st phone, an
easy test I didn't crack a book for. And I was familiar enough, having
made a living from electronics for 20+ year including the physics of
klystrons when I saw the notice that the test was available so I walked
into the profs classroom and laid my fee for the test on his desk.
Allocated 4 hours, I handed it back to him in 45 minutes. Of 125
questions, I blacked the right box on 123. He I found had been teaching
that course for 5 years, I was the first that passed it. Registered as
NB-116. 20 some years later I checked to see how many had passed since,
they were up to NB-122. That's telling indeed.

Now I'm in my 90th year, and with a fading short term memory and a pita
on this list, but I'm still here till I miss roll call some morning.

Take care and stay well ANDY.

[Andy Smith]

Hi,

On Mon, Feb 19, 2024 at 10:06:23PM -0500, gene heskett wrote:
> Andy, look at that CET after my name in the sig, that stands for Certified
> Electronics Tachnician.

There isn't a polite way to say this really but unfortunately I am
unable to take you seriously as you've posted so many outright
incorrect assertions to this mailing list in the past.

I can list off my qualifications and experience and still be told
pretty often that I don't know what I am talking about, and sometimes
I probably don't, so let's leave it at that.

Regards,

[jeremy ardley]

On 20/2/24 08:48, Andy Smith wrote:
> Hi,
>
> On Mon, Feb 19, 2024 at 04:12:44PM -0300, Eike Lantzsch ZP5CGE / KY4PZ wrote:
>> The notorious red SATA cables - I threw them out long ago. The red
>> pigment eats up the fine copper threads, changing the impedance of the
>> cable and eventually making false contact before failing completely.
> I've never heard of this. I did a bit of searching around and all I
> can find is assertions that cable colour doesn't matter for SATA. I
> can't seem to find anything about red pigment damaging the copper.
> Have you got a reference so I can learn more?

I find it unlikely that the color of the outer sheath of a cable affects
the conductors as they have their own individual sheaths usually of a
different material to the sheath.

It's possible that some manufacturer made cables with faulty individual
insulation and their brand used a red outer sheath. In that case the
color of the sheath correlates with faulty cables but is not the cause
of the faulty cables.

[gene heskett]

Thats a good way to cap this Andy, thanks. stay well.

[David Christensen]

On 2/19/24 18:07, Felix Miata wrote:
> My experience with that particular color cables matches Gene's. Cut one open, and
> out comes a powdery substance instead of clean copper strands. I think most for
> gen 1.0 SATA 2 decades ago, so there shouldn't be many still around bogging down
> 3.0 drives.

About 10 (?) years ago, I seem to recall trouble-shooting a SATA
connection problem and coming to the conclusion that the (red) SATA
cable was the problem. I cannot recall if I had heard Gene's story at
the time. I believe I decided to cut off one end, taking a 50% chance
of getting something I could use as a break-out/ pig tail. To my
surprise, there was no copper within the cable, just brownish dust!
Unfortunately, I did not photograph the cable and it is long gone.


4 or more years ago, I was plagued with SATA III connection issues;
likely due to old SATA I and SATA II cables and mobile racks. I bought
a bunch of black SATA cables marked "6 Gbps" with locking connectors and
got rid of all of my existing cables (most of which were red). I later
retired all of my SATA I and SATA II mobile racks, moved most of my
drives internal, and bought a few SATA III mobile racks for off-site
backup drives. My SATA connection problems are finally resolved.


David

[Eike Lantzsch ZP5CGE / KY4PZ]

Experience ...
"notoriously bad" on my work bench.
Audio cables, SATA cables, even red cables of 1.5mm2 upwards. The
corrosion can be seen although it takes decades for the thicker cables
to deteriorate.
It very much depends on where the cables have been manufactured.
Never had problems with European made or US made telephone cables with
wires with red sheeths. But copper cable manufacturing has been
outsourced to Asia (and Argentina - Pirelli but those are good) many
decades ago.

[Eike Lantzsch ZP5CGE / KY4PZ]

On Dienstag, 20. Februar 2024 06:58:31 -03 Eike Lantzsch ZP5CGE / KY4PZ

If you open the sheeth of the red SATA cable, you will see that at least
three wires have no extra sheeth but are directly embedded into the red
plastic. 4 are shielded. So I guess that those wires are not affected.
There is one naked wire in the middle and two to the right and left.

[gene heskett]

How many more nearly identical story's can be teased out of this group
of old hands at this game of making moving electrons do useful things?
>
> David

[Jörg-Volker Peetz]

Hi,

did you take a look at the smartctl output?

Somewhere I read, for maintainance of an SSD all it's cells should be read from
time to time like this

sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress

where device is something like sda or nvme0n1, especially if it was switched off
for a longer period. At least, it shows the current read performance of the device.
An SSD should regularly be trimmed, if in use. This is to assist it's wear
leveling process.

What's your opinion?

Regards,
Jörg.

[Henning Follmann]

On Wed, Feb 21, 2024 at 12:00:17PM +0100, Jörg-Volker Peetz wrote:
> Hi,
>
> did you take a look at the smartctl output?
>
> Somewhere I read, for maintainance of an SSD all it's cells should be read
> from time to time like this
>
> sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress

Where did you read that? That seems like a huge waste of time.

>
> where device is something like sda or nvme0n1, especially if it was switched
> off for a longer period. At least, it shows the current read performance of
> the device.
> An SSD should regularly be trimmed, if in use. This is to assist it's wear
> leveling process.
>

If you should manually kick off trim is a hotly debated issue.
It mainly depends on the use of the drive.
In most cases however do not alter any of how the system was install by
your friendly installer.


> What's your opinion?
How much time do you have :)


-H



--
Henning Follmann | hfol...@itcfollmann.com

[Jörg-Volker Peetz]

Henning Follmann wrote on 21/02/2024 14:16:
> On Wed, Feb 21, 2024 at 12:00:17PM +0100, Jörg-Volker Peetz wrote:

<snip>

>> Somewhere I read, for maintainance of an SSD all it's cells should be read
>> from time to time like this
>>
>> sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress
>
> Where did you read that? That seems like a huge waste of time.
>

As far as I remember, the idea behind this suggestion is to help the SSD
firmware detect bad blocks or cells early on and to mask them out. Of course, a
good firmware with it's wear leveling algorithm
(https://en.wikipedia.org/wiki/Wear_leveling) should do this by itself.

Regards,
Jörg.

[David Christensen]

I prefer to run a SMART long test periodically. This should read every
cell, including those that are reserved and not visible to the OS.


AIUI So long as the SSD can maintain a supply of erased cells via
manufacturer over-provisioning, trim is not required to maintain
performance. If you have workload that does a lot of writes in a short
period of time and exhausts the manufacturer over-provisioning, leaving
free space on the SSD and trimming can be a work-around.


If you are using strong encryption, not trimming will leave crypttext on
disk that creates more work for an attacker. If you are using weak
encryption, not trimming will leave crypttext on disk that an attacker
can recover.


For imaging/ cloning, trimming will zero blocks freed by the OS and
facilitate compression of the image file.


I have a SOHO network with about two dozen disks. Running smartctl by
hand is a PITA. Running fstrim(8) by hand is easy enough. I try to do
both once a month. I need to figure out smartd(8).


David

[Gremlin]

#!/usr/bin/dash -
drives="$(lsblk|grep '^sd')"
for i in $drives;do
case $i in
sd*) sudo smartctl -a /dev/"$i" ;;
*) : ;;
esac
done

[gene heskett]

On 2/21/24 08:17, Henning Follmann wrote:
> On Wed, Feb 21, 2024 at 12:00:17PM +0100, Jörg-Volker Peetz wrote:
>> Hi,
>>
>> did you take a look at the smartctl output?
>>
>> Somewhere I read, for maintainance of an SSD all it's cells should be read
>> from time to time like this
>>
>> sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress
>
> Where did you read that? That seems like a huge waste of time.
>
>>
>> where device is something like sda or nvme0n1, especially if it was switched
>> off for a longer period. At least, it shows the current read performance of
>> the device.
>> An SSD should regularly be trimmed, if in use. This is to assist it's wear
>> leveling process.
>>
> If you should manually kick off trim is a hotly debated issue.
> It mainly depends on the use of the drive.
> In most cases however do not alter any of how the system was install by
> your friendly installer.
>
>

That actually might be a good idea, as it will force a read of
everything, which will trigger a fixit it for any cell that does read
right on the first try.

OTOH, my pi's only get powered down for maintenance, so they've got lots
of spare time to do their thing when you are not looking, And i've not
lost a pi u-sd in quite a few years. So even though the system, with all
the trash collected over a decade might amount to 10G's, they have 64G
to play with. I must be doing something right.

>> What's your opinion?
> How much time do you have :)
>
>
> -H
>
>
>

[to...@tuxteam.de]

Actually... you only have to read regularly those blocks which are
known to have stuff in them. The file system should know which those
are, that's its job.

And then, this is a backup, at least in my book, and yes, you should
do that regularly, even on spinning rust ;-)

Cheers
--
t

[Henning Follmann]

On Wed, Feb 21, 2024 at 05:15:55PM +0100, Jörg-Volker Peetz wrote:

You didn't answer where you read that. I would be interested in that. I do
not claim to be an expert on this and I would like to understand it better.

[Jörg-Volker Peetz]

Henning Follmann wrote on 22/02/2024 08:43:
<snip>

> You didn't answer where you read that. I would be interested in that. I do
> not claim to be an expert on this and I would like to understand it better.
>
> -H

Concededly, I didn't noted that down. It was a discussion like in this blog:

https://forums.linuxmint.com/viewtopic.php?t=349099

Regards,
Jörg.