[gentoo-user] cryptsetup close and device in use when it is not

[Dale]

Howdy,

As some may recall, I use external drives as backups.  They are
encrypted and I use cryptsetup to open and close them.  Open works fine
but close gives me trouble at times.  It doesn't always do this but it
does it more often than not.  It's getting annoying. 

Drive one.  It's a 6TB drive and when it gave this error a few minutes
ago, I restarted udev-trigger to correct it.  When I try to close it, it
says it is in use.  But restarting udev-trigger reset it so I could
close it. 

Drive two, when I tried to close it, it gives me this:


root@fireball / # cryptsetup close 8tb
Device 8tb is still in use.
root@fireball / # mount | grep 8tb
root@fireball / #


As you can tell, it is not mounted.  When I mount, I do it manually from
a fstab entry.  I just do mount /mnt/8tb and it mounts it.  I use umount
to unmount it.  Old school but it works.  Thing is, for some reason it
shows it is in use even tho it is not mounted.  Even restarting
udev-trigger didn't work this time.  I can't figure out what is doing it
exactly and think it is something besides cryptsetup having a problem. 
I'm actually thinking udev for this reason.  I do my backups while I'm
updating the OS.  When I do my updates, all my firefox profiles are
closed and I'm not downloading new files.  So that is when I do my
backups.  Last time it did this, I went to boot runlevel and restarted
udev itself.  That reset it so I could close the drive and cut it off. 
Thing is, I'd like a proper fix for this or a good sound way to reset it
without having to do all that.  Sometimes all I need is to logout and
back in after updates.  Restarting udev and such shouldn't be required.

If it matters, both drives are in identical enclosures and use the same
cable and power source.  One is a SMR and the other a CMR.  One is 6TB
and one is 8TB.  Both are encrypted the same way.  Both use eSATA, not
USB.  I used USB before and it may have been the enclosure but it ruined
drives.  I could never depend on it so I switched to eSATA. 

So, anyone else run into this and have a solution for it?  Do I have
something set up wrong or is this a bug in some package somewhere?  I've
googled and seen where others have the problem but their solutions don't
seem to work and most are very old posts. 

Need some ideas and thoughts here.

Thanks.

Dale

:-)  :-) 

P. S.  I took some meds so I hope the above makes sense.  The meds are
working well right now but I need to close the drive so I can put it
back in the safe.  :/

[Ramon Fischer]

Hello Dale,

this also happens to me sometimes and the culprit was an open process
still accessing the hard drive. Maybe you can solve it like this:

$ lsof /mnt/8tb
zsh       8390 root  cwd    DIR  253,2      4096 27787265 /mnt/8tb
$ kill 8390
$ lsof /mnt/8tb

After that, you should be able to close the drive via "cryptsetup".

-Ramon

On 14/06/2021 06:50, Dale wrote:
> root@fireball / # cryptsetup close 8tb
> Device 8tb is still in use.
> root@fireball / # mount | grep 8tb
> root@fireball / #

--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF

[Dale]

Ramon Fischer wrote:
> Hello Dale,
>
> this also happens to me sometimes and the culprit was an open process
> still accessing the hard drive. Maybe you can solve it like this:
>
>    $ lsof /mnt/8tb
>    zsh       8390 root  cwd    DIR  253,2      4096 27787265 /mnt/8tb
>    $ kill 8390
>    $ lsof /mnt/8tb
>
> After that, you should be able to close the drive via "cryptsetup".
>
> -Ramon
>
> On 14/06/2021 06:50, Dale wrote:

>> root@fireball / # cryptsetup close 8tb
>> Device 8tb is still in use.
>> root@fireball / # mount | grep 8tb
>> root@fireball / #
>

I've tried lsof before, for both mount point and device, it shows
nothing open.  It's weird. 

When this happened last night, just before I posted, I let the drive sit
there while I was doing updates.  Later on, I tried to close it again
and it closed just fine.  I hadn't done anything except let it sit
there.  While I was glad it closed, I wonder why it did it.  Did udev
finally catch up to the state of the drive?  Did some other device
update and allow it to close? 

This is weird.  Everything says it is ready to be closed but it thinks
something is open.  I'm not sure what to point too for the problem.  Yet
anyway. 

Thanks for the tip.  It was worth mentioning.

Dale

:-)  :-)

[Jack]

Is it possible it was still syncing cache out to the physical drive?  I
wonder if iotop would show any activity for that drive if that's the case?

Jack

[Dale]

I may try that next time but the light had stopped blinking for several
minutes.  Since it is a SMR drive, I always leave it running until I
can't feel the heads bumping around.  I don't think it would be that
but, it's worth a try. It may lead to something. 

Will update when it does it again. 

Thanks.

Dale

:-)  :-) 

[Remy Blank]

Dale wrote on 15/06/2021 16:21:
> Ramon Fischer wrote:
>> Hello Dale,
>>
>> this also happens to me sometimes and the culprit was an open process
>> still accessing the hard drive. Maybe you can solve it like this:
>>
>>    $ lsof /mnt/8tb
>>    zsh       8390 root  cwd    DIR  253,2      4096 27787265 /mnt/8tb
>>    $ kill 8390
>>    $ lsof /mnt/8tb
>>
>> After that, you should be able to close the drive via "cryptsetup".
>>
>> -Ramon
>>
>

> I've tried lsof before, for both mount point and device, it shows
> nothing open.  It's weird.

When this happens here, it's because I accessed the drive over NFS. The NFS server sometimes
keeps mount points active, and they don't show up in the output of lsof probably because the NFS
server is in-kernel, so there are no processes associated with it. Restarting the NFS server
allows unmounting.

-- Remy

[Ramon Fischer]

If the "umount" command seems to be hanging next time, it is most likely
due to cache writebacks. You can monitor this like so:

$ watch "grep 'Dirty\|Writeback' /proc/meminfo"

-Ramon

[Dale]

Dale wrote:

> Jack wrote:
>>
>> Is it possible it was still syncing cache out to the physical drive? 
>> I wonder if iotop would show any activity for that drive if that's the
>> case?
>>
>> Jack
>>
>>
>>
>
> I may try that next time but the light had stopped blinking for several
> minutes.  Since it is a SMR drive, I always leave it running until I
> can't feel the heads bumping around.  I don't think it would be that
> but, it's worth a try. It may lead to something. 
>
> Will update when it does it again. 
>
> Thanks.
>
> Dale
>
> :-)  :-) 
>

I had to wait until I was doing backups again to try anything.  The 6TB
drive did fine.  The 8TB drive is giving the in use error.  The drive is
unmounted but iotop didn't show anything and the light isn't blinking
either.  I'd think if it was flushing cache the light would flash and it
would not umount. 

I tried the udev-trigger trick but it didn't work.  I don't know how
long it will stay 'in use' so if anyone has ideas, think and type fast. 
If not, may have to wait until next time to try again.

Dale

:-)  :-) 

[Dale]

Some more info:


root@fireball / # lvs
  LV     VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log
Cpy%Sync Convert
  Home2  Home2  -wi-ao----
<12.74t                                                   
  swap   OS     -wi-ao---- 
12.00g                                                   
  usr    OS     -wi-ao---- 
35.00g                                                   
  var    OS     -wi-ao---- 
52.00g                                                   
  backup backup -wi-ao----
698.63g                                                   
root@fireball / # vgs
  VG     #PV #LV #SN Attr   VSize    VFree 
  Home2    2   1   0 wz--n-  <12.74t      0
  OS       1   3   0 wz--n- <124.46g <25.46g
  backup   1   1   0 wz--n-  698.63g      0
root@fireball / # pvs
  PV         VG     Fmt  Attr PSize    PFree 
  /dev/sda7  OS     lvm2 a--  <124.46g <25.46g
  /dev/sdb1  Home2  lvm2 a--    <5.46t      0
  /dev/sdc1  Home2  lvm2 a--    <7.28t      0
  /dev/sdd1  backup lvm2 a--   698.63g      0

root@fireball / # cryptsetup close 8tb
Device 8tb is still in use.
root@fireball / #

As you can see, lvm doesn't even show the device but it is still under
/dev as tho it is available.  Weird. 

I found this but at the end, the command doesn't help me.  I'm not sure
why.  It does talk about using LUKS on top of LVM causing this problem. 
Since the fix doesn't work, is this a different problem??


https://linux-blog.anracom.com/tag/device-still-in-use/


I've tried every command I can find and it still shows busy.  I even
restarted udev and lvm.  Still busy. 

Ideas?

Dale

:-)  :-) 

[Dale]

Found another tidbit of info that may shed some light on this problem. 
I'm not sure what it means tho.



root@fireball / # lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
<<<SNIP>>>
sdj                 8:144  1   7.3T  0 disk 
└─sdj1              8:145  1   7.3T  0 part 
  └─8tb           254:5    0   7.3T  0 crypt /mnt/8tb
sr0                11:0    1  1024M  0 rom  
root@fireball / #


So, something called crypt seems to have it open.  Now how do I tell it
to go away?  Hmmmm.  Then I came up with this idea:


root@fireball / # ps aux | grep crypt
root       493  0.0  0.0      0     0 ?        I<   Jun13   0:00 [cryptd]
root     11509  0.0  0.0   7728  2448 pts/2    S+   00:30   0:00 grep
--colour=auto crypt
root     23667  0.0  0.0      0     0 ?        I<   Jun20   0:00
[kcryptd_io/254:]
root     23668  0.0  0.0      0     0 ?        I<   Jun20   0:00
[kcryptd/254:5]
root     23669  0.0  0.0      0     0 ?        S    Jun20   0:00
[dmcrypt_write/2]
root@fireball / #


So kcryptd is the offender it seems since it matches MAJ:MIN info.  I
assume that is kernel related not KDE.  Can I just kill that process? 
Will it do damage if I kill it or is there a better way? 

Dale

:-)  :-) 

[Dale]

After staring at it a while, it hit me that lsblk is showing it as still
mounted, even tho I umounted already without error.  So, I just ran the
umount command again.  After that, it closed just fine.  So, it seems to
be mounted twice, not once.  I mount using this command 'mount /mnt/8tb'
which uses fstab to mount the correct UUID device to that mount point. 
Surely it only mounts once.  Always has in the past.  So why is it being
mounted twice now?  None of my scripts used for backups includes any
mounting commands.  There's also only one entry in fstab as well. 

Anyone have any idea while it gets mounted twice?  Does the cryptsetup
open command mount it in some way and then I mount it manually as well??
When I opened it just now, it didn't show anything as mounted.  So it
doesn't appear to be the open command.  What else could it be??

Ideas? 

Dale

:-)  :-) 

[Dale]

The same drive just did it again.  I had to reboot to get it to reset. 
This is Linux not windoze.  Rebooting to reset something like that is
plain ridiculous.  I also checked, this time it was mounted only once. 
When I tried to umount it again, it said it wasn't mounted.  The thing
that really gets me, nothing can tell me what it is that is using the
device.  It says it is in use but no clue what it is using it. 

To test a theory, I changed fstab to mount by label instead of UUID. 
Maybe that will change something.  I dunno.  I do know, if this doesn't
improve soon, I'm going to erase the drive and either find another
encryption tool or just not encrypt it at all.  I'm not rebooting every
time this thing wants to act ugly.  I'm used to rebooting like every 6
months or so.  Usually when the power fails. 

If anyone has any ideas, please post.  In the meantime, I'm going to try
this mounting by label idea. 

Dale

:-)  :-)

[Dale]

Dale wrote:

Here's a update.  I'm doing my weekly updates which means I close web
browsers, to conserve memory mostly.  While I'm doing updates, I update
my backups.  As usual, the 6TB drive did fine.  I ran the usual
commands, got the proper response and everything worked fine.  The 8TB
drive did the same.  It ran every command from decrypting it to
unmounting and closing without a single problem.  I even opened it,
mounted it, unmounted it and closed it again.  Still, no problems at all. 

The only thing I changed, I mounted by label instead of UUID.  Can
someone explain to me why that should matter?  It's the same thing being
mounted so one would think it wouldn't matter at all.  Heck, most even
say mounting by UUID is the best way.  Dang near impossible to have two
of anything with the same UUID.  So why does it work fine with labels
but not UUID? 

I'm looking forward to someone having a clue on this.  I'm very
confused, happy it works but confused for sure. This makes no sense. 

Dale

:-)  :-) 

[Ramon Fischer]

This is just a guess. Maybe you have two devices with the same UUID?

If so, you can change it with:

$ cryptsetup --uuid="<some_uuid>" luksUUID "/dev/sdx1"

-Ramon

[Dr Rainer Woitok]

Ramon, Dale,

On Tuesday, 2021-07-06 20:40:32 +0200, Ramon Fischer wrote:

> This is just a guess. Maybe you have two devices with the same UUID?
>
> If so, you can change it with:
>
> $ cryptsetup --uuid="<some_uuid>" luksUUID "/dev/sdx1"

Good idea. But to find out whether or not this is the cause of Dale's
problems I would suggest to first run "cryptsetup" without the "--uuid"
option (in order to get the UUID listed) and to then compare this with
the output from "ls /dev/disk/by-uuid".

Sincerely,
Rainer

[Ramon Fischer]

Interesting.

I have some other ideas, but this is really grasping at straws. Create a
backup of the backup drive before doing any tests, since you have to
move it a lot for this:

1. Connect the hard drive to a different eSATA port
2. Try another eSATA cable
3. Try to mount the hard drive on different devices
4. Try different hard drive cases with different connection types
(Maybe you have a better enclosure with USB or even FireWire, which
does not damage your drive?)
5. Connect it internally via SATA and try to mount it
6. Mirror the hard drive to a second hard drive and try to mount the
second one

I think, this would entirely cover Layer 1 of the OSI Layer Model[1]. :)

-Ramon

[1] https://en.wikipedia.org/wiki/OSI_model

On 07/07/2021 20:08, Dale wrote:

> Well, it's midweek and I wanted to test this theory even tho it is
> early.  Plus, it's raining outside so I'm a bit bored.  I pulled the
> backup drive from the safe and did a backup.  While it was backing up
> new stuff, I ran this:
>
>
> root@fireball / # blkid | grep dde669
> /dev/mapper/8tb: LABEL="8tb-backup"
> UUID="0277ff1b-2d7c-451c-ae94-f20f42dde669" BLOCK_SIZE="4096" TYPE="ext4"
> root@fireball / # ls /dev/disk/by-uuid | grep dde669
> 0277ff1b-2d7c-451c-ae94-f20f42dde669
> root@fireball / #
>
>
> I just grepped the last little bit of the UUID to see if anything else
> matched.  It didn't.  I tried both methods just in case.  It was
> grasping at straws a bit but hey, sometimes that straw solves the
> problem.  I might add, I unmounted the drive and cryptsetup closed it
> first time with not a single error.  It didn't even burp.  Given I've
> done this several times with no problem after doing the UUID way with
> consistent errors, I think it is safe to assume that changing from UUID
> to labels solves this problem.  The question now is this, why?  It's not
> like one mounts something different or anything.  It's the same device,
> just using a different link basically.
>
> This is thoroughly confusing.  It just doesn't make sense at all.
> Either way should work exactly the same.
>
> I'm open to ideas on this.  Anybody have one?  I'll test it if I can
> even if it is a serious grasp at a small straw.  ;-)

[Ramon Fischer]

OK, if it could be "udev", you might want to try to check the following:

$ grep -rF "<part_of_uuid>" /etc/udev/rules.d/
$ grep -rF "<part_of_uuid>" /lib/udev/rules.d/
$ grep -rF "<part_of_uuid>" /etc

You could also try to search for the partition device, maybe there will
be some interesting configuration files.

If you are using "systemd", you might want to check every service unit
file as well:

$ systemctl

Recently, I had a similar issue with "cryptsetup" on Raspbian, where the
"/etc/crypttab" was faulty, which may be applicable here. It had the
following entry:

# <accident_paste_with_uuid> # <target name> <source device> [...]
<entry1>
<entry2>

Therefore, the systemd service unit
"systemd-cryptsetup@dev-disk-by\x2duuid-#<accident_paste_with_uuid> #
<target name> <source device> [...]" - if I remember correctly - failed.
It seems, that "systemd-cryptsetup-generator" only searches for matching
UUIDs in "/etc/crypttab", even, if they are commented and creates
service units for each match in "/run/systemd/generator/".
I remember, that I had issues to access the hard drive. Nevertheless, I
was able to mount it normally, due to the other correct entry(?).

By removing the accidentally pasted UUID from "/etc/crypttab" and
rebooting, I was able to use the hard drive without issues again.

Maybe this is something, where you could poke around? :)

-Ramon

On 12/07/2021 10:31, Dale wrote:
> Ramon Fischer wrote:

>> Interesting.
>>
>> I have some other ideas, but this is really grasping at straws. Create
>> a backup of the backup drive before doing any tests, since you have to
>> move it a lot for this:
>>
>>    1. Connect the hard drive to a different eSATA port
>>    2. Try another eSATA cable
>>    3. Try to mount the hard drive on different devices
>>    4. Try different hard drive cases with different connection types
>>    (Maybe you have a better enclosure with USB or even FireWire, which
>>    does not damage your drive?)
>>    5. Connect it internally via SATA and try to mount it
>>    6. Mirror the hard drive to a second hard drive and try to mount the
>>    second one
>>
>> I think, this would entirely cover Layer 1 of the OSI Layer Model[1]. :)
>>
>> -Ramon
>>
>> [1] https://en.wikipedia.org/wiki/OSI_model
>>
>

> That's a lot of effort.  It's annoying it doesn't close like it should
> but doing all that would be a tedious task as well.  It would eliminate
> a lot of potential causes tho.  Thing is, I think it's a software issue
> not hardware.
>
> To add to this, the 6TB drive just did the same thing.  I had been using
> UUID to mount it since it was working.  After getting the same problem
> as the other drive, I changed it.  It took some effort to get it to
> close but restarting lvm and friends did eventually get it to close.  I
> then changed it to mount by label like I been doing with the 8TB drive.
>
> I think by just continuing to note what it is doing, eventually the
> problem will show itself.  Personally, I sort of wonder if it is a udev
> problem or lvm.  Thing is, a lot of this software works together so
> closely, it's hard to know where one stops and the other starts.
>
> I finished my backups to the 8TB drive and it worked start to finish
> with no errors at all.  I guess we'll see what happens next week with
> the 6TB drive.  See if it starts to work again with no problem or still
> has issues of some kind.  So far, mounting by label seems to have worked
> well for the 8TB drive.
>
> Will update again as things move along.

[Frank Steinmetzger]

Am Wed, Jul 07, 2021 at 01:08:55PM -0500 schrieb Dale:

> root@fireball / # blkid | grep dde669
> /dev/mapper/8tb: LABEL="8tb-backup"
> UUID="0277ff1b-2d7c-451c-ae94-f20f42dde669" BLOCK_SIZE="4096" TYPE="ext4"
> root@fireball / # ls /dev/disk/by-uuid | grep dde669
> 0277ff1b-2d7c-451c-ae94-f20f42dde669
> root@fireball / #

I followed this thread, and couldn’t remember ever having the same issue.
But today I was bitten: It’s a 3 TB external USB drive from Intenso.

Yesterday I was in the middle of a backup (it’s my main backup drive), but I
had to sleep and so sent the machine into standby. I had to start the PC
again a few minutes later in order to unmount an sshfs of it on another
machine, and sent it right back to sleep.

Just now I switched the PC back on and the drive was gone and off (USB
enclosures tend to spin down the drive when USB disconnects). So I pulled
the USB cable and plugged it back in for the drive to start and be
rediscovered. That worked and I resumed the backup, but this enclosure has
the nasty habit of sometimes intermittently disconnecting on its own.

Its device was not gone (it usually disconnects for a tiny moment and then
comes back, probably a USB issue), so I just tried to open it again in
Dolphin, which gave me:
Error unlocking /dev/sdd1: Failed to activate device: File exists

$ blkid | grep luks
/dev/mapper/luks-6a55a712-773e-4cd8-9776-fc9b6f39a998: LABEL="backup" UUID="50ed9519-cd9c-4d11-b78a-9f057b089362" BLOCK_SIZE="4096" TYPE="ext4"

$ ls -l /dev/disk/by-uuid/6a55a*
lrwxrwxrwx 10 root 2021-07-25 21:34 /dev/disk/by-uuid/6a55a712-773e-4cd8-9776-fc9b6f39a998 -> ../../sdd1

$ lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS

[…]
sdd 8:48 0 2,7T 0 disk
└─sdd1 8:49 0 2,7T 0 part

$ mount | grep -E 'luks|sdd'
[nothing]

$ cryptsetup luksClose luks-6a55a712-773e-4cd8-9776-fc9b6f39a998
Device luks-6a55a712-773e-4cd8-9776-fc9b6f39a998 is still in use.

I don’t quite like this bad habit of the enclosure, but a 3 TB drive is a 3
TB drive. I just looked at smart to see how old it is, because it has only
350 hours of power-on time, but it must be at least 5 years old. And
smartctl tells me there is a firmware update available! (for Windows, Mac
and—lo and behold—a bootable ISO, let’s hope it works with USB sticks).

Perhaps this fixes the issue. Dale, maybe you should look for the same.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Emacs is a great operating system, which only lacks a good editor.

[Dale]

That's interesting.  I have two different drives, can't recall but may
be the same brand.  While using UUID to mount it, it would either fail
every time or in the case of the smaller drive, fail on occasion but not
every time.  The smaller drive worked most of the time but after a
couple failures, I switched to mounting by label.  Since switching both
drives to mount by labels, neither has had a single issue.  My backups
last time went without a hitch.  I was actually planning to post that
after my next backup if nothing failed.  As it is, I think switching to
labels has fixed it. 

I've tried external drives connected by USB before and hated them.  Slow
when they do work and buggy at that.  I've had more drives go bad when
using USB enclosures than I've ever had on IDE or (e)SATA.  I've had two
drives fail after years of service that were IDE or SATA.  I have three
drives that are bricks and all of them were in USB enclosures and far
young to die.  I paid more for eSATA external enclosures and have had no
problems with drives going dead yet.  All of them have far surpassed the
drives in the USB enclosures.  Heck, this 'in use' problem is the first
issue I recall having.  Fast too.  The SMR drive not so much but the CMR
drive is as fast as what is in my system connected to the mobo itself. 

The thing about this, I have no idea why switching to labels works.  The
UUID, label and such are nothing but links to the real device.  It
should make no difference at all which one is used to mount with.  I'm
no guru or anything but it just shouldn't matter. 

Bad thing is, I don't use anything Microsoft here.  Can a drive's
firmware be updated on Linux?  I think my drives are either Seagate or
WD.  I tend to stick with those two, unless it is a really awesome
deal.  I've never updated the firmware on a drive before.  I have my
mobo and my router but not a drive. 

Dale

:-)  :-)

[Frank Steinmetzger]

Am Sun, Jul 25, 2021 at 06:10:19PM -0500 schrieb Dale:

> That's interesting.  I have two different drives, can't recall but may
> be the same brand.

The actual drive in my enclusure is a Seagate, BTW.

> I've tried external drives connected by USB before and hated them.  Slow
> when they do work and buggy at that.

Theoretically, HDDs are not able to saturate USB 3. And from my observation,
I do get their maximum performance – my 2.5″ 1 TB WD delivers about 80-90 MB/s
read speed and said Intenso/Seagate 3.5″ gives me up to around 140 MB/s tops.
I used dstat to gather those numbers.

> I've had more drives go bad when using USB enclosures than I've ever had
> on IDE or (e)SATA.

Interesting, I can’t really confirm such a correlation from the drives I
have lying around. And I don’t see how USB can cause damage to a drive.
Except for physical impacts owing to the fact that USB drives are easily
moved around.

> I've had two drives fail after years of service that were IDE or SATA.  I
> have three drives that are bricks and all of them were in USB enclosures
> and far young to die.

Perhaps they became too hot during operation. Enclosures don’t usually
account for thermals. Didn’t you mention you lived in a hot area?

> I paid more for eSATA external enclosures and have had no
> problems with drives going dead yet.  All of them have far surpassed the
> drives in the USB enclosures.

Hm... bad (in the sense of cheap) USB controllers on the mobo or the
enclosures? Or bad USB cables? What kind of HDD death are we talking about?
Certainly not bad sectors, right?

> Bad thing is, I don't use anything Microsoft here.  Can a drive's
> firmware be updated on Linux?

Well, that seagate update ISO didn’t work with USB and I think all my CDRW
are now serving as saucers for cups. So I downloaded the EXE and ran it on
my gaming Windows. It actually set up a temporary boot of a tiny linux which
tried the firmware update. Infortunately it didn’t detect the drive and the
text went by too fast. Might give it another try some other time.

> I think my drives are either Seagate or WD.  I tend to stick with those
> two, unless it is a really awesome deal.

Yea. First the SMR fiasco became public and then there was some other PR
stunt they did that I can’t remember right now, and I said “I can’t buy WD
anymore”. But there is no real alternative these days. And CMR drives are
becoming ever rarer, especially in the 2.5″ realm. Except for one single
seagate model, there isn’t even a bare SATA drive above 2 TB available on
the market! Everything above that size is external USB stuff. And those
disks don’t come with standard SATA connectors anymore, but have the USB
socket soldered onto their PCB.

> I've never updated the firmware on a drive before.

Me neither. I think I updated an SSD once.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Similar regulations in different post departments are TELECOMpatible.

[Dale]

Frank Steinmetzger wrote:
> Am Sun, Jul 25, 2021 at 06:10:19PM -0500 schrieb Dale:
>
>> I've tried external drives connected by USB before and hated them.  Slow
>> when they do work and buggy at that.
> Theoretically, HDDs are not able to saturate USB 3. And from my observation,
> I do get their maximum performance – my 2.5″ 1 TB WD delivers about 80-90 MB/s
> read speed and said Intenso/Seagate 3.5″ gives me up to around 140 MB/s tops.
> I used dstat to gather those numbers.

I think all my USB ports are USB2.  It's slower.  What you post above is
about what I get on my external eSATA.  If I were using USB3, things may
be different.  Maybe.  ;-)

>> I've had more drives go bad when using USB enclosures than I've ever had
>> on IDE or (e)SATA.
> Interesting, I can’t really confirm such a correlation from the drives I
> have lying around. And I don’t see how USB can cause damage to a drive.
> Except for physical impacts owing to the fact that USB drives are easily
> moved around.
>

Those particular drives sat on the desk next to my computer.  They
rarely moved.  Heck, I rarely unplugged them.  Just turn them off when
done. 

>> I've had two drives fail after years of service that were IDE or SATA.  I
>> have three drives that are bricks and all of them were in USB enclosures
>> and far young to die.
> Perhaps they became too hot during operation. Enclosures don’t usually
> account for thermals. Didn’t you mention you lived in a hot area?
>

Every enclosure I buy has a fan.  The enclosures were pretty well built
as far as the case goes.

>> I paid more for eSATA external enclosures and have had no
>> problems with drives going dead yet.  All of them have far surpassed the
>> drives in the USB enclosures.
> Hm... bad (in the sense of cheap) USB controllers on the mobo or the
> enclosures? Or bad USB cables? What kind of HDD death are we talking about?
> Certainly not bad sectors, right?
>

After a while I'd start getting errors and it would either remount ro or
just unmount completely.  After a while, the drives wouldn't respond at
all.  They spin up but it's as if they are not connected with the data
cable.  Eventually, I plugged them into my computer as SATA drives. 
They still wouldn't show up.  It was as if they were not there.  They
did spin up tho. 

>> I think my drives are either Seagate or WD.  I tend to stick with those
>> two, unless it is a really awesome deal.
> Yea. First the SMR fiasco became public and then there was some other PR
> stunt they did that I can’t remember right now, and I said “I can’t buy WD
> anymore”. But there is no real alternative these days. And CMR drives are
> becoming ever rarer, especially in the 2.5″ realm. Except for one single
> seagate model, there isn’t even a bare SATA drive above 2 TB available on
> the market! Everything above that size is external USB stuff. And those
> disks don’t come with standard SATA connectors anymore, but have the USB
> socket soldered onto their PCB.
>

I bought a SMR before I was aware of the problems with them.  It's just
a backup drive but I still have to wait for it to stop bumping before I
power it off.  Sometimes it only takes a few minutes, sometimes it bumps
for a while.  The CMR I use as a backup drive, different data, is
smaller.  It doesn't do that so I can unhook it right after it finishes. 

>> I've never updated the firmware on a drive before.
> Me neither. I think I updated an SSD once.
>

I've never had a SSD.  Thinking about it tho. 

Hmmm, just realized I didn't do my usual Sunday updates and backups. 
Ooooops.  :/

Dale

:-) :-) 

[Wols Lists]

On 26/07/21 22:00, Frank Steinmetzger wrote:
> Am Sun, Jul 25, 2021 at 06:10:19PM -0500 schrieb Dale:
>


>
>> I've had more drives go bad when using USB enclosures than I've ever had
>> on IDE or (e)SATA.
>
> Interesting, I can’t really confirm such a correlation from the drives I
> have lying around. And I don’t see how USB can cause damage to a drive.
> Except for physical impacts owing to the fact that USB drives are easily
> moved around.
>
>> I've had two drives fail after years of service that were IDE or SATA. I
>> have three drives that are bricks and all of them were in USB enclosures
>> and far young to die.

I've bought "add your own drive" USB enclosures, and ime they kill
drives. The big one killed a 3.5" drive dead, and the little one stunned
a 2.5" (as in, it no longer worked in the enclosure, I managed to revive
it ...)

I've never had any internal drives die on me (although I have rescued
other peoples' dead drives).

>
> Perhaps they became too hot during operation. Enclosures don’t usually
> account for thermals. Didn’t you mention you lived in a hot area?
>
>> I paid more for eSATA external enclosures and have had no
>> problems with drives going dead yet. All of them have far surpassed the
>> drives in the USB enclosures.

I've now bought a dual USB/SATA chassis you can hot-plug the drives
into. I haven't used that enough to form opinions on its reliability.

>


>
>> I think my drives are either Seagate or WD. I tend to stick with those
>> two, unless it is a really awesome deal.
>
> Yea. First the SMR fiasco became public and then there was some other PR
> stunt they did that I can’t remember right now, and I said “I can’t buy WD
> anymore”. But there is no real alternative these days. And CMR drives are
> becoming ever rarer, especially in the 2.5″ realm. Except for one single
> seagate model, there isn’t even a bare SATA drive above 2 TB available on
> the market! Everything above that size is external USB stuff. And those
> disks don’t come with standard SATA connectors anymore, but have the USB
> socket soldered onto their PCB.
>

Are you talking 2.5" drives here? There are plenty of 3.5" large CMR
drives. But as far as I can tell there are effectively only three drive
manufacturers left - Seagate, WD and Toshiba.

The SMR stunt was a real cock-up as far as raid was concerned - they
moved their WD Red "ideal for raid and NAS" drives over to SMR and
promptly started killing raid arrays left right and centre as people
replaced drives ... you now need Red Pro so the advice for raid is just
"Avoid WD".

From what I can make out with Seagate, the old Barracuda line is pretty
much all CMR, they had just started making some of them SMR when the
brown stuff hit the rotating blades. So now it seems pretty clear, they
renamed the SMR drives BarraCuda (note the *slight* change), and they
still make CMR drives as FireCuda. Toshiba "I know nuttin'".

Cheers,
Wol

Cheers,
Wol

[Frank Steinmetzger]

Am Thu, Jul 29, 2021 at 05:46:16PM +0100 schrieb Wols Lists:

> > Yea. First the SMR fiasco became public and then there was some other PR
> > stunt they did that I can’t remember right now, and I said “I can’t buy WD
> > anymore”. But there is no real alternative these days. And CMR drives are
> > becoming ever rarer, especially in the 2.5″ realm. Except for one single
> > seagate model, there isn’t even a bare SATA drive above 2 TB available on
> > the market! Everything above that size is external USB stuff. And those
> > disks don’t come with standard SATA connectors anymore, but have the USB
> > socket soldered onto their PCB.
> >
> Are you talking 2.5" drives here?

I meant in general, but – as I said – “especially in the 2.5″ realm”. ;-)
For 3.5″, it’s mostly the low-capacity drives that are affected. Probably
because here the ratio of fixed cost (case, electronics) vs. per-capacity
cost (platters, heads) is higher, so the pressure to reduce manufacturing
cost is also higher. High-capacity drives tend to remain CMR at the mo’.

> The SMR stunt was a real cock-up as far as raid was concerned - they
> moved their WD Red "ideal for raid and NAS" drives over to SMR and
> promptly started killing raid arrays left right and centre as people
> replaced drives ... you now need Red Pro so the advice for raid is just
> "Avoid WD".

Red Plus is fine, too. I think the “Plus” is marketing speak for non-SMR.
Which is why probably SMRs now have the price tag of old CMRs, and the new
CMRs have a “plus” on the price tag.

> From what I can make out with Seagate, the old Barracuda line is pretty
> much all CMR, they had just started making some of them SMR when the
> brown stuff hit the rotating blades.

Seagate made a statement that their NAS drives are not and never will be SMR.



In case someone is interested, here’s a little experience report:

Two days ago, I bought a 2.5″ WD My Passport 4 TB for a new off-site backup
strategy I want to implement. They even killed the rubber feet on the
underside to save a few cents. >:'-( ) Interestingly, the even cheaper
elements series (which is the cheapest because it has no complimentary
sofware and no encryption or password feature) still has them. Probably
because its case design is older.

I just finished transferring my existing Borg backup repos. Right at the
beginning, I tested a small repo of 3 GiB and I got good throughput. After
around 2 GiB or so the drive went down to 10 MiB/s for a very long time
(writing at least another 3 GiB, I have no idea what that was).

I was already pondering my options. But once that was over, I’ve since been
writing 1,2 TiB to the drive with rsync happily without any glitches,
averaging at slightly above 100 MiB/s. I used SMR-friendly ext4 settings and
Borg uses datafiles of 500 MiB size, which greatly reduces sprinkled
metadata writes b/c it’s only a few thousand files instead of millions.

According to smartctl, the drive claims to support Trim, but so far I’ve
been unsuccessful to invoke it with fstrim. First I had to enable the
allow-discard option in the underlying LUKS container, which is disabled by
default for security reasons. But either I’m still missing a detail, or the
USB-SATA-bridge really does not support it. Or it does, but the kernel is
unaware: yesterday I read an article about enabling a flag for the USB
controller via a custom UDEV rule. Who knows.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

For a pessimist, the day has 24 bruises.

[Frank Steinmetzger]

Am Thu, Jul 29, 2021 at 10:55:18PM +0200 schrieb Frank Steinmetzger:
> In case someone is interested, here’s a little experience report:

> […]

> I just finished transferring my existing Borg backup repos.

> […]

> I’ve since been writing 1,2 TiB to the drive with rsync happily without
> any glitches, averaging at slightly above 100 MiB/s.

Haha, and now I can’t unmount it (device is busy) despite of lsof reporting
no uses. So I did umount -l. That took care of the file system. But I can’t
close the LUKS container either: “device is still in use”. So we are back on
topic after all.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Even years have an expiry date.

[William Kenworthy]

I am using a seagate USB3 backup disk (4Tb SMR) for borgbackup on
btrfs.  Yes, it works well on regular backups, but its dismally slow 
for anything else (operations that read or write large amounts of data
at once):

1. Adding a lot of new data to a repo is extra slow

2. btrfs scrub (a couple of days)

3. borg repair (days!)

I had an unscheduled crash that lost a btrfs segment - scrub showed it
as an uncorrectable error so I deleted the file involved and borg repair
zeroed that part of the repo so its still ok. On a regular backup run
its fine, but if recovery time if you have an error is a real problem.

BillK

[Frank Steinmetzger]

Am Thu, Jul 29, 2021 at 11:31:48PM +0200 schrieb Frank Steinmetzger:
> Am Thu, Jul 29, 2021 at 10:55:18PM +0200 schrieb Frank Steinmetzger:
> > In case someone is interested, here’s a little experience report:
> > […]
> > I just finished transferring my existing Borg backup repos.
> > […]
> > I’ve since been writing 1,2 TiB to the drive with rsync happily without
> > any glitches, averaging at slightly above 100 MiB/s.
>
> Haha, and now I can’t unmount it (device is busy) despite of lsof reporting
> no uses. So I did umount -l. That took care of the file system. But I can’t
> close the LUKS container either: “device is still in use”. So we are back on
> topic after all.

Riddle solved:
I was about to reboot the machine and for that I closed all tmux panes. That
revealed an mc instance put into the background. It didn’t show a directory
on the drive that I couldn’t unmount, but closing the program allowed me to
close the luks container.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

You shouldn’t always wind up clocks and kids;
you also have to let them run.

[Rich Freeman]

On Fri, Jul 30, 2021 at 1:14 AM William Kenworthy <bi...@iinet.net.au> wrote:
>
> 2. btrfs scrub (a couple of days)
>

Was this a read-only scrub, or did this involve repair (such as after
losing a disk/etc)?

My understanding of SMR is that it is supposed to perform identically
to CMR for reads. If you've just recently done a bunch of writes I
could see there being some slowdown due to garbage collection (the
drive has a CMR cache which gets written out to the SMR regions), but
other than that I'd think that reads would perform normally.

Now, writes are a whole different matter and SMR is going to perform
terribly unless it is a host-managed drive (which the consumer drives
aren't), and the filesystem is SMR-aware. I'm not aware of anything
FOSS but in theory a log-based filesystem should do just fine on
host-managed SMR, or at least as well as it would do on CMR (log-based
filesystems tend to get fragmented, which is a problem on non-SSDs
unless the application isn't prone to fragmentation in the first
place, such as for logs).

Honestly I feel like the whole SMR thing is a missed opportunity,
mainly because manufacturers decided to use it as a way to save a few
bucks instead of as a new technology that can be embraced as long as
you understand its benefits and limitations. One thing I don't get is
why it is showing up on all sorts of smaller drives. I'd think the
main application would be for large drives - maybe a drive that is
14TB as CMR could have been formatted as 20TB as SMR for the same
price, and somebody could make that trade-off if it was worth it for
the application. Using it on smaller drives where are more likely to
be general-purpose is just going to cause issues for consumers who
have no idea what they're getting into, particularly since the changes
were sneaked into the product line. Somebody really needs to lose
their job over this...

--
Rich

[antlists]

On 30/07/2021 15:29, Rich Freeman wrote:
> Honestly I feel like the whole SMR thing is a missed opportunity,
> mainly because manufacturers decided to use it as a way to save a few
> bucks instead of as a new technology that can be embraced as long as
> you understand its benefits and limitations. One thing I don't get is
> why it is showing up on all sorts of smaller drives.

It's showing up on desktops because - PROVIDED THE CACHE IS LARGER THAN
THE AMOUNT OF DOCUMENTS IS ABLE TO GET THROUGH - it performs just fine.
So if you're using a pre-installed OS, and you mostly use your computer
to surf the web, watch youtube videos, etc, you're not going to stress
said cache.

Then Windows decides to download a major update and response times go
pear-shaped ...

Basically, so long as you don't fill the cache, your typical luser is
unlikely to notice. The snag is people like us are much more likely to
do things that put the cache under i/o pressure, and it's like a
motorway - once the traffic goes above the carrying capacity, throughput
COLLAPSES.

Cheers,
Wol

[Rich Freeman]

On Fri, Jul 30, 2021 at 12:50 PM antlists <antl...@youngman.org.uk> wrote:
>
> On 30/07/2021 15:29, Rich Freeman wrote:
> > Honestly I feel like the whole SMR thing is a missed opportunity,
> > mainly because manufacturers decided to use it as a way to save a few
> > bucks instead of as a new technology that can be embraced as long as
> > you understand its benefits and limitations. One thing I don't get is
> > why it is showing up on all sorts of smaller drives.
>
> It's showing up on desktops because - PROVIDED THE CACHE IS LARGER THAN
> THE AMOUNT OF DOCUMENTS IS ABLE TO GET THROUGH - it performs just fine.
> So if you're using a pre-installed OS, and you mostly use your computer
> to surf the web, watch youtube videos, etc, you're not going to stress
> said cache.

Well, in such a configuration an NVMe or even a SATA SSD would be FAR
more desirable.

Though, I guess an issue is consumers who buy for the numbers on the
specs who don't know better. 4TB is better than 0.25TB, so SMR it
is...

--
Rich

[William Kenworthy]

No, it was a normal scrub (read only is an option) - I did the scrub
hoping it wasn't necessary but aware that crash halting the OS while
doing a backup while the system was generating ooops after an upgrade
wasn't going to guarantee a clean shutdown. Ive just kicked off a scrub
-r and am getting 41Mb/s speed via the status check (its a usb3 on the
disk side, and usb2 on the PC - configuration: driver=usb-storage
maxpower=30mA speed=480Mbit/s). I will monitor for a couple of hours and
see what happens then swap to a standard scrub and compare the read rate.

rattus ~ # date && btrfs scrub status
/run/media/wdk/cae17311-19ca-4e3c-b476-304e02c50694
Sat 31 Jul 2021 10:55:43 AWST
UUID:             cae17311-19ca-4e3c-b476-304e02c50694
Scrub started:    Sat Jul 31 10:52:07 2021
Status:           running
Duration:         0:03:35
Time left:        22:30:40
ETA:              Sun Aug  1 09:26:23 2021
Total to scrub:   3.23TiB
Bytes scrubbed:   8.75GiB  (0.26%)
Rate:             41.69MiB/s

Error summary:    no errors found


lsusb: Bus 003 Device 007: ID 0bc2:331a Seagate RSS LLC Desktop HDD 5TB
(ST5000DM000)

(seagate lists it as a 5Tb drive managed SMR)

It was sold as a USB3 4Tb desktop expansion drive, fdisk -l shows "Disk
/dev/sde: 3.64 TiB, 4000787029504 bytes, 7814037167 sectors" and Seagate
is calling it 5Tb - marketing!

BillK

[Wols Lists]

On 31/07/21 04:14, William Kenworthy wrote:
> (seagate lists it as a 5Tb drive managed SMR)
>
> It was sold as a USB3 4Tb desktop expansion drive, fdisk -l shows "Disk
> /dev/sde: 3.64 TiB, 4000787029504 bytes, 7814037167 sectors" and Seagate
> is calling it 5Tb - marketing!

Note that it's now official, TB is decimal and TiB is binary, so a 4TB
drive being 3.64TiB makes sense. TB is 10e9, while TiB is 2e30.

btw, you're scrubbing over USB? Are you running a raid over USB? Bad
things are likely to happen ...

Cheers,
Wol

[William Kenworthy]

I am amused in a cynical way at disk manufacturers using decimal values ...

Its not raid, just a btrfs single on disk (no partition).  Contains a
single borgbackup repo for an offline backup of all the online
borgbackup repo's I have for a 3 times a day backup rota of individual
machines/data stores - I get an insane amount of de-duplication that way
for a slight decrease in conveniance!

BillK

[William Kenworthy]

Still almost same scrub speed and 22.5 hours after running for nearly 2
1/2 hours.

rattus ~ # btrfs scrub status
/run/media/wdk/cae17311-19ca-4e3c-b476-304e02c50694

UUID:             cae17311-19ca-4e3c-b476-304e02c50694
Scrub started:    Sat Jul 31 10:52:07 2021
Status:           running

Duration:         2:22:44
Time left:        20:04:49
ETA:              Sun Aug  1 09:19:43 2021
Total to scrub:   3.23TiB
Bytes scrubbed:   350.41GiB  (10.59%)
Rate:             41.90MiB/s

Error summary:    no errors found

rattus ~ #


Cancelled and restarted it as a normal scrub - same speed/timings - I
think if errors are found, thats when it would slow down.

rattus ~ # btrfs scrub status
/run/media/wdk/cae17311-19ca-4e3c-b476-304e02c50694
UUID:             cae17311-19ca-4e3c-b476-304e02c50694
Scrub started:    Sat Jul 31 13:18:51 2021
Status:           running
Duration:         0:00:05
Time left:        22:27:47
ETA:              Sun Aug  1 11:46:43 2021
Total to scrub:   3.23TiB
Bytes scrubbed:   209.45MiB  (0.01%)
Rate:             41.89MiB/s

Error summary:    no errors found

rattus ~ #

BillK

[Rich Freeman]

On Sat, Jul 31, 2021 at 12:58 AM William Kenworthy <bi...@iinet.net.au> wrote:
>
> I am amused in a cynical way at disk manufacturers using decimal values ...
>

So, the disk manufacturers obviously have marketing motivations.
However, IMO the programming community would be well-served to just
join basically every other profession/industry on the planet and use
the SI units. If you want to use GiB to measure things by all means
do so, but at least stick the "i" in your output.

You're not going to change ANYTHING by using SI decimal prefixes to
refer to base-2 units. Everybody on the planet who isn't a programmer
is already using SI prefixes, recognizes SI as the authoritative
standards body, and most of the governments on the planet probably
have the SI prefixes enacted as a matter of law. No court on the
planet is going to recognize even the most accomplished computer
scientists on the planet as speaking with authority on this matter.

All sticking to the old prefix meanings does is confuse people,
because when you say "GB" nobody knows what you mean.

Plus it creates other kinds of confusion. Suppose you're measuring
recording densities in KB/mm^2. Under SI prefixes 1KB/mm^2 equals
1MB/m^2, and that is why basically every engineer/scientist/etc on the
planet loves the metric system. If you're going to use base-2 units
for bytes and base-10 for meters, now you have all sorts of conversion
headaches. The base-2 system only makes sense if you never combine
bytes with any other unit. I get that programming tends to be a bit
isolated from engineering and so we like to pretend that never
happens, but in EVERY other discipline units of measure tend to be
combined all the time, and it certainly happens in engineering real
computers that don't use infinitely long tapes and only exist in CS
textbooks. :)

Just to combine replies: by "read-only" scrubbing I wasn't referring
to using "-r" but rather just that the scrub wasn't repairing
anything. A scrub operation will repair problems it finds
automatically, and that would of course take a huge hit on SMR. I'd
expect a scrub that doesn't encounter problems to perform similarly on
CMR/SMR, and something that does a ton of repair to perform terribly
on SMR.

Your numbers suggest that the SMR drive is fine for scrubbing without
errors (and if you have no mirroring/parity of data then you can't do
repairs anyway). I'm guessing the drive was just busy while
scrubbing, and obviously a busy spinning disk isn't going to scrub
very quickly (that might be tunable, but if you prioritize scrubbing
then regular IO will tank - typically you want scrubbing at idle
priority).

--
Rich

[Rich Freeman]

On Fri, Jul 30, 2021 at 11:50 PM Wols Lists <antl...@youngman.org.uk> wrote:
>
> btw, you're scrubbing over USB? Are you running a raid over USB? Bad
> things are likely to happen ...

So, USB hosts vary in quality I'm sure, but I've been running USB3
drives on lizardfs for a while now with zero issues.

At first I was shucking them and using LSI HBAs. That was a pain for
a bunch of reasons, and I would have issues probably due to the HBAs
being old or maybe cheap cable issues (and new SAS hardware carries a
hefty price tag).

Then I decided to just try running a drive on USB3 and it worked fine.
This isn't for heavy use, but it basically performs identically to
SATA. I did the math and for spinning disks you can get 2 drives per
host before the data rate starts to become a concern. This is for a
distributed filesystem and I'm just using gigabit ethernet, and the
cluster is needed more for capacity than IOPS, so USB3 isn't the
bottleneck anyway.

I have yet to have a USB drive have any sort of issue, or drop a
connection. And they're running on cheap Pi4s for the most part
(which have two USB3 hosts). If for some reason a drive or host
dropped the filesystem is redundant at the host level, and it also
gracefully recovers data if a host shows back up, but I have yet to
see that even happen due to a USB issue. I've had far more issues
when I was trying to use LSI HBAs on RockPro64 SBCs (which have a PCIe
slot - I had to also use a powered riser).

Now, if you want to do something where you're going to be pulling
closer to max bandwidth out of all your disks at once and you have
more than a few disks and you have it on 10GbE or faster, then USB3
could be a bottleneck unless you have a lot of hosts (though even then
adding USB3 hosts to the motherboard might not be any harder than
adding SATA hosts).

--
Rich

[William Kenworthy]

I'll generally agree with your USB3 comments - besides the backup disk I
am running moosefs on 5 odroid HC2's (one old WD red or green on each,
the HC2 is a 32 bit BIG.little arm system and uses a built in USB sata
connection - excellent on a 5.12 kernel, just ok on 4.x series) and an
Odroid C4 (arm64) with 2 asmedia USB3 adaptors from ebay - the adaptors
are crap but do work somewhat with the right tweaks! and a single sata
ssd on the master (intel).  I tried using moosefs with a rpi3B in the
mix and it didn't go well once I started adding data - rpi 4's were not
available when I set it up.  I think that SMR disks will work quite well
on moosefs or lizardfs - I don't see long continuous writes to one disk
but a random distribution of writes across the cluster with gaps between
on each disk (1G network).

With a good adaptor, USB3 is great ... otherwise its been quite
frustrating :(  I do suspect linux and its pedantic correctness trying
to deal with hardware that isn't truly standardised (as in the
manufacturer probably supplies a windows driver that covers it up) is
part of the problem.  These adaptors are quite common and I needed to
apply the ATA command filter and turn off UAS using the usb tweaks
mechanism to stop the crashes and data corruption.  The comments in the
kernel driver code for these adaptors are illuminating!

BillK

[Rich Freeman]

On Sat, Jul 31, 2021 at 8:59 AM William Kenworthy <bi...@iinet.net.au> wrote:
>
> I tried using moosefs with a rpi3B in the
> mix and it didn't go well once I started adding data - rpi 4's were not
> available when I set it up.

Pi2/3s only have USB2 as far as I'm aware, and they stick the ethernet
port on that USB bus besides. So, they're terrible for anything that
involves IO of any kind.

The Pi4 moves the ethernet off of USB, upgrades it to gigabit, and has
two USB3 hosts, so this is just all-around a missive improvement.
Obviously it isn't going to outclass some server-grade system with a
gazillion PCIe v4 lanes but it is very good for an SBC and the price.

I'd love server-grade ARM hardware but it is just so expensive unless
there is some source out there I'm not aware of. It is crazy that you
can't get more than 4-8GiB of RAM on an affordable arm system.

> I think that SMR disks will work quite well
> on moosefs or lizardfs - I don't see long continuous writes to one disk
> but a random distribution of writes across the cluster with gaps between
> on each disk (1G network).

So, the distributed filesystems divide all IO (including writes)
across all the drives in the cluster. When you have a number of
drives that obviously increases the total amount of IO you can handle
before the SMR drives start hitting the wall. Writing 25GB of data to
a single SMR drive will probably overrun its CMR cache, but if you
split it across 10 drives and write 2.5GB each, there is a decent
chance they'll all have room in the cache, take the write quickly, and
then as long as your writes aren't sustained they can clear the
buffer.

I think you're still going to have an issue in a rebalancing scenario
unless you're adding many drives at once so that the network becomes
rate-limiting instead of the disks. Having unreplicated data sitting
around for days or weeks due to slow replication performance is
setting yourself up for multiple failures. So, I'd still stay away
from them.

If you have 10GbE then your ability to overrun those disks goes way
up. Ditto if you're running something like Ceph which can achieve
higher performance. I'm just doing bulk storage where I care a lot
more about capacity than performance. If I were trying to run a k8s
cluster or something I'd be on Ceph on SSD or whatever.

> With a good adaptor, USB3 is great ... otherwise its been quite
> frustrating :( I do suspect linux and its pedantic correctness trying
> to deal with hardware that isn't truly standardised (as in the
> manufacturer probably supplies a windows driver that covers it up) is
> part of the problem. These adaptors are quite common and I needed to
> apply the ATA command filter and turn off UAS using the usb tweaks
> mechanism to stop the crashes and data corruption. The comments in the
> kernel driver code for these adaptors are illuminating!

Sometimes I wonder. I occasionally get errors in dmesg about
unaligned writes when using zfs. Others have seen these. The zfs
developers seem convinced that the issue isn't with zfs but it simply
is reporting the issue, or maybe it happens under loads that you're
more likely to get with zfs scrubbing (which IMO performs far worse
than with btrfs - I'm guessing it isn't optimized to scan physically
sequentially on each disk but may be doing it in a more logical order
and synchronously between mirror pairs). Sometimes I wonder if there
is just some sort of bug in the HBA drivers, or maybe the hardware on
the motherboard. Consumer PC hardware (like all PC hardware) is
basically a black box unless you have pretty sophisticated testing
equipment and knowledge, so if your SATA host is messing things up how
would you know?

--
Rich

[antlists]

On 31/07/2021 05:58, William Kenworthy wrote:
> Its not raid, just a btrfs single on disk (no partition).  Contains a
> single borgbackup repo for an offline backup of all the online
> borgbackup repo's I have for a 3 times a day backup rota of individual
> machines/data stores - I get an insane amount of de-duplication that way
> for a slight decrease in conveniance!

So - are you using btrfs's replication ability to push a backup? Does it
just push the changed blocks?

I'm planning to pull a similar stunt - I've got my eye on an 8TB Toshiba
N300, which I shall put lvm on, and then (my filesystems are all ext4)
do a snapshot and in-place rsync.

Given that my entire dataset is about 2.5TB (including films, photos etc
that don't change), again, I've got a massive amount of space to hold
backups for ages ...

Cheers,
Wol

[Frank Steinmetzger]

Am Sat, Jul 31, 2021 at 08:12:40AM -0400 schrieb Rich Freeman:

> Plus it creates other kinds of confusion. Suppose you're measuring
> recording densities in KB/mm^2. Under SI prefixes 1KB/mm^2 equals
> 1MB/m^2

*Cough* actually, 1 GB/m^2
;-)

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Rather to meditate than sit around.

[Frank Steinmetzger]

Am Sat, Jul 31, 2021 at 12:58:29PM +0800 schrieb William Kenworthy:

> Its not raid, just a btrfs single on disk (no partition).  Contains a
> single borgbackup repo for an offline backup of all the online
> borgbackup repo's I have for a 3 times a day backup rota of individual
> machines/data stores

So you are borg’ing a repo into a repo? I am planning on simply rsync’ing
the borg directory from one external HDD to another. Hopefully SMR can cope
with this adequatly.

And you are storing several machines into a single repo? The docs say this
is not supported officially. But I have one repo each for /, /home and data
for both my PC and laptop. Using a wrapper script, I create snapshots that
are named $HOSTNAME_$DATE in each repo.

> - I get an insane amount of de-duplication that way for a slight decrease
> in conveniance!

And thanks to the cache, a new snapshots usually is done very fast. But for
a yet unknown reason, sometimes Borg re-hashes all files, even though I
didn’t touch the cache. In that case it takes 2½ hours to go through my
video directory.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

“If a ship with 1000 investment bankers sinks in heavy seas, it is a
tragedy. If only one of them can swim, it is a disaster.” – Urban Priol

[Rich Freeman]

On Sat, Jul 31, 2021 at 8:41 PM Frank Steinmetzger <War...@gmx.de> wrote:
>
> Am Sat, Jul 31, 2021 at 08:12:40AM -0400 schrieb Rich Freeman:
>
> > Plus it creates other kinds of confusion. Suppose you're measuring
> > recording densities in KB/mm^2. Under SI prefixes 1KB/mm^2 equals
> > 1MB/m^2
>
> *Cough* actually, 1 GB/m^2

Ugh, this is why I always did bad on easy tests - I never check my
work. Indeed. I knew there was an 1E6 factor in there but maybe I
forgot I was already starting with 1E3...

--
Rich

[William Kenworthy]

On 31/7/21 9:30 pm, Rich Freeman wrote:
> On Sat, Jul 31, 2021 at 8:59 AM William Kenworthy <bi...@iinet.net.au> wrote:
>> I tried using moosefs with a rpi3B in the
>> mix and it didn't go well once I started adding data - rpi 4's were not
>> available when I set it up.
> Pi2/3s only have USB2 as far as I'm aware, and they stick the ethernet
> port on that USB bus besides. So, they're terrible for anything that
> involves IO of any kind.
>
> The Pi4 moves the ethernet off of USB, upgrades it to gigabit, and has
> two USB3 hosts, so this is just all-around a missive improvement.
> Obviously it isn't going to outclass some server-grade system with a
> gazillion PCIe v4 lanes but it is very good for an SBC and the price.
>
> I'd love server-grade ARM hardware but it is just so expensive unless
> there is some source out there I'm not aware of. It is crazy that you
> can't get more than 4-8GiB of RAM on an affordable arm system.

Checkout the odroid range.  Same or only slightly $$$ more for a much
better unit than a pi (except for the availability of 8G ram on the pi4)
None of the pi's I have had have come close though I do not have a pi4
and that looks from reading to be much closer in performance.  The
Odroid sites includes comparison charts of odroid aganst the rpi and it
also shows it getting closer in performance.  There are a few other
companies out there too.  I am hoping the popularity of the pi 8G will
push others to match it. I found the supplied 4.9 or 4.14 kernels
problematic with random crashes, espicially if usb was involved.  I am
currently using the 5.12 tobetter kernels and aarch64 or arm32 bit
gentoo userlands.

>
>> I think that SMR disks will work quite well
>> on moosefs or lizardfs - I don't see long continuous writes to one disk
>> but a random distribution of writes across the cluster with gaps between
>> on each disk (1G network).
> So, the distributed filesystems divide all IO (including writes)
> across all the drives in the cluster. When you have a number of
> drives that obviously increases the total amount of IO you can handle
> before the SMR drives start hitting the wall. Writing 25GB of data to
> a single SMR drive will probably overrun its CMR cache, but if you
> split it across 10 drives and write 2.5GB each, there is a decent
> chance they'll all have room in the cache, take the write quickly, and
> then as long as your writes aren't sustained they can clear the
> buffer.

Not strictly what I am seeing.  You request a file from MFS and the
first first free chunkserver with the data replies.  Writing is similar
in that (depending on the creation arguments) a chunk is written
wherever responds fastest then replicated.  Replication is done under
control of an algorithm that replicates a set number of chunks at a time
between a limited number of chunkservers in a stream depending on
replication status.  So I am seeing individual disk activity that is
busy for a few seconds, and then nothing for a short period - this
pattern has become more pronounced as I added chunkservers and would
seem to match the SMR requirements.  If I replace/rebuild (resilver) a
chunkserver, that one is a lot busier, but still not 100% continuous
write or read.  Moosefs uses a throttled replication methodology.  This
is with 7 chunkservers and 9 disks - more is definitely better for
performance.

> I think you're still going to have an issue in a rebalancing scenario
> unless you're adding many drives at once so that the network becomes
> rate-limiting instead of the disks. Having unreplicated data sitting
> around for days or weeks due to slow replication performance is
> setting yourself up for multiple failures. So, I'd still stay away
> from them.

I think at some point I am going to have to add an SMR disk and see what
happens - cant do it now though.

>
> If you have 10GbE then your ability to overrun those disks goes way
> up. Ditto if you're running something like Ceph which can achieve
> higher performance. I'm just doing bulk storage where I care a lot
> more about capacity than performance. If I were trying to run a k8s
> cluster or something I'd be on Ceph on SSD or whatever.

Tried ceph - run away fast :) I have a lot of nearly static data but
also a number of lxc instances (running on an Odroid N2) with both the
LXC instance and data stored on the cluster.  These include email,
calendaring, dns, webservers etc. all work well.  The online borgbackup
repos are also stored on it. Limitations on community moosefs is the
single point of failure that is the master plus the memory resource
requirements on the master.  I improved performance and master memory
requirements considerably by pushing the larger data sets (e.g., Gib of
mail files) into a container file stored on MFS and loop mounted onto
the mailserver lxc instance.  Convoluted but very happy with the
improvement its made.

>> With a good adaptor, USB3 is great ... otherwise its been quite
>> frustrating :( I do suspect linux and its pedantic correctness trying
>> to deal with hardware that isn't truly standardised (as in the
>> manufacturer probably supplies a windows driver that covers it up) is
>> part of the problem. These adaptors are quite common and I needed to
>> apply the ATA command filter and turn off UAS using the usb tweaks
>> mechanism to stop the crashes and data corruption. The comments in the
>> kernel driver code for these adaptors are illuminating!
> Sometimes I wonder. I occasionally get errors in dmesg about
> unaligned writes when using zfs. Others have seen these. The zfs
> developers seem convinced that the issue isn't with zfs but it simply
> is reporting the issue, or maybe it happens under loads that you're
> more likely to get with zfs scrubbing (which IMO performs far worse
> than with btrfs - I'm guessing it isn't optimized to scan physically
> sequentially on each disk but may be doing it in a more logical order
> and synchronously between mirror pairs). Sometimes I wonder if there
> is just some sort of bug in the HBA drivers, or maybe the hardware on
> the motherboard. Consumer PC hardware (like all PC hardware) is
> basically a black box unless you have pretty sophisticated testing
> equipment and knowledge, so if your SATA host is messing things up how
> would you know?
>

BillK

[William Kenworthy]

On 1/8/21 8:50 am, Frank Steinmetzger wrote:
> Am Sat, Jul 31, 2021 at 12:58:29PM +0800 schrieb William Kenworthy:
>
>> Its not raid, just a btrfs single on disk (no partition).  Contains a
>> single borgbackup repo for an offline backup of all the online
>> borgbackup repo's I have for a 3 times a day backup rota of individual
>> machines/data stores
> So you are borg’ing a repo into a repo? I am planning on simply rsync’ing
> the borg directory from one external HDD to another. Hopefully SMR can cope
> with this adequatly.
>
> And you are storing several machines into a single repo? The docs say this
> is not supported officially. But I have one repo each for /, /home and data
> for both my PC and laptop. Using a wrapper script, I create snapshots that
> are named $HOSTNAME_$DATE in each repo.

Basicly yes: I use a once per hour snapshot of approximately 500Gib of
data on moosefs, plus borgbackups 3 times a day to individual repos on
moosefs for each host.  3 times a day, the latest snapshot is stuffed
into a borg repo on moosefs and the old  snapshots are deleted.  I
currently manually push all the repos into a borg repo on the USB3 SMR
drive once a day or so.

1. rsync (and cp etc.) are dismally slow on SMR - use where you have to,
avoid otherwise.

2. borgbackup with small updates goes very quick

3. borgbackup often to keep changes between updates small - time to
backup will stay short.

4. borg'ing a repo into a repo works extreemly well - however there are
catches based around backup set names and the file change tests used.
(ping me if you want the details)

5. Yes, I have had disasters (i.e., a poorly thought out rm -rf in a
moosefs directory, unstable power that took awhile to cure, ...)
requiring underfire restoration of both large and small datasets - it works!

6. Be careful of snapshot resources on moosefs - moosefs has a defined
amount of memory for each file stored.  Even with the lazy snapshot
method, taking a snapshot will about double the memory usage on the
master for that portion of the filesystem.  Also taking too many
snapshots multiplies the effect.  Once you go into swap, it becomes a
recovery effort.  Also keep in mind that trashtime is carried into the
snapshot so the data may exist in trash even after deletion - its
actually easy to create a DOS condition by not paying attention to this.

BillK

[William Kenworthy]

On 1/8/21 11:36 am, William Kenworthy wrote:
> On 1/8/21 8:50 am, Frank Steinmetzger wrote:
>> Am Sat, Jul 31, 2021 at 12:58:29PM +0800 schrieb William Kenworthy:
>>
>>> Its not raid, just a btrfs single on disk (no partition).  Contains a
>>> single borgbackup repo for an offline backup of all the online
>>> borgbackup repo's I have for a 3 times a day backup rota of individual
>>> machines/data stores
>> So you are borg’ing a repo into a repo? I am planning on simply rsync’ing
>> the borg directory from one external HDD to another. Hopefully SMR can cope
>> with this adequatly.
>>
>> And you are storing several machines into a single repo? The docs say this
>> is not supported officially. But I have one repo each for /, /home and data
>> for both my PC and laptop. Using a wrapper script, I create snapshots that
>> are named $HOSTNAME_$DATE in each repo.
> Basicly yes: I use a once per hour snapshot of approximately 500Gib of
> data on moosefs, plus borgbackups 3 times a day to individual repos on
> moosefs for each host.  3 times a day, the latest snapshot is stuffed
> into a borg repo on moosefs and the old  snapshots are deleted.  I
> currently manually push all the repos into a borg repo on the USB3 SMR
> drive once a day or so.
>
> 1. rsync (and cp etc.) are dismally slow on SMR - use where you have to,
> avoid otherwise.
>

> forgot to mention

1a. borgbackup repos are not easily copy'able - each repo has a unique
ID and copy'ing via rsync creates a duplicate, not a new repo with a new
cache and metadata which depending on how you use can cause
corruption/data loss.  Google it.

BillK

[William Kenworthy]

On 1/8/21 8:50 am, Frank Steinmetzger wrote:

> Am Sat, Jul 31, 2021 at 12:58:29PM +0800 schrieb William Kenworthy:
>

> ...

> And thanks to the cache, a new snapshots usually is done very fast. But for
> a yet unknown reason, sometimes Borg re-hashes all files, even though I
> didn’t touch the cache. In that case it takes 2½ hours to go through my
> video directory.
>

Borg will do that as an extra method of ensuring its not missed any
changes.  I think the default is every 26 times it visits a file so its
a big hit the first time it starts but semi-randomises over time, it can
be set or disabled via an environment variable.

BillK

[Rich Freeman]

On Sat, Jul 31, 2021 at 11:05 PM William Kenworthy <bi...@iinet.net.au> wrote:
>
> On 31/7/21 9:30 pm, Rich Freeman wrote:
> >
> > I'd love server-grade ARM hardware but it is just so expensive unless
> > there is some source out there I'm not aware of. It is crazy that you
> > can't get more than 4-8GiB of RAM on an affordable arm system.
> Checkout the odroid range. Same or only slightly $$$ more for a much
> better unit than a pi (except for the availability of 8G ram on the pi4)

Oh, they have been on my short list.

I was opining about the lack of cheap hardware with >8GB of RAM, and I
don't believe ODROID offers anything like that. I'd be happy if they
just took DDR4 on top of whatever onboard RAM they had.

My SBCs for the lizardfs cluster are either Pi4s or RockPro64s. The
Pi4 addresses basically all the issues in the original Pis as far as
I'm aware, and is comparable to most of the ODroid stuff I believe (at
least for the stuff I need), and they're still really cheap. The
RockPro64 was a bit more expensive but also performs nicely - I bought
that to try playing around with LSI HBAs to get many SATA drives on
one SBC.

I'm mainly storing media so capacity matters more than speed. At the
time most existing SBCs either didn't have SATA or had something like
1-2 ports, and that means you're ending up with a lot of hosts. Sure,
it would perform better, but it costs more. Granted, at the start I
didn't want more than 1-2 drives per host anyway until I got up to
maybe 5 or so hosts just because that is where you see the cluster
perform well and have decent safety margins, but at this point if I
add capacity it will be to existing hosts.

> Tried ceph - run away fast :)

Yeah, it is complex, and most of the tools for managing it created
concerns that if something went wrong they could really mess the whole
thing up fast. The thing that pushed me away from it was reports that
it doesn't perform well only a few OSDs and I wanted something I could
pilot without buying a lot of hardware. Another issue is that at
least at the time I was looking into it they wanted OSDs to have 1GB
of RAM per 1TB of storage. That is a LOT of RAM. Aside from the fact
that RAM is expensive, it basically eliminates the ability to use
low-power cheap SBCs for all the OSDs, which is what I'm doing with
lizardfs. I don't care about the SBCs being on 24x7 when they pull a
few watts each peak, and almost nothing when idle. If I want to
attach even 4x14TB hard drives to an SBC though it would need 64GB of
RAM per the standards of Ceph at the time. Good luck finding a cheap
low-power ARM board that has 64GB of RAM - anything that even had DIMM
slots was something crazy like $1k at the time and at that point I
might as well build full PCs.

It seems like they've backed off on the memory requirements, maybe,
but I'd want to check on that. I've seen stories of bad things
happening when the OSDs don't have much RAM and you run into a
scenario like:
1. Lose disk, cluster starts to rebuild.
2. Lose another disk, cluster queues another rebuild.
3. Oh, first disk comes back, cluster queues another rebuild to
restore the first disk.
4. Replace the second failed disk, cluster queues another rebuild.

Apparently at least in the old days all the OSDs had to keep track of
all of that and they'd run out of RAM and basically melt down, unless
you went around adding more RAM to every OSD.

With LizardFS the OSDs basically do nothing at all but pipe stuff to
disk. If you want to use full-disk encryption then there is a CPU hit
for that, but that is all outside of Lizardfs and dm-crypt at least is
reasonable. (zfs on the other hand does not hardware accelerate it on
SBCs as far as I can tell and that hurts.)

> I improved performance and master memory
> requirements considerably by pushing the larger data sets (e.g., Gib of
> mail files) into a container file stored on MFS and loop mounted onto
> the mailserver lxc instance. Convoluted but very happy with the
> improvement its made.

Yeah, I've noticed as you described in the other email memory depends
on number of files, and it depends on having it all in RAM at once.
I'm using it for media storage mostly so the file count is modest. I
do use snapshots but only a few at a time so it can handle that.
While the master is running on amd64 with plenty of RAM I do have
shadow masters set up on SBCs and I do want to be able to switch over
to one if something goes wrong, so I want RAM use to be acceptable.
It really doesn't matter how much space the files take up - just now
many inodes you have.

--
Rich

[Frank Steinmetzger]

Am Sun, Aug 01, 2021 at 11:46:02AM +0800 schrieb William Kenworthy:

> >> And you are storing several machines into a single repo? The docs say this
> >> is not supported officially. But I have one repo each for /, /home and data
> >> for both my PC and laptop. Using a wrapper script, I create snapshots that
> >> are named $HOSTNAME_$DATE in each repo.
> > Basicly yes: I use a once per hour snapshot of approximately 500Gib of
> > data on moosefs, plus borgbackups 3 times a day to individual repos on
> > moosefs for each host.  3 times a day, the latest snapshot is stuffed
> > into a borg repo on moosefs and the old  snapshots are deleted.  I
> > currently manually push all the repos into a borg repo on the USB3 SMR
> > drive once a day or so.
> >
> > 1. rsync (and cp etc.) are dismally slow on SMR - use where you have to,
> > avoid otherwise.
> >
>
> > forgot to mention
>
> 1a. borgbackup repos are not easily copy'able - each repo has a unique
> ID and copy'ing via rsync creates a duplicate, not a new repo with a new
> cache and metadata which depending on how you use can cause
> corruption/data loss.  Google it.

Yup. Today I did my (not so) weekly backup and rsynced the repo to the new
drive. After that I wanted to compare performance of my old 3 TB drive and
the new SMR one by deleting a snapshot from the repo on each drive. But Borg
objected on the second deletion, because “the cache was newer”. But that’s
okay. I actually like this, as this will prevent me from chaning two repos
in parallel which would make them incompatible.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Fat stains become like new if they are regularly treated with butter.

[Frank Steinmetzger]

Ah, you’re right. I recently lowered the TTL in my wrapper script. This
might have been the trigger. I did that because I was dismayed by the size
of the cache (1 GiB), which made my root partition rather cramped. But then
I had the epiphany of simply moving the cache to the big data partition.
Problem solved (but forgot to revert the TTL).

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

What was that disease called again that makes you forget everything?

[Frank Steinmetzger]

Am Sun, Aug 01, 2021 at 11:36:36AM +0800 schrieb William Kenworthy:

> >> Its not raid, just a btrfs single on disk (no partition).  Contains a
> >> single borgbackup repo for an offline backup of all the online
> >> borgbackup repo's I have for a 3 times a day backup rota of individual
> >> machines/data stores
> > So you are borg’ing a repo into a repo? I am planning on simply rsync’ing
> > the borg directory from one external HDD to another. Hopefully SMR can cope
> > with this adequatly.
> >
> > And you are storing several machines into a single repo? The docs say this
> > is not supported officially. But I have one repo each for /, /home and data
> > for both my PC and laptop. Using a wrapper script, I create snapshots that
> > are named $HOSTNAME_$DATE in each repo.
>
> Basicly yes: I use a once per hour snapshot of approximately 500Gib of
> data on moosefs, plus borgbackups 3 times a day to individual repos on
> moosefs for each host.

So you have:
Host A ──[hourly]──> Online-Borg A ─┐
├──[3/day]──> Offline-Borg
Host B ──[hourly]──> Online-Borg B ─┘
?

> 3 times a day, the latest snapshot is stuffed into a borg repo on moosefs
> and the old  snapshots are deleted.

How do you stuff just the latest snapshot of a repo into another repo?

> 3. borgbackup often to keep changes between updates small - time to
> backup will stay short.

Apparently you are dealing with some kind of productive, high-availability
or high-throughput system. I only have my personal stuff, so I don’t mind it
taking a few minutes. That allows me to watch progress bars and numbers
going up. :)

> 4. borg'ing a repo into a repo works extreemly well - however there are
> catches based around backup set names and the file change tests used.
> (ping me if you want the details)

But what is the point of that? Why not simply keep the last x hourly/
daily/weekly snapshots? The only thing I can think of is to have a
small(ish) short-term repo and keep the larger archival repo separate.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

“If it’s true that our species is alone in the universe, then I’d have to say
the universe aimed rather low and settled for very little.” – George Carlin

[William Kenworthy]

Keep in  mind that both repos have the same ID - you should also rsync
the cache and security directories as well as they are now out of sync
(hence the warning).  Be very careful on how you do this - you are one
step away from losing the while repo if the cache gets out of sync.  The
docs warn against rsyncing two repos and then using them at the same
time for a good reason.

BillK

[William Kenworthy]

Hi Frank,

Not quite - I see I could have been clearer.  I "experiment" a lot - which means things break so I need to get back running quickly.  So the purpose of the online repos and snapshots is just for that - quick recovery.  Longer term I want to have an offline copy for a real disaster (yes, I have needed to restore almost 20TiB including the online backups on the moosefs system three months ago - self caused :) as well as often needing to step back a few days/weeks for a historical copy of a file, or less often a system restore on a host.  By the way, this discussion has been useful as I found some failed backups due to a missing dependency in the borgbackup ebuild when looking closer at it - has a bug.  Need to do some watcher scripts to detect that.

stage 1: online, immediately available

Hosts (those with actual attached storage - a mixture of intel, arm32 and arm64 devices are backed up to their own borg repo 3 times a day via push.  One repo per machine on moosefs.

A separate script does an hourly backup of VM, LXC images, and various data stores via a moosefs snapshot.

stage 2: resource management for the snapshots

3 times a day, a script does a borg create on the latest snapshop at the time, and when complete deletes all previous snapshots (-1) so at that point I have two older snapshots available + a couple created during the borg run - note that large multi GiB snapshots can quickly use up all memory (32GiB) on the moosefs master unless culled regularly.

stage 3: offline because disasters happen :)

All borg repos are on moosefs with a single root directory (/mnt/mfs/backups) so once every day or so I manually mount the offline disk and do a borg create on the backup directory.  I was doing this once a week, but operationally its easier to do 5-10 minutes every day than an hour once a week due to the scale of changes over the longer time period.

So, its looks like:

Host A    ──[3/day]──> Online-Borg A   ─┐
                                        ├─────────────────────────[common directory]---[manual, 1/day]──> Offline-Borg
Host ...  ──[3/day]──> Online-Borg ... ─┘                             |
?                                                                     |

Snapshots ──[hourly]──> Online-Borg "snaps" ─[3/day] ─────────────────┘ 

BillK

[Frank Steinmetzger]

Am Mon, Aug 02, 2021 at 01:38:31PM +0800 schrieb William Kenworthy:

> > Yup. Today I did my (not so) weekly backup and rsynced the repo to the new
> > drive. After that I wanted to compare performance of my old 3 TB drive and
> > the new SMR one by deleting a snapshot from the repo on each drive. But Borg
> > objected on the second deletion, because “the cache was newer”. But that’s
> > okay. I actually like this, as this will prevent me from chaning two repos
> > in parallel which would make them incompatible.
> >
> Keep in  mind that both repos have the same ID - you should also rsync
> the cache and security directories as well as they are now out of sync
> (hence the warning).

That thought crossed my mind recently but I was unsure how to store the
cache. But since the repo is a monolith, it should suffice to rsync
the whole cache directory to the backup drive (or do it as a tar).

The only problem is the temporal sequence:
1. Host A runs borg and gets a current cache.
2. Host B runs borg on the same repo and gets a current cache.
2a. Host A now has an outdated cache.

Usually, Host B uses Host A via ssh as remote location of the repository.
So I could simply run a borg command on Host A to update the cache somehow.

> Be very careful on how you do this - you are one step away from losing the
> while repo if the cache gets out of sync.  The docs warn against rsyncing
> two repos and then using them at the same time for a good reason.

I won’t use them at the same time. It will always be one direction:
Hosts --[borg]--> Main backup drive --[rsync]--> secondary backup drive

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

My concience is clean! After all, I’ve never used it.

[Frank Steinmetzger]

Am Mon, Aug 02, 2021 at 02:12:24PM +0800 schrieb William Kenworthy:
> >>> And you are storing several machines into a single repo? The docs say this
> >>> is not supported officially. But I have one repo each for /, /home and data
> >>> for both my PC and laptop. Using a wrapper script, I create snapshots that
> >>> are named $HOSTNAME_$DATE in each repo.
> >> Basicly yes: I use a once per hour snapshot of approximately 500Gib of
> >> data on moosefs, plus borgbackups 3 times a day to individual repos on
> >> moosefs for each host.
> > So you have:
> > Host A ──[hourly]──> Online-Borg A ─┐
> > ├──[3/day]──> Offline-Borg
> > Host B ──[hourly]──> Online-Borg B ─┘
> > ?

> > […]

>
> Hi Frank,
>
> Not quite - I see I could have been clearer.  I "experiment" a lot -
> which means things break so I need to get back running quickly.  So the
> purpose of the online repos and snapshots is just for that - quick
> recovery.

Whenever you say snapshot, you meen moosefs snapshots, right? Up until this
thread I’ve never heard of that FS.

I would love to play more with storage systems, moving stuff around, backing
it up, assigning space and so on (basically play admin for a few people),
but apart from my ZFS-based NAS, I have nothing that would need this. I run
a nextcloud instance on my shared internet host and one on my raspi. That’s
as far as it gets. :D

> stage 1: online, immediately available
>
> Hosts (those with actual attached storage - a mixture of intel, arm32
> and arm64 devices are backed up to their own borg repo 3 times a day via
> push.  One repo per machine on moosefs.
>
> A separate script does an hourly backup of VM, LXC images, and various
> data stores via a moosefs snapshot.
>
> stage 2: resource management for the snapshots
>
> 3 times a day, a script does a borg create on the latest snapshop at the
> time

So you mount the latest snapshot or access it in some other way and borg
*its* content, not the live data, right?

> and when complete deletes all previous snapshots (-1) so at that
> point I have two older snapshots available + a couple created during the
> borg run - note that large multi GiB snapshots can quickly use up all
> memory (32GiB) on the moosefs master unless culled regularly.

Sounds a bit delicate to me. If one link fails for some reason undetectedly,
you risk clog-up.

> stage 3: offline because disasters happen :)
>
> All borg repos are on moosefs with a single root directory
> (/mnt/mfs/backups) so once every day or so I manually mount the offline
> disk and do a borg create on the backup directory.

What happens if that daily borg runs while the repos are being written to?

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Even a Bonsai dreams of greatness.

[William Kenworthy]

You could delete and rebuild the cache each time (or I think there is a
way to do without it).  There are quite a few threads on the borg lists
about this in the past (usually people trying to recover trashed repos)
- you might ask there if there is a way to deal with changing the ID now?

In any case, I think doing it the way you are has a fairly high chance
you will irretrievably trash both repos.

BillK

[William Kenworthy]

Yes, though a moosefs snapshot is really a lazy copy of the data to a
new location - issue the mfsmakesnapshot command and a few seconds later
you have an identical copy of possibly terabytes of data in a new
location with almost no extra disk space needed - though moosefs needs
memory allocated to track the contents. i.e. think of it like
symlink/hardlink to the original data until a file is changed whereupon
its links are broken and its new data - its a little more complex than
that but that's the gist of it.  If you need data from a snapshot - you
just copy it out or use it in place which breaks the link if written to.

>
>> and when complete deletes all previous snapshots (-1) so at that
>> point I have two older snapshots available + a couple created during the
>> borg run - note that large multi GiB snapshots can quickly use up all
>> memory (32GiB) on the moosefs master unless culled regularly.
> Sounds a bit delicate to me. If one link fails for some reason undetectedly,
> you risk clog-up.

Problems so far relate to borg failing to run for some reason - notice
it and fix it, no problems overall as the rest keeps working

>
>> stage 3: offline because disasters happen :)
>>
>> All borg repos are on moosefs with a single root directory
>> (/mnt/mfs/backups) so once every day or so I manually mount the offline
>> disk and do a borg create on the backup directory.
> What happens if that daily borg runs while the repos are being written to?

To avoid this I mostly use fcron which has serialisation features so as
long as I coordinate start and take into account run times its good. 
The manual copy is of course a bit tricky but again, its a timing
issue.  If I make a mistake, I would expect to include a repo that might
need a check/repair before use.  To borg, its just backing up files - it
doesn't care that its another borg repo, in use or not.  It still treats
open files the same way as any other files - try and read them, but skip
if unable to. Two borg instance cant work in the same repo, but one can
and one can back that repo up at the same time because it just sees it
as files if that makes sense.  In reality, I have not needed to deal
with it yet.  The great thing about borg and the way this rota is
structured is that I have a history I can go back to if necessary.  In
my experimenting in getting this right, I pay attention to the warnings
borg issues and often delete the cache and security directories to make
sure everything stays sane.

Getting way off-topic here so we can take this off list if you are
interested, or maybe others are interested here?

BillK


BillK

[Frank Steinmetzger]

Am Tue, Aug 03, 2021 at 07:10:03AM +0800 schrieb William Kenworthy:

> >> Keep in  mind that both repos have the same ID - you should also rsync
> >> the cache and security directories as well as they are now out of sync
> >> (hence the warning).
> > That thought crossed my mind recently but I was unsure how to store the
> > cache. But since the repo is a monolith, it should suffice to rsync
> > the whole cache directory to the backup drive (or do it as a tar).
> >
> > The only problem is the temporal sequence:
> > 1. Host A runs borg and gets a current cache.
> > 2. Host B runs borg on the same repo and gets a current cache.
> > 2a. Host A now has an outdated cache.
> >
> > Usually, Host B uses Host A via ssh as remote location of the repository.
> > So I could simply run a borg command on Host A to update the cache somehow.
> >
> >> Be very careful on how you do this - you are one step away from losing the
> >> while repo if the cache gets out of sync.  The docs warn against rsyncing
> >> two repos and then using them at the same time for a good reason.
> > I won’t use them at the same time. It will always be one direction:
> > Hosts --[borg]--> Main backup drive --[rsync]--> secondary backup drive
> >
> You could delete and rebuild the cache each time (or I think there is a
> way to do without it).

If the cache can be easily rebuilt, then there’d be no need to store it at
all. At least that’s what I hoped, but was shown otherwise; I deleted the
whole cache, wanting to clean it up from cruft, and then the next run took
hours due to complete re-hash.

> There are quite a few threads on the borg lists about this in the past
> (usually people trying to recover trashed repos)

I’ll give them a read in a quiet hour.

> - you might ask there if there is a way to deal with changing the ID now?

Why would I need to change the ID? As already explained, I will only ever
borg to the primary backup disk and mirror that to another disk with rsync.
And when the primary fails, I use the secondary as drop-in replacement.

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Sometimes the fingers are faster then grammar.

[Frank Steinmetzger]

Am Tue, Aug 03, 2021 at 10:18:06AM +0200 schrieb Frank Steinmetzger:

> > You could delete and rebuild the cache each time (or I think there is a
> > way to do without it).
>
> If the cache can be easily rebuilt, then there’d be no need to store it at
> all.

Here’s an afterthought that just hit me:
there should actually be no point in archiving the cache at all. If you had
a disaster and do a full restore from borg, the old cache data becomes
invalid anyways, because the files’ inodes will now be different. AFAIK,
inodes are one way of detecting file changes. Different inode → file must be
different → rehash.

(Unless `borg extract` updates the borg cache for files it restores, which I
doubt because the destination path is arbitrary.)

--
Grüße | Greetings | Qapla’
Please do not share anything from, with or about me on any social network.

Normal people believe that if it ain't broke, don't fix it. Engineers
believe that if it ain't broke, it doesn't have enough features yet.

[William Kenworthy]

On 6/8/21 4:40 am, Frank Steinmetzger wrote:
> Am Tue, Aug 03, 2021 at 10:18:06AM +0200 schrieb Frank Steinmetzger:
>
>>> You could delete and rebuild the cache each time (or I think there is a
>>> way to do without it).
>> If the cache can be easily rebuilt, then there’d be no need to store it at
>> all.
> Here’s an afterthought that just hit me:
> there should actually be no point in archiving the cache at all. If you had
> a disaster and do a full restore from borg, the old cache data becomes
> invalid anyways, because the files’ inodes will now be different. AFAIK,
> inodes are one way of detecting file changes. Different inode → file must be
> different → rehash.
>
> (Unless `borg extract` updates the borg cache for files it restores, which I
> doubt because the destination path is arbitrary.)
>

Agreed - I do get a warning on restore and my first choice is always
delete the cache AND the security directory - I should just go ahead and
do it anyway I guess.

Also, it would be a good time to read the borg create statement
(https://borgbackup.readthedocs.io/en/stable/usage/create.html) for the
file change detection parameters - moosefs and snapshots required
non-default options to get it right.

BillK

[Ramon Fischer]

Hi Dale,

> So, something says it is busy but eventually
> releases it if left alone for a while.  I'd like to know what it is and
> if it is really in use or not.  Thing is, I can't find a way to know
> what it is that is using it.  The dmsetup command shows it is in use but
> no way to know what is using it.
I could reproduce this issue by killing my desktop process, unmounting
the home partition and playing some "kill process" bingo. I could
backtrace it to one unkillable process "kcryptd":

1. Kill "awesomewm": <CTRL + ALT> + Backspace
2. Kill other processes accessing "/home/"
3. umount /home
4. cryptsetup close crypthome
Device crypthome is still in use
5. dmsetup info /dev/mapper/crypthome
Name:              crypthome
State:             ACTIVE
Read Ahead:        256
Tables present:    LIVE
Open count:        1
Event number:      0
Major, minor:      253, 1
Number of targets: 1
UUID: CRYPT-LUKS2-<some_uuid>-crypthome
6. Kill any unnecessary process and try "cryptsetup close crypthome"
7. Search for major, minor: ps aux | grep "253:1"
root       150  0.2  0.0      0     0 ?        I    15:21   0:02
[kworker/u16:5-kcryptd/253:1]
8. Does not work: kill 150
9. Does not work and could be dangerous: kill -9 150

So, there was still one "kcryptd" process left, accessing the hard
drive, but I found no way to kill it.

Maybe this could be helpful?

-Ramon


On 02/08/2021 15:33, Dale wrote:
> Ramon Fischer wrote:
>> OK, if it could be "udev", you might want to try to check the following:
>>
>>    $ grep -rF "<part_of_uuid>" /etc/udev/rules.d/
>>    $ grep -rF "<part_of_uuid>" /lib/udev/rules.d/
>>    $ grep -rF "<part_of_uuid>" /etc
>>
>> You could also try to search for the partition device, maybe there
>> will be some interesting configuration files.
>>
>> If you are using "systemd", you might want to check every service unit
>> file as well:
>>
>>    $ systemctl
>>
>> Recently, I had a similar issue with "cryptsetup" on Raspbian, where
>> the "/etc/crypttab" was faulty, which may be applicable here. It had
>> the following entry:
>>
>>    # <accident_paste_with_uuid> # <target name> <source device> [...]
>>    <entry1>
>>    <entry2>
>>
>> Therefore, the systemd service unit
>> "systemd-cryptsetup@dev-disk-by\x2duuid-#<accident_paste_with_uuid> #
>> <target name> <source device> [...]" - if I remember correctly - failed.
>> It seems, that "systemd-cryptsetup-generator" only searches for
>> matching UUIDs in "/etc/crypttab", even, if they are commented and
>> creates service units for each match in "/run/systemd/generator/".
>> I remember, that I had issues to access the hard drive. Nevertheless,
>> I was able to mount it normally, due to the other correct entry(?).
>>
>> By removing the accidentally pasted UUID from "/etc/crypttab" and
>> rebooting, I was able to use the hard drive without issues again.
>>
>> Maybe this is something, where you could poke around? :)
>>
>> -Ramon
> I'm running openrc here.  I don't recall making any udev rules
> recently.  This is a list of what I have.
>
>
> root@fireball / # ls -al /etc/udev/rules.d/
> total 20
> drwxr-xr-x 2 root root 4096 Apr 27 15:07 .
> drwxr-xr-x 3 root root 4096 Jul 27 03:17 ..
> -rw-r--r-- 1 root root 2064 Apr 27 15:07 69-libmtp.rules
> -rw-r--r-- 1 root root 1903 Apr  4  2012 70-persistent-cd.rules
> -rw-r--r-- 1 root root  814 Jan  1  2008 70-persistent-net.rules
> -rw-r--r-- 1 root root    0 Mar 22  2015 80-net-name-slot.rules
> root@fireball / #
>
>
> One is for CD/DVD stuff.  I wonder if I can remove that now.  Two is for
> network cards and top one is something to do with my old Motorola cell
> phone, rest in peace.
>
> All this said, it did it again last night.  I tried a few things and
> went to bed while my updates were compiling.  When I got up a bit ago,
> it closed just fine.  So, something says it is busy but eventually
> releases it if left alone for a while.  I'd like to know what it is and
> if it is really in use or not.  Thing is, I can't find a way to know
> what it is that is using it.  The dmsetup command shows it is in use but
> no way to know what is using it.
>
> Dale
>
> :-)  :-)

--
GPG public key: 5983 98DA 5F4D A464 38FD CF87 155B E264 13E6 99BF