Cannot mount LTO5 Tapes

[Alexandre Denault]

Hi,

For the last few weeks now, I've been unable to mount any of my LTO-5 drive. I always get the following error

stored/block.cc:1004 Read error on fd=6 at file:blk 0:0 on device "TapeDrive0" (/dev/nst0). ERR=Input/output error.

At first, I suspected the drive was at fault. However, xTalk did not detect any problem with the drive itself.

Test Summary
        Drive Detection Test -------- : Passed
        SCSI Interconnect Test ------ : Passed
        Tape Load Test -------------- : Passed
        Quick Write/Read Test ------- : Passed
        Small Buffer Wr/Rd Test ----- : Passed
        System Level Test ----------- : Passed
        Time Test ------------------- : Passed
        Multi-Pattern Test ---------- : Passed
        LTO Error Rate Test  -------- : Passed
No Problem Detected
The tests did not detect any problems.

The test suite seems quite complete, including a read of 50Gb of data and a validation read.

I also tried with a different LTO5 drive, but got the same result. 

I formatted one my tape and re-labeled it. I was able to mount that tape perfectly and complete some backups on it with no problems. However, all my old backups on tapes remain inaccessible. Right now, it looks like I will eventually have to reformat and relabel all my tapes to use them with Bareos. I have over 50 tapes with 2 years of backups.

This problem started around the time I updated to Bareos 23, although I can't confirm that this is the problem. I am also unable to test downgrading to Bareos 22 as I can't find any package for that.

My drive is a a Quantum Ultrium 5 SAS (internal) and I use a mix of Fujifilm and Quantum LTO 5 tapes. 

Director:  23.0.4~pre61.010c81fdc (03 July 2024) Debian GNU/Linux 11 (bullseye)
Storage (tape): 23.0.4~pre74.8cb0a0c26 (10 July 2024) Debian GNU/Linux 11 (bullseye)
Any suggestions to what I should tried to debug this problem? I would rather not loose 2 years of backups if I can.

Thanks,

 

[Alexandre Denault]

Hi,

I've also noticed the following in dmesg:

[st0] Incorrect block size.

Could it be that Bareos is not reading the correct blocksize for the header of the tape?

Cheers,

[Alexandre Denault]

Hi,

So it took a quick bit of debugging, but I discovered that Debian had changed my default block size to 512. Issuing

mt -f /dev/nst0 defblksize 0

brought the drive back to variable block size, which fixed the problem.

I can always add in more information if someone is curious.

Cheers,

[Bruno Friedmann (bruno-at-bareos)]

In breaking changes 23.0.0 release note you will notice some information about the blocksize

https://docs.bareos.org/Appendix/ReleaseNotes.html#breaking-changes

;-)

Maybe your drive or scsi adapter in not able to run 1MB

[Alan Brown]

On 14/07/2024 17:13, Alexandre Denault wrote:

Hi,

So it took a quick bit of debugging, but I discovered that Debian had changed my default block size to 512. Issuing

WTF? 

Tape drives with fixed block sizes haven't been sold for more than 25 years, if not 30!

This needs a bug report filed upstream

[Alan Brown]

On 15/07/2024 08:20, Bruno Friedmann (bruno-at-bareos) wrote:

> Maybe your drive or scsi adapter in not able to run 1MB

_ALL_ LTO drives (even LTO-1) can handle block sizes of at least 1MB and
you would need to be running a pretty awful (5-10MB/s) scsi adaptor for
a restriction there

In any case, the only way changing block size will affect readability is
if you do so between jobs on the same tape


The change of default block size in the OS/distro is a serious issue and
needs to be investigated. No tape hardware sold in the last 2 decades
uses fixed block sizes let alone 512 byte ones, so there's no reason for
this to suddenly crop up in Debian

[Andreas Rogge]

Am 15.07.24 um 20:56 schrieb Alan Brown:

>
> _ALL_ LTO drives (even LTO-1) can handle block sizes of at least 1MB and
> you would need to be running a pretty awful (5-10MB/s) scsi adaptor for
> a restriction there

You wish!

After changing the default block size to 1 MB we have seen more than one
large enterprise setup that could not handle 1 MB blocks.
The drives were fine, the controllers were fine, but some intermediate
component (*cough* FC director *cough*) didn't allow the large SCSI
frames to pass.

Also, there are people attaching their drives to the external
SAS-interface of (rather cheap) RAID controllers. Some of these don't
allow block sizes of 1 MB either.

Best Regards,
Andreas

--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

[Alexandre Denault]

Hi,

Ii it possible that OS was always defaulting to 512 and Bareos before 23.0 was just ignoring the setting. I have a hard time explaining what happened, other than "mt -f /dev/nst0 defblksize 0" fixed the problem. 

I check my Bareos configuration for the last 2 years ( I keep it versioned in git ) and I've never set a default block size, so I know I didn't change anything in Bareos.

I'm running Debian Bullseye (11). 

My SAS controller is a Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2, most likely a Dell Perc H200 controller that I flashed to IT mode.

The tape drive is a Quantum Ultrium 5 SAS Internal Half Height.

Hopefully this helps,

Alex

[markus....@greenpocket.de]

Hi,

thanks a lot for this post. We had exactly the same problem today and were able to solve it with "mt -f /dev/nst0 defblksize 0" too.

Interestingly, writing to tape worked for a few days after upgrading to Bareos 23 (including kernel updates and server restart). But this weekend we had a planned power outage and had to power down the entire infrastructure including the tape library. After bringing everything up again, the drive was not accessible - just as you described.

Our hardware seems to be nearly identical to yours: LSI SAS2008 Fusion SAS-Controller (Perc H200), Quantum Ultrium 5 SAS drive in a Quantum Superloader 3. We are using Ubuntu 22.04 though.

The next time I'm in the office, I will test whether this command will permanently resolve the problem or whether it will reappear after restarting the tape library.

Cheers,

Markus