Bareos with Amazon VTL issue

[Balagopal V . Pillai]

    I have a new bareos setup that uses Amazon VTL (configured with one drive and 10 slots to mimic a loader) that worked fine for the past 2 months. It looks like the tape gateway got a software update recently and backups have started failing today. I sent the existing virtual tapes to archive and started with fresh ones. Labeling gives the following error -

3307 Issuing autochanger "unload slot 2, drive 0" command.
3304 Issuing autochanger "load slot 1, drive 0" command.
3305 Autochanger "load slot 1, drive 0", status is OK.
stored/block.cc:750 Write error at 0:0 on device "tapedrive-0" (/dev/nst0). ERR=Invalid argument.
stored/block.cc:768 Write error on fd=5 at file:blk 0:0 on device "tapedrive-0" (/dev/nst0). ERR=Invalid argument.
Backspace record at EOT failed. ERR=Input/output error
3912 Failed to label Volume: ERR=backends/generic_tape_device.cc:686 ioctl MTBSR error on "tapedrive-0" (/dev/nst0). ERR=Input/output error.

When this happens, there is the following error in the log -

[st0] Write not multiple of tape block size

        Here is the output of tapeinfo -

        Product Type: Tape Drive
Vendor ID: 'IBM     '
Product ID: 'ULT3580-TD5     '
Revision: '0103'
Attached Changer API: No
SerialNumber: 'CDDF10A401'
MinBlock: 0
MaxBlock: 1048576
SCSI ID: 0
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: 0x48
Density Code: 0x0
BlockSize: 65536
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0xffffffff
DeCompType: 0xffffffff
BOP: yes
Block Position: 0
Partition 0 Remaining Kbytes: 1042431
Partition 0 Size in Kbytes: 1048576
ActivePartition: 0
EarlyWarningSize: 0


         I tried the following from bareos-sd device config and the issue persists -

    Minimum Block Size = 1048576
    Maximum Block Size = 1048576

        Tar works fine with block size of 65536 or 1048576. I was wondering if there is anything else I can try to fix this. Would bareos-sd have a debug option to print out the block size being attempted? Thanks a lot.

[Balagopal V . Pillai]

I found the fix for the issue. Since tapeinfo confirmed 65536 to be a valid block size, I placed Label Block Size = 65536 in the device section of sd and that fixed the issue. It is odd that Amazon VTL worked like a charm for two months and an update broke it. bareos-sd -f -d 100 would pump out the current block size applied for label. Hope it helps somebody using a similar setup. 

[Balagopal V . Pillai]

Here is another update. The above settings worked with labeling, but during the backups the same error popped up and broke backups. I ended up adding two more lines in SD device config to set min and max block sizes as 65536. In hindsight, I should have set it as 1 MB as per Amazon recommendation. But this works for now. Something changed with this latest tape gateway update from Amazon that enforced block sizes in virtual tapes to be multiples of 65536 (up to 1 MB) After setting those, backups have started working. I am guessing that bareos would try a range of block sizes during backup and some of them may not be multiples of default tape block sizes.  

[Andreas Rogge]

Just for the record: I think this is a Bug in Amazon VTL. It pretends to
be an LTO5 drive, but it doesn't behave like a LTO5 drive.

Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

[Balagopal V . Pillai]

Thanks Andreas. Since I set the block sizes, backups are working with no issues so far. I think Amazon VTL has started to behave like a typical LTO 5 drive with fixed block sizes. So far, it might have been very lenient with block sizes it seems. I checked through the release notes for the tape gateway software and it has no mention of this change. I guess that is part and parcel of living with a cloud VTL service.. :-)