Bareos + Amazon AWS Storage Gateway VTL - Unable to pass btape test - write succeeds, read fails

[Leon Ingleright]

So I'm currently setting up Bareos with Amazon AWS Storage Gateway Virtual Tape Library (VTL). There are very few guides out there for doing this, so I'm assuming it hasn't been tried much yet. I have managed to get over quite a few obstacles, but I have hit a major roadblock for which I can't seem to find any information.

The setup so far:

I have installed Bareos director, storage daemon, and file daemon services on an Amazon AWS EC2 instance running Ubuntu 14.04.3 LTS. I have attached the VTL to this machine using open-iscsi, which mounted the VTL as a number of SCSI devices, namely one auto-changer and 10 tape drives. Using udev rules, I was able to provide static names for each device (as generic and standard /dev names tend to change/scramble on reboot) and set them up in the Bareos storage daemon configuration.

Output of # lsscsi -g >

[2:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st3 /dev/sg7
[3:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st0 /dev/sg0
[4:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st1 /dev/sg10
[5:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st2 /dev/sg1
[6:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st4 /dev/sg9
[7:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st9 /dev/sg8
[8:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st7 /dev/sg4
[9:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st8 /dev/sg6
[10:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st5 /dev/sg2
[11:0:0:0] tape IBM ULT3580-TD5 0103 /dev/st6 /dev/sg3
[12:0:0:0] mediumx AWS Gateway-VTL 0100 /dev/sch0 /dev/sg5

Output of # ls /dev/tape/by-path >

ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-02-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-02-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-03-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-03-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-04-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-04-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-05-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-05-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-06-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-06-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-07-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-07-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-08-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-08-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-09-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-09-lun-0-nst
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-10-lun-0
ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-10-lun-0-nst

On recommendation from some other articles I read, I used the 'nst' names when creating each device resource to prevent a corruption problem described when using a tape drive that auto-rewinds. I set up the auto-changer and device directives as follows: >

Autochanger {
Name = I2M-AWS-TDC
Device = I2M-AWS-TD0
Device = I2M-AWS-TD1
Device = I2M-AWS-TD2
Device = I2M-AWS-TD3
Device = I2M-AWS-TD4
Device = I2M-AWS-TD5
Device = I2M-AWS-TD6
Device = I2M-AWS-TD7
Device = I2M-AWS-TD8
Device = I2M-AWS-TD9
Changer Command = "/usr/lib/bareos/scripts/mtx-changer %c %o %S %a %d"
Changer Device = "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0"
}

Device {
Name = I2M-AWS-TD0
Archive Device = "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst"
Device Type = Tape
Media Type = ULTRIUM5
Autochanger = yes
Drive Index = 0
Check Labels = yes
Automatic Mount = yes
Label Type = IBM
Label Media = yes
Maximum Concurrent Jobs = 100
Maximum Open Wait = 3600
Always Open = yes
}

... (TD1-TD8 here; identical to above) ...

Device {
Name = I2M-AWS-TD9
Archive Device = "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-10-lun-0-nst"
Device Type = Tape
Media Type = ULTRIUM5
Autochanger = yes
Drive Index = 9
Check Labels = yes
Automatic Mount = yes
Label Type = IBM
Label Media = yes
Maximum Concurrent Jobs = 100
Maximum Open Wait = 3600
Always Open = yes
}

All configuration seemed to work at this point, so I created/loaded some virtual tapes into slots 1-5 and started testing one of the tape drives, according to the Bareos documentation. I used mtx to mount a blank tape in drive 0:

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 load 4 0
Loading media from Storage Element 4 into drive 0...done

and proceeded with the TAR test:

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 rewind

(no output = all is well)

# tar cvf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 .

(omitted list of directories here -- tar completed successfully)

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 rewind

(again, no output = all is well)

# tar tvf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0

(omitted list of directories here -- spat back the same list I put in)

So all was fine up to that point. Then I shut down the storage daemon and proceeded to run the btape test:

# btape -v -c /etc/bareos/bareos-sd.conf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst
Tape block granularity is 1024 bytes.
btape: butil.c:274-0 Using device: "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst" for writing.
btape: btape.c:487-0 open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): OK
*test

=== Write, rewind, and re-read test ===

I'm going to write 10000 records and an EOF
then write 10000 records and an EOF, then rewind,
and re-read the data to verify that it is correct.

This is an *essential* feature ...

btape: btape.c:1172-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:619-0 Wrote 1 EOF to "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)
btape: btape.c:1188-0 Wrote 10000 blocks of 64412 bytes.
btape: btape.c:619-0 Wrote 1 EOF to "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)
btape: btape.c:1230-0 Rewind OK.
12-Feb 20:02 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst). ERR=Input/output error.
btape: btape.c:1250-0 Read block 1 failed! ERR=Input/output error
*

And this is the error I've gotten every time I've gone through the above procedure. The write portion of the test succeeds (or at least it appears to), but the read portion fails with the same above error every time.

I have also attempted to mount/label the volume in bconsole:

*label storage=I2M-AWS-Offsite-Storage volume=I2M-AWS-BCM-Catalog-Backup-1 pool=I2M-AWS-BCM-Pool slot=4
Automatically selected Catalog: include-aws-i2m-bcm-cat
Using Catalog "include-aws-i2m-bcm-cat"
Select Drive:
1: Drive 0
2: Drive 1
3: Drive 2
4: Drive 3
5: Drive 4
6: Drive 5
7: Drive 6
8: Drive 7
9: Drive 8
10: Drive 9
Select drive (1-10): 1
Connecting to Storage daemon I2M-AWS-Offsite-Storage at i2m-bcm.include-im.com:9103 ...
Sending label command for Volume "I2M-AWS-BCM-Catalog-Backup-1" Slot 4 ...
3914 Failed to label Volume (no media): ERR=generic_tape_device.c:140 Unable to open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): ERR=No medium found

Label command failed for Volume I2M-AWS-BCM-Catalog-Backup-1.
Do not forget to mount the drive!!!
*

This was with the slot 4 tape unmounted from drive 0 -- I was expecting Bareos to mount it automatically to do the labeling, but it appeared not to work, so then I tried mounting it myself with mtx before running the label command:

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 load 4 0
Loading media from Storage Element 4 into drive 0...done

*label storage=I2M-AWS-Offsite-Storage volume=I2M-AWS-BCM-Catalog-Backup-1 pool=I2M-AWS-BCM-Pool slot=4
Select Drive:
1: Drive 0
2: Drive 1
3: Drive 2
4: Drive 3
5: Drive 4
6: Drive 5
7: Drive 6
8: Drive 7
9: Drive 8
10: Drive 9
Select drive (1-10): 1
Connecting to Storage daemon I2M-AWS-Offsite-Storage at i2m-bcm.include-im.com:9103 ...
Sending label command for Volume "I2M-AWS-BCM-Catalog-Backup-1" Slot 4 ...
generic_tape_device.c:140 Unable to open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): ERR=No medium found
ansi_label.c:319 ANSI Volume label name "I2M-AWS-BCM-Catalog-Backup-1" longer than 6 chars.
3912 Failed to label Volume: ERR=generic_tape_device.c:140 Unable to open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): ERR=No medium found

Label command failed for Volume I2M-AWS-BCM-Catalog-Backup-1.
Do not forget to mount the drive!!!
*

--and came up with the same error.

I haven't been able to find much of anything on this situation in searches...most of the remotely similar problems were with physical tape drives with completely different configurations, and none of them exactly matched the output I'm getting here.

I've searched through various logs and also did a 200-level debug output from the Bareos storage daemon, but there is nothing obvious telling me what the problem is.

In the interest of not flooding this message with any more screen output than I already have, I will provide files/outputs to anyone who specifies what they would like to see. Hopefully someone out there has possibly dealt with this or may have an idea as to what's going on.

Thanks!

[Marco van Wieringen]

On 02/12/16 09:58 PM, Leon Ingleright wrote:
...

> Connecting to Storage daemon I2M-AWS-Offsite-Storage at i2m-bcm.include-im.com:9103 ...
> Sending label command for Volume "I2M-AWS-BCM-Catalog-Backup-1" Slot 4 ...
> generic_tape_device.c:140 Unable to open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): ERR=No medium found
> ansi_label.c:319 ANSI Volume label name "I2M-AWS-BCM-Catalog-Backup-1" longer than 6 chars.
> 3912 Failed to label Volume: ERR=generic_tape_device.c:140 Unable to open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): ERR=No medium found
>
> Label command failed for Volume I2M-AWS-BCM-Catalog-Backup-1.
> Do not forget to mount the drive!!!
> *
>
> --and came up with the same error.
>
> I haven't been able to find much of anything on this situation in searches...most of the remotely similar problems were with physical tape drives with completely different configurations, and none of them exactly matched the output I'm getting here.
>
> I've searched through various logs and also did a 200-level debug output from the Bareos storage daemon, but there is nothing obvious telling me what the problem is.
>
> In the interest of not flooding this message with any more screen output than I already have, I will provide files/outputs to anyone who specifies what they would like to see. Hopefully someone out there has possibly dealt with this or may have an idea as to what's going on.

You seem to have several problems, first of all the drive index are not right
as when mtx loads something into your virtual drive it uses the drive index
and then it tries using it and gets "No Medium found". I would also start with
one drive and get that working. Further more don't use ANSI labels they
are limited (to 6 chars as you see) and only needed if you ever think
about sharing volumes with other commercial (mostly IBM) backup software.

So I think your puzzle is to get the drive mappings right e.g. what drive
index maps to what "logical tape drive" on the OS.

--
Marco van Wieringen marco.van...@bareos.com
Bareos GmbH & Co. KG Phone: +49-221-63069389
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, M. Außendorf, J. Steffens,
P. Storz, M. v. Wieringen

[Leon Ingleright]

On Sat, Feb 13, 2016 at 2:54 AM, Marco van Wieringen <marco.van...@bareos.com> wrote:

 

You seem to have several problems, first of all the drive index are not right
as when mtx loads something into your virtual drive it uses the drive index
and then it tries using it and gets "No Medium found". I would also start with
one drive and get that working. Further more don't use ANSI labels they
are limited (to 6 chars as you see) and only needed if you ever think
about sharing volumes with other commercial (mostly IBM) backup software.

So I think your puzzle is to get the drive mappings right e.g. what drive
index maps to what "logical tape drive" on the OS.

--
Marco van Wieringen                   marco.van...@bareos.com
Bareos GmbH & Co. KG                  Phone: +49-221-63069389
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, M. Außendorf, J. Steffens,
                 P. Storz, M. v. Wieringen

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/lVsSM5UlgKQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hi Marco,

Thanks for your quick reply!  So based on your suggestions, I did some more configuring and testing.  I pared down bareos-sd.conf to include just the autochanger and one tape drive (0), as follows:

Autochanger {

  Name            = I2M-AWS-TDC

  Device          = I2M-AWS-TD0

  Changer Command = "/usr/lib/bareos/scripts/mtx-changer %c %o %S %a %d"

  Changer Device  = "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0"

}

Device {

  Name                    = I2M-AWS-TD0

  Archive Device          = /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst

  Device Type             = Tape

  Media Type              = ULTRIUM5

  Autochanger             = yes

  Drive Index             = 0

  Maximum Concurrent Jobs = 1

}

I left out any external labels as well, as you suggested.  So now the drive index (0) should be correct since there is only one drive configured.  I started with tape drive 0 empty, and loaded up a fresh set of 5 blank tapes (100 GB each) into slots 1-5:

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 status | grep 'Data Transfer Element 0'

Data Transfer Element 0:Empty

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 status | grep ':Full'

      Storage Element 1:Full :VolumeTag=IIMC362093                      

      Storage Element 2:Full :VolumeTag=IIMC2A208F                      

      Storage Element 3:Full :VolumeTag=IIMC372092                      

      Storage Element 4:Full :VolumeTag=IIMC342091                      

      Storage Element 5:Full :VolumeTag=IIMC352090

I wanted to test the autochanger with btape, but I knew that the btape autochanger test required that there be an unloaded tape in slot 1; however, I couldn't start btape without a tape loaded in the drive, so I loaded the tape from slot 2:

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 load 2 0

Loading media from Storage Element 2 into drive 0...done

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 status | grep ':Full'

Data Transfer Element 0:Full (Storage Element 2 Loaded):VolumeTag = IIMC2A208F                      

      Storage Element 1:Full :VolumeTag=IIMC362093                      

      Storage Element 3:Full :VolumeTag=IIMC372092                      

      Storage Element 4:Full :VolumeTag=IIMC342091                      

      Storage Element 5:Full :VolumeTag=IIMC352090

Next, I started btape and ran the autochanger test, to make sure that Bareos would handle changing tapes using the configuration in bareos-sd.conf:

# btape -c /etc/bareos/bareos-sd.conf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst

Tape block granularity is 1024 bytes.

btape: butil.c:274-0 Using device: "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst" for writing.

btape: btape.c:487-0 open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): OK

*autochanger

Ah, I see you have an autochanger configured.

To test the autochanger you must have a blank tape

 that I can write on in Slot 1.

Do you wish to continue with the Autochanger test? (y/n): y

=== Autochanger test ===

3301 Issuing autochanger "loaded" command.

Slot 2 loaded. I am going to unload it.

3302 Issuing autochanger "unload 2 0" command.

unload status=OK 0

3303 Issuing autochanger "load 1 0" command.

3303 Autochanger "load 1 0" status is OK.

btape: btape.c:487-0 open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): OK

btape: btape.c:1587-0 Rewound "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)

btape: btape.c:1594-0 Wrote EOF to "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)

The test autochanger worked!!

*quit

This succeeded.  I verified that the tape-swap had taken place:

# mtx -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-mediachanger-lun-0 status | grep ':Full'

Data Transfer Element 0:Full (Storage Element 1 Loaded):VolumeTag = IIMC362093                      

      Storage Element 2:Full :VolumeTag=IIMC2A208F                      

      Storage Element 3:Full :VolumeTag=IIMC372092                      

      Storage Element 4:Full :VolumeTag=IIMC342091                      

      Storage Element 5:Full :VolumeTag=IIMC352090

Since the slot 1 tape was loaded and blank, I ran the TAR test one more time:

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 rewind

# echo $?

0

# tar cvf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 .

./

./.ssh/

./.ssh/authorized_keys

./install_bareos

./.bash_history

./.bashrc

---snip---

./.local/share/mc/

./.local/share/mc/history

./.local/share/mc/filepos

./.local/share/mc/mcedit/

./.selected_editor

./.profile

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0 rewind

# echo $?

0

# tar tvf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0

drwx------ root/root         0 2016-02-12 16:05 ./

drwx------ root/root         0 2016-01-21 20:27 ./.ssh/

-rw------- root/root       548 2016-01-21 20:27 ./.ssh/authorized_keys

-rwxr-xr-x root/root       476 2016-02-11 15:47 ./install_bareos

-rw------- root/root     56482 2016-02-12 21:21 ./.bash_history

-rw-r--r-- root/root      3106 2014-02-20 02:43 ./.bashrc

---snip---

drwx------ root/root             0 2016-02-14 13:43 ./.local/share/mc/

-rw------- root/root          2780 2016-02-14 13:43 ./.local/share/mc/history

-rw-r--r-- root/root          3344 2016-02-14 13:43 ./.local/share/mc/filepos

drwx------ root/root             0 2016-01-21 22:45 ./.local/share/mc/mcedit/

-rw-r--r-- root/root            72 2016-01-21 22:45 ./.selected_editor

-rw-r--r-- root/root           140 2014-02-20 02:43 ./.profile

So I was able to read and write successfully to the tape using TAR.  I then erased the tape and proceeded to run the btape basic test:

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst rewind

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst erase

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst rewind

# btape -c /etc/bareos/bareos-sd.conf /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst

Tape block granularity is 1024 bytes.

btape: butil.c:274-0 Using device: "/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst" for writing.

btape: btape.c:487-0 open device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst): OK

*test

=== Write, rewind, and re-read test ===

I'm going to write 10000 records and an EOF

then write 10000 records and an EOF, then rewind,

and re-read the data to verify that it is correct.

This is an *essential* feature ...

btape: btape.c:1172-0 Wrote 10000 blocks of 64412 bytes.

btape: btape.c:619-0 Wrote 1 EOF to "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)

btape: btape.c:1188-0 Wrote 10000 blocks of 64412 bytes.

btape: btape.c:619-0 Wrote 1 EOF to "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)

btape: btape.c:1230-0 Rewind OK.

14-Feb 14:26 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst). ERR=Input/output error.

btape: btape.c:1250-0 Read block 1 failed! ERR=Input/output error

*quit

As you can see, once again, the write portion of the test is successful -- the two 10000-block files are written to the tape without error, and btape also rewinds the tape without error.  But as soon as btape tries to read back the files from the tape, it throws the EIO message and halts the test.

Just to verify that the write portion of the test actually succeeded, I manually checked the tape:

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst status

SCSI 2 tape drive:

File number=0, block number=-1, partition=0.

Tape block size 0 bytes. Density code 0x0 (default).

Soft error count since last status=0

General status bits on (1010000):

 ONLINE IM_REP_EN

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsf

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst status

SCSI 2 tape drive:

File number=1, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x0 (default).

Soft error count since last status=0

General status bits on (81010000):

 EOF ONLINE IM_REP_EN

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsf

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst status

SCSI 2 tape drive:

File number=2, block number=0, partition=0.

Tape block size 0 bytes. Density code 0x0 (default).

Soft error count since last status=0*quit

General status bits on (81010000):

 EOF ONLINE IM_REP_EN

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsf

/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst: Input/output error

As expected, I was able to forward-space by file exactly twice, each time status showing the drive positioned at the beginning of each file.  On the third attempt, I got the expected error, as btape only wrote two files to the tape.  I also forward-spaced by record:

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst rewind

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsf

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsr 2000

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst status

SCSI 2 tape drive:

File number=1, block number=2000, partition=0.

Tape block size 0 bytes. Density code 0x0 (default).

Soft error count since last status=0

General status bits on (1010000):

 ONLINE IM_REP_EN

# mt -f /dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst fsr 8001

/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst: Input/output error

Since the btape test had written 10,000 records per file, I was able to forward-space over the first 2,000 records, but attempting to forward-space another 8,001 records (one more record than the end of the file), I got the expected error.  In any case, this confirms for me that btape was, indeed, successful in writing to the tape.  I was even able to successfully maneuver between the files and records using the commands in btape; however, any attempt to read one or any number of blocks always throws the EIO error.  So the only thing not working from the btape/Bareos perspective is the read function.

Just to be sure I didn't miss anything, I ran the btape test again in full debug mode; here is the output from the beginning of the rewind:

btape (400): generic_tape_device.c:1115-0 rewind res=0 fd=3 "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst)

btape: btape.c:1230-0 Rewind OK.

btape (250): block.c:960-0 Full read in read_block_from_device() len=64512

btape (250): block.c:1000-0 Read device got: ERR=Input/output error

btape (250): sd_plugins.c:237-0 No bplugin_list: generate_plugin_event ignored.

btape (850): message.c:1519-0 Enter Jmsg type=4

btape (850): message.c:848-0 Enter dispatch_message type=4 msg=btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst). ERR=Input/output error.

btape (850): message.c:1112-0 STDOUT for following msg: btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst). ERR=Input/output error.

14-Feb 15:14 btape JobId 0: Error: block.c:1003 Read error on fd=3 at file:blk 0:0 on device "I2M-AWS-TD0" (/dev/tape/by-path/ip-172.30.0.99:3260-iscsi-iqn.1997-05.com.amazon:sgw-f4896c9d-tapedrive-01-lun-0-nst). ERR=Input/output error.

btape: btape.c:1250-0 Read block 1 failed! ERR=Input/output error

btape (950): record.c:289-0 Enter free_record.

btape (950): record.c:293-0 Data buf is freed.

btape (950): record.c:295-0 Leave free_record.

*quit

There is still no indication of what is causing the EIO, but you can clearly see that it happens right after the call to read_block_from_device().  Digging into the source code, it appears that the error happens at line 992 of block.c:

status = dev->read(block->buf, (size_t)block->buf_len);

That's as granular as I could trace it.  Whatever is happening in the call to dev->read(...) is causing the EIO error, but there's no real indication of what the problem is.  I have checked permissions everywhere (the 'bareos' user is, in fact, a member of 'tape' group, which has rw permissions for all of the tape drives), and I don't see anything that should be stopping the read function from working.  I have tailed dmesg, syslog, and kern logs through the above processes, but nothing useful shows up in any of them.

Any further insight or suggestions would be greatly appreciated.  Thanks!

[Andrew Replogle]

I just wanted to throw on a "Me Too" to this thread.

I'm using 15.2.2 with the latest AWS SG VTL.

I saw in the changelog, that 15.4.x had a refactored SD and I was curious if that might address whatever issue we're seeing here.

I'm going to try and get more info with sysdig / trace.

@Leon, were you able to make any progress here? I've tried switching the media-changer device type (it was a long shot since the read issue seems to be specific to the drive?) but no luck there.

Regards.

[Andrew Replogle]

Just a heads up, I updated to nightly (16.1) and the btape test still fails on read.

[Andrew Replogle]

FYI - reported issue here: https://bugs.bareos.org/view.php?id=639

[Sassy Natan]

Did u manage to find a solution for this?

It seems I have the same issue

[Leon Ingleright]

No, I never did get an answer to this.  My former company ended up going with a commercial solution and I left shortly after.  It's too bad that I couldn't get it to work; I think it would have been a great cost-effective backup solution, especially for small businesses.

On Jul 31, 2016 5:03 PM, "Sassy Natan" <sas...@gmail.com> wrote:

Did u manage to find a solution for this?

It seems I have the same issue

[Sassy Natan]

Yes,
I totally agree with you.
It seems like a great solution, and based on the documentation and http://osbconf.org/wp-content/uploads/2015/10/Using-AWS-Virtual-Tape-Library-as-Storage-for-Bacula-Bareos-by-Alberto-Gimen%C3%A9z.pdf it should work

But something wired when u try to write or make a label.
I follow the command you did, and I have no idea what is wrong.

Looking in the C code compare to Bacula, it seem to me related block.c code under line 644 with ENOSPC.

Maybe some one will help from the Bareos.

I can switch to Bacula, I know it should work there, but really want to stick with Bareos.

Thanks for the quick replay

[Faelens, Ruben (Belgium)]

Dear list,

My (ghetto) alternative:
- Backup to disk. Every job uses a new volume, labelled based on Job + Creation Time
- When a Job completes, use Python Boto [1] to upload the file to Amazon Glacier and delete it on local disk. Store the Glacier file ID somewhere on disk (and store the filename in Glacier metadata, see FastGlacier [2] for more info).

- Restore operations require you to retrieve the files back, but it's easy to write a command-line tool to do this.
- Volume purging requires you to manually delete the volume from Glacier

Kind regards,
Ruben

[1] : http://boto.cloudhackers.com/en/latest/
[2] : https://fastglacier.com/fastglacier-metadata-format.aspx

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.

To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Information in this email and any attachments is confidential and
intended solely for the use of the individual(s) to whom it is addressed
or otherwise directed. Please note that any views or opinions presented
in this email are solely those of the author and do not necessarily
represent those of the Company.
Finally, the recipient should check this email and any attachments for
the presence of viruses. The Company accepts no liability for any damage
caused by any virus transmitted by this email.
All SGS services are rendered in accordance with the applicable SGS
conditions of service available on request and accessible at
http://www.sgs.com/en/Terms-and-Conditions.aspx