The truth about hardware compression
Conrad Lawes
One thing that I have never gotten to work is hardware compression. I used to use LTO3 tapes that natively storage 400GB and 800GB with compression. Never got beyond 400GB.
I've upgraded by tape library to a Dell TL2000 tape library with LTO6 tape drive.
LTO6 natively stores 2.5TB and 6.25TB with compression. Hardware compression is turned on. However, I never get above 2.5TB of data on each tape.
I read somewhere that you must spool your data before writing to tape because Bacula/Bareos requires a steady stream for compression to work. Tried that, still no dice. And yes, I have disabled soft compression for my backup jobs. All the documentation I found online are limited and/or out-of-date.
I'm presently using 2 LTO6 tapes to complete a full backup (~3.5TB of data). Obviously, my 2nd tape is under utilized and should not be required if 6.25TB capacity is achievable.
Is there any H/W compression expert that can shed some light on what is required by Bareos to get compression working as advertised?
Thanks.
lst_...@kwsoft.de
Zitat von Conrad Lawes <pxe...@gmail.com>:
delivered data. If the compression ratio is too small or negative the
compression is turned off. What is needed to get a good ration is
always the same:
- data which are not already compressed, so no jpeg,mpeg,zip and the like
- no compression done in software before transfering the data to tape
- no encryption because this hides the redundancy the compression is
looking for
spooling data is about speed, not compression. So simply do fill a
tape with well compressable data and compare the ratio with for
example a zip archive.
Regards
Andreas
Marco van Wieringen
>
>
> Zitat von Conrad Lawes <pxeboot <at> gmail.com>:
Correct spooling is only needed to not get shoe-shining e.g. keep
the drives streaming (for LTO6 you need quite some speed to keep
things running e.g. see
https://en.wikipedia.org/wiki/Linear_Tape-Open
e.g. 160 Mb/s uncompressed which is probably more then any single disk
can give.
The whole compression also kind of depends on your operating system
and even on the kernel version used. The Amanda project has some notes
on this:
https://wiki.zmanda.com/index.php/Hardware_compression
(b.t.w. searching on google on linux st driver compression has quite
some hits. ST is the kernel driver that abstracts the SCSI Tape over
either SAS or FC.)
You also might want to look into the tapeinfo of your tape drive,
for my drive in my T24 tandberg library with LTO4 SAS drive on
Solaris this gives:
15:14 [root:corona][319] ~ > tapeinfo -f /dev/rmt/0ubn
Product Type: Tape Drive
Vendor ID: 'HP '
Product ID: 'Ultrium 4-SCSI '
Revision: 'U52U'
Attached Changer API: No
SerialNumber: 'HU1145KF03'
MinBlock: 1
MaxBlock: 16777215
Ready: yes
BufferedMode: yes
Medium Type: Not Loaded
Density Code: 0x44
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
Block Position: 8126794
Partition 0 Remaining Kbytes: 400308
Partition 0 Size in Kbytes: 400308
ActivePartition: 0
EarlyWarningSize: 0
NumPartitions: 0
MaxPartitions: 0
The following things are mostly interesting:
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
And things have also worked quite well for LTO3 I always got more
then 400 Gb and now with LTO4 tapes also get more then 800 Gb. I had
some problems before with bad LTO tapes but now with HP certified
LTO3 (Yellow) and LTO4 (Green) things look quite ok. Also keep in mind
its the famous disk vendor GB and NOT GiB
Some examples:
For incrementals/differentials:
| 29 | MHE345L3 | Full | 1 | 561,414,389,760 | 1,326 |
157,680,000 | 1 | 0 | 0 | LTO | 2011-08-10 08:06:56 |
| 33 | MHE349L3 | Full | 1 | 241,996,640,256 | 169 |
157,680,000 | 1 | 0 | 0 | LTO | 2011-11-05 08:05:40 |
| 36 | MHE352L3 | Full | 1 | 403,016,076,288 | 484 |
157,680,000 | 1 | 0 | 0 | LTO | 2011-10-16 08:13:47 |
| 55 | MHE361L3 | Full | 1 | 418,060,339,200 | 737 |
157,680,000 | 1 | 0 | 0 | LTO | 2011-02-03 08:08:48 |
| 82 | PLN003L3 | Full | 1 | 828,656,640,000 | 1,077 |
157,680,000 | 1 | 0 | 0 | LTO | 2012-07-14 10:05:40 |
| 86 | PLN007L3 | Full | 1 | 614,782,393,344 | 1,552 |
157,680,000 | 1 | 7 | 1 | LTO | 2013-03-31 10:06:20 |
| 95 | PLN016L3 | Full | 1 | 481,131,463,680 | 1,259 |
157,680,000 | 1 | 16 | 1 | LTO | 2013-10-25 09:05:31 |
| 104 | PLN102L4 | Full | 1 | 885,321,116,672 | 1,374 |
157,680,000 | 1 | 23 | 1 | LTO | 2014-06-05 10:08:36 |
| 115 | PLN107L4 | Append | 1 | 979,187,072,000 | 2,357 |
157,680,000 | 1 | 5 | 1 | LTO | 2015-07-10 09:01:05 |
For Fulls:
| 101 | PLN020L3 | Full | 1 | 508,205,988,864 | 138
| 157,680,000 | 1 | 20 | 1 | LTO | 2013-09-01 09:19:29 |
| 102 | PLN021L3 | Full | 1 | 499,406,358,528 | 139
| 157,680,000 | 1 | 21 | 1 | LTO | 2013-10-06 09:09:07 |
| 103 | PLN101L4 | Full | 1 | 1,010,252,371,968 | 269
| 157,680,000 | 1 | 22 | 1 | LTO | 2013-12-01 09:40:53 |
| 105 | PLN103L4 | Full | 1 | 992,451,511,296 | 263
| 157,680,000 | 1 | 1 | 1 | LTO | 2014-02-02 09:51:31 |
| 106 | PLN104L4 | Full | 1 | 1,012,483,519,488 | 265
| 157,680,000 | 1 | 2 | 1 | LTO | 2014-04-06 09:50:07 |
| 107 | PLN105L4 | Full | 1 | 1,005,624,935,424 | 258
| 157,680,000 | 1 | 3 | 1 | LTO | 2014-06-01 09:43:43 |
| 112 | PLN106L4 | Full | 1 | 1,001,966,861,312 | 249
| 157,680,000 | 1 | 4 | 1 | LTO | 2014-07-20 09:14:42 |
That is all on Solaris with a tape device named /dev/rmt/0ubn which means
use ultra compression (u) BSD type (b) non-rewinding (n)
--
Marco van Wieringen marco.van...@bareos.com
Bareos GmbH & Co. KG Phone: +49-221-63069389
http://www.bareos.com
Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: Stephan Dühr, M. Außendorf, J. Steffens,
P. Storz, M. v. Wieringen
Bruno Friedmann
or do some test with uncompressed tif image.
You will then achieve those ratio.
Every binary is mostly compressed, all jpeg png files too and you will probably seen on a few %
--
Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
openSUSE Member & Board, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot