Linux Virtual Tape Library
177 views
Skip to first unread message
mhvtl
Sep 3, 2010, 1:39:40 PM9/3/10
to MHVTL - Virtual Tape Library
Linux Virtual Tape Library
Description
vtltape
vtllibrary
vtlcmd
Download - stable
Download - development
License:
GPL v2
Q. Why GPL license.
A. Because I like the idea of sharing.
I don't profess to know everything (not even close). Having something
I can share and allow others to share back is a good idea.
Virtual Tape Library consists of several components.
LLD - A low level driver implemented as a kernel module - mhvtl
Target devices - Daemons vtltape(1) and vtllibrary(1) which implement
SCSI target device(s) in user-space
Utility commands mktape(1), vtlcmd(1)
And startup scripts build_library_config(1), make_vtl_devices(1)
The kernel module is based on the scsi_debug kernel module (http://
sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note:
As of 0.16, there are no default devices.
The support scripts will add Drives (SSC devices) & library (SMC)
depending on the contents of /etc/mhvtl/library_contents and /etc/
mhvtl/device.conf.
A char device back-end has been included with the vtl LLD driver This
allows data and SCSI commands to be passed from the LLD to user-mode
daemons (SCSI targets) which constently poll the driver and process
any outstanding SCSI commands.
vtltape
vtltape(1) is the usermode SSC target daemon which writes/reads data
to data files in the /opt/vtl directory (if a virtual tape has been
loaded). The virtual tape files include a Medium Auxiliary Memory
(MAM) data structure to store persistent data (number of tape loads,
total data written/read, media type etc).
vtllibrary
vtllibrary(1) is the usermode SMC target daemon which reads its
configuration from the file /etc/mhvtl/library_contents(5) at startup.
The number of storage slots are built dynamically when the daemon
starts. Hence changing the number of storage slots and media access
slots are a matter of modifying the file contents and restarting the
vtllibrary(1) daemon. All 'library' commands are performed on data
structures in memory ONLY.
vtlcmd
A utility vtlcmd(1) is used to administrator the daemons vtltape(1)
and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between
vtlcmd(1), vtllibrary(1) and vtltape(1)
When a SCSI 'move medium' from a storage slot to a tape drive is
requested, the media location is updated in vtllibrary(1) data
structures, and the barcode of the media id is passed via the message
queue to the vtltape(1) daemon in question. A file open of the barcode
is attempted in the /opt/vtl directory and if successful, the
vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit
Ready requests. An unload SCSI command to the tape drive will close
the data file.
Media can be moved out of the VTL via the Media Access Port. Once
media is logically moved into the MAP slots, the MAP entries can be
cleared using the vxcmd:
# vtlcmd library empty map
The media can be 'moved back again' by re-starting the VTL user-space
daemons:
Media can be loaded into the MAP via a :
# vtlcmd library load map <barcode>
or restart the vtl
# /etc/init.d/mhvtl restart
TapeAlert
TapeAlert flags can be set using the vtlcmd(1)
e.g. vtlcmd [index|library] TapeAlert [flags]
Where index is the message Q offset associated with the drive (or the
string 'library').
e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000
# vtlcmd 1 TapeAlert 80000
A listing of TapeAlert flags can be found at t10 home page for SSC
devices Annex A.
Examples
mhvtl is registered as HBA #3:
# lsscsi -g
[0:0:0:0] disk MAXTOR ATLAS10K4_36SCA DFM0 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x3 SCSI BP 1.1 - /dev/sg1
[2:0:0:0] disk SEAGATE ST336607FC 0003 /dev/sdb /dev/sg2
[2:0:1:0] disk SEAGATE ST336607FC 0003 /dev/sdc /dev/sg3
[2:0:2:0] mediumx ATL 1500 6.0 - /dev/sg4
[2:0:2:1] tape QUANTUM SuperDLT1 2323 /dev/st0 /dev/sg5
[2:0:2:2] tape QUANTUM SuperDLT1 2323 /dev/st1 /dev/sg6
[2:0:2:3] process ATTO 310-LV 1.42 - /dev/sg7
[3:0:0:0] mediumx STK L700 vtl0 - /dev/sg8
[3:0:0:1] tape SONY SDX-900V 5400 /dev/st2 /dev/sg9
[3:0:0:2] tape SONY SDX-900V 5400 /dev/st3 /dev/sg10
[3:0:0:3] tape QUANTUM SDLT600 5400 /dev/st4 /dev/sg11
[3:0:0:4] tape QUANTUM SDLT600 5400 /dev/st5 /dev/sg12
[3:0:0:5] tape QUANTUM SDLT600 5400 /dev/st6 /dev/sg13
[3:0:0:6] tape IBM ULT3580-TD3 5400 /dev/st7 /dev/sg14
[3:0:0:7] tape IBM ULT3580-TD3 5400 /dev/st8 /dev/sg15
[3:0:0:8] tape IBM ULT3580-TD3 5400 /dev/st9 /dev/sg16
Long term goals
Eventually, the functionality will be moved across to the SCSI Target
Framework (stgt) http://stgt.berlios.de/
The stgt is a very nicely designed SCSI taret framwork which can
support all target types along with multiple SCSI transports in a very
structured way.
However, until time permits (and my c coding standards improve),
features and improvements on this code base will continue.
Howto get help
Somebody has put up a discussion board (and has lots of great
documentation on working with different backup software)
http://mhvtl.nimsa.us/
Howto help out
There are many way of providing assistance
Testing with backup software X and reporting success/failure. If
possible, test any patches that may result of testing.
If you are willing, patches are very welcome.
Please read the Linux Documentation/CodingStyle
Note: I only discovered this documentation guide recently, so patches
to update old code style to the CodingStyle are also welcome.
Please run any patches thru the Linux 'mhvtl-<version>/scripts/
checkpatch.pl' for coding style sanity check before sending.
Download
Stable vers:
mhvtl-0.16-13.src.rpm - Source RPM
mhvtl-0.16-13.i586.rpm - 32bit RPM
mhvtl-0.16-13.x86_64.rpm - 64bit RPM
mhvtl-2010-05-09.tgz - Source (tar ball) - (Builds mhvtl-0.16-12)
Change log:
mhvtl-2010-05-09 / mhvtl-0.16-13
*** Happy Mothers Day ***
- Allow media sizes larger than 2G on 32bit platforms
- Implement STK vendor specific 'LOAD DISPLAY' (Logs info via syslog)
- Fix core dump on invalid NAA strings
- Support VENDOR ID strings with embedded spaces
- Return LOG SENSE 'Tape Capacity' in OEM format (Bytes/KB/MB values)
- Limit SMC op codes when in offline state
mhvtl-2010-03-10 / mhvtl-0.16-12
- Introduce multiple library support. Device.conf format had to
change.
- Introduce script to convert old device.conf to new format
- Test media loaded on 'REZERO' op code
- Fix MAM struct size to be 1024 bytes
- Fix SPR RESERVATION bug.
- Dynamic change of device queue depth (kernel module change)
- Improve SCSI OP code logging
mhvtl-2009-12-16 / mhvtl-0.16-11
New and improved Compression: entry in device.conf (Thanks Kevan)
- Compression: factor [1-9] enabled [0|1]
Factor 1 == Fastest compression -> 9 == Best compression (zlib)
- Add tests to only allow LTO media to be mounted in LTO drives. If
media is 'unknown', then the mount will succeed.
mhvtl-2009-12-02 / mhvtl-0.16-10
Fix silent data corruption (Thanks Kevan)
Increase max LUNs from 7 to 32
mhvtl-2009-11-29 / mhvtl-0.16-9
Check string length of definitions within device.conf
Compression now set/resetable via MODE SELECT. (Thanks Kevan)
- Default compression set in device.conf
Compression: X (where X is a value from 0 [off], 1 [ Fastest
compression], => 9 [Best compression]
Default library INQUIRY string changed from STK/L700 to SPECTRA/PYTHON
Fix kernel compile on Ubuntu 9.10 (2.6.31+)
Fix max block size on x86_64 (Thanks Kevan)
SPIN/SPOUT: Return correct list of supported pages.
mhvtl-2009-11-10 / mhvtl-0.16-8
Add ability to compress data written to backing storage file.
Overhaul READ ELEMENT STATUS. Thanks Kevan Rehm.
Makefile overhaul.
- DESTDIR understood.
- Ability to define default location for media files & config file
parent directories
- Ability to define default user/group account for mhvtl daemons.
mhvtl-2009-10-09 / mhvtl-0.16-7
Fix READ ELEMENT STATUS for All element types. Many thanks to Bernardo
Clavijo for the report and fix. This bug caused TSM to core dump.
Improve (make more life like) Media Access Port access.
Added 'map open' and 'map close' options to the vtlcmd syntax.
If the MAP is open, then the 'Medium Transport Element' fails to move
media to/from a MAP port.
If the MAP is closed, then the user (via vtlcmd) can not empty MAP or
place new media into the MAP.
mhvtl-2009-10-01 / mhvtl-0.16-6
Fix test comparison within MHVTL_DBG macro
Fix output of the dump_tape utility
mhvtl-2009-09-15 / mhvtl-0.16-4
Bug fix only - no functionality changes. No need to rebuild kernel
module from 0.16-3
Fix up VERBOSE testing with MHVTL_DBG macro
Clean up more logging message(s)
mhvtl-2009-09-02 / mhvtl-0.16-3
Increased max block size from 256k to kernel default (kernel module
needs to be rebuilt)
Re-worked logging to use a single macro 'MHVTL_DBG' which will
either use syslog in verbose mode or printf if started in debug
mode.
mhvtl-2009-08-15 / mhvtl-0.16-2
Changed/updated load MAP() functionality.
Previous implementation re-read config file and parsed the MAP
slots.
This is 'interactive' using the vtlcmd command.
e.g. vtlcmd library load map <BARCODE>
Corrected NAA field in VPD page 0x83. This was a hard coded string.
Now reads entry from /etc/mhvtl/device.conf
' NAA: 11:22:33:44:ab:cd:ef:03' (8 octet value)
Remove dependency on sg3_utils
mhvtl-2009-08-05 / mhvtl-0.16-1
- Fix kernel module oops with logical unit removal (Many thanks to
Jean-François CHALARD for the fix)
- Fill in loadMap() routine within vtllibrary (Many thanks to Jean-
François CHALARD for the code)
- Fix compile issue on SLES 9
mhvtl-2009-06-09 / mhvtl-0.16-0:
- Renamed final pieces from 'vtl' to 'mhvtl' (e.g. /etc/mhvtl/ instead
of /etc/vtl)
- Almost everything is handled in user-space now. As daemons start,
the new logical unit is added to the pseduo hba driver. As the
daemon(s) are terminated, the logical unit is removed from the pseudo
hba driver.
- kernel module 'oops' if unloaded and re-loaded. It is stable
otherwise. Root cause unknown at this point in time.
Development version:
Download :
mhvtl-2010-09-01.tgz (builds mhvtl-0.18-10)
mhvtl-0.18-10.x86_64.rpm
mhvtl-0.18-10.i586.rpm
mhvtl-0.18-10.src.rpm
Kevan Rehm has re-written the on-disk data structure.
(It's amazing how much effort can be rolled up in one simple
sentence).
Each 'tape' is now stored under its own subdirectory /opt/mhvtl/
<barcode>
A tape consists of three files.
- Data file => Container of data
- Index file => Contains an array of header structures (pointers into
the data file)
- Meta file => Contains the MAM structure, followed by a meta data
header, followed by a variable-length array of filemark block numbers.
mhvtl-2010-09-01.tgz / mhvtl-0.18-10:
- Fix TapeAlert
- Added more descriptive logging around SPACE OP code attempting to
track down problem with HP Dataprotector
- Added (undocumented) utility 'tapeexerciser' which is a simple
utility to test tape read/writes and positioning. WARNING: This will
overwrite your media..
- Fix 'is vtl running' logic in rc script. Failed test if 'vtl user'
is root (not a recommended config)
mhvtl-2010-07-09.tgz / mhvtl-0.18-9:
- Fix WORM handling - Reported by Albert Pauw
- Fix (finished) LTO5 drive & media handling introduced in 0.18-8
mhvtl-2010-06-23.tgz / mhvtl-0.18-8:
- Several Security Protocol IN updates - thanks Albert Pauw
* Return certificate data
* Correct length for 'KEY FORMATS'
* Correct length for SPIN SUPPORTED PAGES
- Fix kernel compile on RedHat AS4
- Media/drive matching now 'dynamic' and defined in device.conf
- Added man page for device.conf
- Fix media corruption when media is 'formatted'
- Add LTO5 & SDLT-S4 drive/media types
- Handle INQUIRY correctly after media change (return SAM_STAT_GOOD)
- Updated rc script so all devices created on Target & LUN. i.e. Don't
use
channel. Some application software has trouble if only the channel
is unique.
mhvtl-2010-05-08.tgz / mhvtl-0.18-7:
Fix: 32bit platforms - Allow media size to be greater than 2G
Fix: core dump on invalid defined NAA strings
Fix: Support SMC devices with embedded space chars in VENDOR ID
Implement STK Vendor unique LOAD DISPLAY SCSI op code. (Logs display
info via syslog)
mhvtl-2010-05-02.tgz / mhvtl-0.18-6:
Fix: vtlcmd command line parsing. Fix 'list map' option
Fix: import of media via MAP (off-by-one)
Fix: TapeCapacity LOG SENSE data.
Now returns bytes/KB/MB as per OEM (Thanks 'kamy')
Support VENDOR ID with embedded spaces. i.e. "HP Ultrium"
Honour MAP status: Fail attempts for robot to move media in CAP if CAP
is open.
Relax dependencies on /proc in favour of /sys (note: Work in progress)
mhvtl-2010-04-01.tgz / mhvtl-0.18-5:
Fix: vtlcmd command line parsing. Thanks Herbert Stadler
Fix: Silence local_irq_save(). Normally seen on fedora. Thanks Norm
Lunda
Fix: URLs to this homepage.
mhvtl-2010-03-05.tgz / mhvtl-0.18-4:
Fix: ELEMENT STATUS - found and fixed by Norm Lunde (many thanks)
- This was preventing TSM to work with mhvtl
Add queue depth() callback in kernel module (module date string
20100303)
Test for media mounted before returning status on 'REWIND' op code.
Fix typos in rc script
mhvtl-2010-01-28.tgz / mhvtl-0.18-2...rpm:
The ability to support multiple libraries on the one host !! Thanks
once again to Kevan.
Note: There is no conversion of existing configuration file(s) to the
new device.conf/library_contents format. Please remove all files
under /etc/mhvtl/ before installing mhvtl-2010-01-28 (mhvtl-0.18-2)
What hardware does it work with:
Tested with Pentium x86 (32bit) and Pentium / AMD x86_64
New: IBM Z-Series. To quote:
"We just fired up the VTL on our z series IBM mainframe. Works like a
charm."
What software does it work with:
Symantec NetBackup.
5.1 (Recommend 5.1MP4 minimum)
6.0
6.5.x
7.0
Supports SCSI Persistent Reservation
Encryption via KMS (NetBackup 6.5.2 and later)
Note: All mhvtl code development is tested using NetBackup
Last set of testing (0.18-8 & -9) with NetBackup 7.0, I had to revert
the 'STK/L700' to 'SPECTRA/PYTHON' change made back in Nov 2009. It's
just a manual edit of /etc/mhvtl/device.conf
The problem is that NetBackup 7.0 fails to extract the device serial
number if SPECTRA/PYTHON is used and hence the automagic device
discovery & configuration fails to place the drives within the robot.
Symantec BackupExec.
For the extraordinary writeup : http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37
Includes how to setup the scst (http://mhvtl.nimsa.us/viewtopic.php?
f=23&t=32) to present the vtl via iSCSI
I've receive updates from a anonymous contributor known as 'nia':
Successful Testing of Solaris running mhvtl via iSCSI:
http://mhvtl.nimsa.us/viewtopic.php?f=23&t=47
Successful Testing of Sun NDMP working with mhvtl via iSCSI
http://mhvtl.nimsa.us/viewtopic.php?f=20&t=53
Successful Testing of Linux ZFS-fuse file-system with "dedup on" for /
opt/mhvtl http://mhvtl.nimsa.us/viewtopic.php?f=13&t=31
Successful Testing of HP Data Protector Linux media agent
http://mhvtl.nimsa.us/viewtopic.php?f=29&t=51
Successful Testing for Backup Exec 2010 on Windows Server 2008 R2.
Tested version of mhvtl is 0.18.2 via iSCSI http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159
Successful Testing for NetBackup 7 Master on Windows 2008 R2 via iSCS
http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159
EMC/Legato NetWorker
Is no longer causing headaches !
Don't use 'STK/L700' Use 'SPECTRA/PYTHON' instead.
Install mhvtl-2009-12-16.tgz (mhvtl-0.16-11).
Build kernel module from source
# rpm -Uvh mhvtl-0.16-11.src.rpm
# cd /usr/src/packages/BUILD
# tar xvfz ../SOURCES/mhvtl-2009-12-16.tgz
# cd mhvtl-0.16/kernel
# make
# make install
Start the daemons
/etc/init.d/mhvtl start
Configure NetWorker as normal via jbconfig.
An ex colleague who has some very useful/helpful tips with NetWorker
(If your interested in NetWorker, his blogs & tips are well worth
following)
http://nsrd.info/blog/2009/07/13/carry-a-jukebox-with-you-if-youre-using-linux/
Also a follow up article
http://nsrd.info/blog/2009/11/14/networker-and-linuxvtl-redux/
Bakbone Netvault
(2010-01-09) Thanks to Nick Couchman who has reported NetVault 8.x and
mhvtl-2009-12-16 (mhvtl-0.16-11) do indeed work together.
To quote Nick "I have a NetVault 8.x installation here that I'm using,
and I just installed LinuxVTL2 and am successfully backing up data
from NetVault to LinuxVTL2. I'm not sure if any issues exist for
restores and the link, but I'll let you know if I run into anything.
All appears to be working fine at this point! FWIW I'm running my
NetVault server on RHEL5."
TSM
Thanks to Bernardo Clavijo, TSM now works with mhvtl - Need at least
0.18-4 release.
Bacular
A report from Beat Rubischon to advise Bacular works also.
Two changes were needed on Bacula's
bacula-sd.conf:
Device {
Name = Drive-1
Drive Index = 0
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes;
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
AutoChanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No
}
Multiple tape drives are supported too.
New: 2010-05-21
Report by Albert Pauw.. Many thanks.
# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No
I have used it this way for the last two months, but due to some
questions on the mhvtl forum I tried
bacula without these additions. Seems the latest versions of mhvtl
work out of the box with bacula,
so these additions are not needed anymore.
Getting Started
To install and get the package running:
1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version
- check using uname -p)
2. Build kernel module from source you just installed:
cd /usr/src/packages/BUILD
tar xvfz ../SOURCES/mhvtl-2009-12-16.tgz
cd mhvtl-0.16/kernel
make
make install
3. Start the package using the rc script.
/etc/init.d/mhvtl start
4. Check it's all running:
lsscsi -g
ps -ef|grep vtl (default with 5 daemons - 1 vtllibrary & 4 vtltape)
5. Configure your backup software ;)
mark_...@symantec.com
Description
vtltape
vtllibrary
vtlcmd
Download - stable
Download - development
License:
GPL v2
Q. Why GPL license.
A. Because I like the idea of sharing.
I don't profess to know everything (not even close). Having something
I can share and allow others to share back is a good idea.
Virtual Tape Library consists of several components.
LLD - A low level driver implemented as a kernel module - mhvtl
Target devices - Daemons vtltape(1) and vtllibrary(1) which implement
SCSI target device(s) in user-space
Utility commands mktape(1), vtlcmd(1)
And startup scripts build_library_config(1), make_vtl_devices(1)
The kernel module is based on the scsi_debug kernel module (http://
sg.danny.cz/sg/sdebug26.html). mhvtl.ko is a pseudo HBA (LLD). Note:
As of 0.16, there are no default devices.
The support scripts will add Drives (SSC devices) & library (SMC)
depending on the contents of /etc/mhvtl/library_contents and /etc/
mhvtl/device.conf.
A char device back-end has been included with the vtl LLD driver This
allows data and SCSI commands to be passed from the LLD to user-mode
daemons (SCSI targets) which constently poll the driver and process
any outstanding SCSI commands.
vtltape
vtltape(1) is the usermode SSC target daemon which writes/reads data
to data files in the /opt/vtl directory (if a virtual tape has been
loaded). The virtual tape files include a Medium Auxiliary Memory
(MAM) data structure to store persistent data (number of tape loads,
total data written/read, media type etc).
vtllibrary
vtllibrary(1) is the usermode SMC target daemon which reads its
configuration from the file /etc/mhvtl/library_contents(5) at startup.
The number of storage slots are built dynamically when the daemon
starts. Hence changing the number of storage slots and media access
slots are a matter of modifying the file contents and restarting the
vtllibrary(1) daemon. All 'library' commands are performed on data
structures in memory ONLY.
vtlcmd
A utility vtlcmd(1) is used to administrator the daemons vtltape(1)
and vtllibrary.
Message queue (key 0x4d61726b) is used to pass messages between
vtlcmd(1), vtllibrary(1) and vtltape(1)
When a SCSI 'move medium' from a storage slot to a tape drive is
requested, the media location is updated in vtllibrary(1) data
structures, and the barcode of the media id is passed via the message
queue to the vtltape(1) daemon in question. A file open of the barcode
is attempted in the /opt/vtl directory and if successful, the
vtltape(1) daemon will now return a ASC/ASCQ 0x0/0x0 to any Test Unit
Ready requests. An unload SCSI command to the tape drive will close
the data file.
Media can be moved out of the VTL via the Media Access Port. Once
media is logically moved into the MAP slots, the MAP entries can be
cleared using the vxcmd:
# vtlcmd library empty map
The media can be 'moved back again' by re-starting the VTL user-space
daemons:
Media can be loaded into the MAP via a :
# vtlcmd library load map <barcode>
or restart the vtl
# /etc/init.d/mhvtl restart
TapeAlert
TapeAlert flags can be set using the vtlcmd(1)
e.g. vtlcmd [index|library] TapeAlert [flags]
Where index is the message Q offset associated with the drive (or the
string 'library').
e.g. To set flag 14h (Clean now) the 20th bit (14h) needs to be set:
Pull out the binary to decimal calculator
1000 0000 0000 0000 0000 (20 bits) => hex => 80000
# vtlcmd 1 TapeAlert 80000
A listing of TapeAlert flags can be found at t10 home page for SSC
devices Annex A.
Examples
mhvtl is registered as HBA #3:
# lsscsi -g
[0:0:0:0] disk MAXTOR ATLAS10K4_36SCA DFM0 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x3 SCSI BP 1.1 - /dev/sg1
[2:0:0:0] disk SEAGATE ST336607FC 0003 /dev/sdb /dev/sg2
[2:0:1:0] disk SEAGATE ST336607FC 0003 /dev/sdc /dev/sg3
[2:0:2:0] mediumx ATL 1500 6.0 - /dev/sg4
[2:0:2:1] tape QUANTUM SuperDLT1 2323 /dev/st0 /dev/sg5
[2:0:2:2] tape QUANTUM SuperDLT1 2323 /dev/st1 /dev/sg6
[2:0:2:3] process ATTO 310-LV 1.42 - /dev/sg7
[3:0:0:0] mediumx STK L700 vtl0 - /dev/sg8
[3:0:0:1] tape SONY SDX-900V 5400 /dev/st2 /dev/sg9
[3:0:0:2] tape SONY SDX-900V 5400 /dev/st3 /dev/sg10
[3:0:0:3] tape QUANTUM SDLT600 5400 /dev/st4 /dev/sg11
[3:0:0:4] tape QUANTUM SDLT600 5400 /dev/st5 /dev/sg12
[3:0:0:5] tape QUANTUM SDLT600 5400 /dev/st6 /dev/sg13
[3:0:0:6] tape IBM ULT3580-TD3 5400 /dev/st7 /dev/sg14
[3:0:0:7] tape IBM ULT3580-TD3 5400 /dev/st8 /dev/sg15
[3:0:0:8] tape IBM ULT3580-TD3 5400 /dev/st9 /dev/sg16
Long term goals
Eventually, the functionality will be moved across to the SCSI Target
Framework (stgt) http://stgt.berlios.de/
The stgt is a very nicely designed SCSI taret framwork which can
support all target types along with multiple SCSI transports in a very
structured way.
However, until time permits (and my c coding standards improve),
features and improvements on this code base will continue.
Howto get help
Somebody has put up a discussion board (and has lots of great
documentation on working with different backup software)
http://mhvtl.nimsa.us/
Howto help out
There are many way of providing assistance
Testing with backup software X and reporting success/failure. If
possible, test any patches that may result of testing.
If you are willing, patches are very welcome.
Please read the Linux Documentation/CodingStyle
Note: I only discovered this documentation guide recently, so patches
to update old code style to the CodingStyle are also welcome.
Please run any patches thru the Linux 'mhvtl-<version>/scripts/
checkpatch.pl' for coding style sanity check before sending.
Download
Stable vers:
mhvtl-0.16-13.src.rpm - Source RPM
mhvtl-0.16-13.i586.rpm - 32bit RPM
mhvtl-0.16-13.x86_64.rpm - 64bit RPM
mhvtl-2010-05-09.tgz - Source (tar ball) - (Builds mhvtl-0.16-12)
Change log:
mhvtl-2010-05-09 / mhvtl-0.16-13
*** Happy Mothers Day ***
- Allow media sizes larger than 2G on 32bit platforms
- Implement STK vendor specific 'LOAD DISPLAY' (Logs info via syslog)
- Fix core dump on invalid NAA strings
- Support VENDOR ID strings with embedded spaces
- Return LOG SENSE 'Tape Capacity' in OEM format (Bytes/KB/MB values)
- Limit SMC op codes when in offline state
mhvtl-2010-03-10 / mhvtl-0.16-12
- Introduce multiple library support. Device.conf format had to
change.
- Introduce script to convert old device.conf to new format
- Test media loaded on 'REZERO' op code
- Fix MAM struct size to be 1024 bytes
- Fix SPR RESERVATION bug.
- Dynamic change of device queue depth (kernel module change)
- Improve SCSI OP code logging
mhvtl-2009-12-16 / mhvtl-0.16-11
New and improved Compression: entry in device.conf (Thanks Kevan)
- Compression: factor [1-9] enabled [0|1]
Factor 1 == Fastest compression -> 9 == Best compression (zlib)
- Add tests to only allow LTO media to be mounted in LTO drives. If
media is 'unknown', then the mount will succeed.
mhvtl-2009-12-02 / mhvtl-0.16-10
Fix silent data corruption (Thanks Kevan)
Increase max LUNs from 7 to 32
mhvtl-2009-11-29 / mhvtl-0.16-9
Check string length of definitions within device.conf
Compression now set/resetable via MODE SELECT. (Thanks Kevan)
- Default compression set in device.conf
Compression: X (where X is a value from 0 [off], 1 [ Fastest
compression], => 9 [Best compression]
Default library INQUIRY string changed from STK/L700 to SPECTRA/PYTHON
Fix kernel compile on Ubuntu 9.10 (2.6.31+)
Fix max block size on x86_64 (Thanks Kevan)
SPIN/SPOUT: Return correct list of supported pages.
mhvtl-2009-11-10 / mhvtl-0.16-8
Add ability to compress data written to backing storage file.
Overhaul READ ELEMENT STATUS. Thanks Kevan Rehm.
Makefile overhaul.
- DESTDIR understood.
- Ability to define default location for media files & config file
parent directories
- Ability to define default user/group account for mhvtl daemons.
mhvtl-2009-10-09 / mhvtl-0.16-7
Fix READ ELEMENT STATUS for All element types. Many thanks to Bernardo
Clavijo for the report and fix. This bug caused TSM to core dump.
Improve (make more life like) Media Access Port access.
Added 'map open' and 'map close' options to the vtlcmd syntax.
If the MAP is open, then the 'Medium Transport Element' fails to move
media to/from a MAP port.
If the MAP is closed, then the user (via vtlcmd) can not empty MAP or
place new media into the MAP.
mhvtl-2009-10-01 / mhvtl-0.16-6
Fix test comparison within MHVTL_DBG macro
Fix output of the dump_tape utility
mhvtl-2009-09-15 / mhvtl-0.16-4
Bug fix only - no functionality changes. No need to rebuild kernel
module from 0.16-3
Fix up VERBOSE testing with MHVTL_DBG macro
Clean up more logging message(s)
mhvtl-2009-09-02 / mhvtl-0.16-3
Increased max block size from 256k to kernel default (kernel module
needs to be rebuilt)
Re-worked logging to use a single macro 'MHVTL_DBG' which will
either use syslog in verbose mode or printf if started in debug
mode.
mhvtl-2009-08-15 / mhvtl-0.16-2
Changed/updated load MAP() functionality.
Previous implementation re-read config file and parsed the MAP
slots.
This is 'interactive' using the vtlcmd command.
e.g. vtlcmd library load map <BARCODE>
Corrected NAA field in VPD page 0x83. This was a hard coded string.
Now reads entry from /etc/mhvtl/device.conf
' NAA: 11:22:33:44:ab:cd:ef:03' (8 octet value)
Remove dependency on sg3_utils
mhvtl-2009-08-05 / mhvtl-0.16-1
- Fix kernel module oops with logical unit removal (Many thanks to
Jean-François CHALARD for the fix)
- Fill in loadMap() routine within vtllibrary (Many thanks to Jean-
François CHALARD for the code)
- Fix compile issue on SLES 9
mhvtl-2009-06-09 / mhvtl-0.16-0:
- Renamed final pieces from 'vtl' to 'mhvtl' (e.g. /etc/mhvtl/ instead
of /etc/vtl)
- Almost everything is handled in user-space now. As daemons start,
the new logical unit is added to the pseduo hba driver. As the
daemon(s) are terminated, the logical unit is removed from the pseudo
hba driver.
- kernel module 'oops' if unloaded and re-loaded. It is stable
otherwise. Root cause unknown at this point in time.
Development version:
Download :
mhvtl-2010-09-01.tgz (builds mhvtl-0.18-10)
mhvtl-0.18-10.x86_64.rpm
mhvtl-0.18-10.i586.rpm
mhvtl-0.18-10.src.rpm
Kevan Rehm has re-written the on-disk data structure.
(It's amazing how much effort can be rolled up in one simple
sentence).
Each 'tape' is now stored under its own subdirectory /opt/mhvtl/
<barcode>
A tape consists of three files.
- Data file => Container of data
- Index file => Contains an array of header structures (pointers into
the data file)
- Meta file => Contains the MAM structure, followed by a meta data
header, followed by a variable-length array of filemark block numbers.
mhvtl-2010-09-01.tgz / mhvtl-0.18-10:
- Fix TapeAlert
- Added more descriptive logging around SPACE OP code attempting to
track down problem with HP Dataprotector
- Added (undocumented) utility 'tapeexerciser' which is a simple
utility to test tape read/writes and positioning. WARNING: This will
overwrite your media..
- Fix 'is vtl running' logic in rc script. Failed test if 'vtl user'
is root (not a recommended config)
mhvtl-2010-07-09.tgz / mhvtl-0.18-9:
- Fix WORM handling - Reported by Albert Pauw
- Fix (finished) LTO5 drive & media handling introduced in 0.18-8
mhvtl-2010-06-23.tgz / mhvtl-0.18-8:
- Several Security Protocol IN updates - thanks Albert Pauw
* Return certificate data
* Correct length for 'KEY FORMATS'
* Correct length for SPIN SUPPORTED PAGES
- Fix kernel compile on RedHat AS4
- Media/drive matching now 'dynamic' and defined in device.conf
- Added man page for device.conf
- Fix media corruption when media is 'formatted'
- Add LTO5 & SDLT-S4 drive/media types
- Handle INQUIRY correctly after media change (return SAM_STAT_GOOD)
- Updated rc script so all devices created on Target & LUN. i.e. Don't
use
channel. Some application software has trouble if only the channel
is unique.
mhvtl-2010-05-08.tgz / mhvtl-0.18-7:
Fix: 32bit platforms - Allow media size to be greater than 2G
Fix: core dump on invalid defined NAA strings
Fix: Support SMC devices with embedded space chars in VENDOR ID
Implement STK Vendor unique LOAD DISPLAY SCSI op code. (Logs display
info via syslog)
mhvtl-2010-05-02.tgz / mhvtl-0.18-6:
Fix: vtlcmd command line parsing. Fix 'list map' option
Fix: import of media via MAP (off-by-one)
Fix: TapeCapacity LOG SENSE data.
Now returns bytes/KB/MB as per OEM (Thanks 'kamy')
Support VENDOR ID with embedded spaces. i.e. "HP Ultrium"
Honour MAP status: Fail attempts for robot to move media in CAP if CAP
is open.
Relax dependencies on /proc in favour of /sys (note: Work in progress)
mhvtl-2010-04-01.tgz / mhvtl-0.18-5:
Fix: vtlcmd command line parsing. Thanks Herbert Stadler
Fix: Silence local_irq_save(). Normally seen on fedora. Thanks Norm
Lunda
Fix: URLs to this homepage.
mhvtl-2010-03-05.tgz / mhvtl-0.18-4:
Fix: ELEMENT STATUS - found and fixed by Norm Lunde (many thanks)
- This was preventing TSM to work with mhvtl
Add queue depth() callback in kernel module (module date string
20100303)
Test for media mounted before returning status on 'REWIND' op code.
Fix typos in rc script
mhvtl-2010-01-28.tgz / mhvtl-0.18-2...rpm:
The ability to support multiple libraries on the one host !! Thanks
once again to Kevan.
Note: There is no conversion of existing configuration file(s) to the
new device.conf/library_contents format. Please remove all files
under /etc/mhvtl/ before installing mhvtl-2010-01-28 (mhvtl-0.18-2)
What hardware does it work with:
Tested with Pentium x86 (32bit) and Pentium / AMD x86_64
New: IBM Z-Series. To quote:
"We just fired up the VTL on our z series IBM mainframe. Works like a
charm."
What software does it work with:
Symantec NetBackup.
5.1 (Recommend 5.1MP4 minimum)
6.0
6.5.x
7.0
Supports SCSI Persistent Reservation
Encryption via KMS (NetBackup 6.5.2 and later)
Note: All mhvtl code development is tested using NetBackup
Last set of testing (0.18-8 & -9) with NetBackup 7.0, I had to revert
the 'STK/L700' to 'SPECTRA/PYTHON' change made back in Nov 2009. It's
just a manual edit of /etc/mhvtl/device.conf
The problem is that NetBackup 7.0 fails to extract the device serial
number if SPECTRA/PYTHON is used and hence the automagic device
discovery & configuration fails to place the drives within the robot.
Symantec BackupExec.
For the extraordinary writeup : http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37
Includes how to setup the scst (http://mhvtl.nimsa.us/viewtopic.php?
f=23&t=32) to present the vtl via iSCSI
I've receive updates from a anonymous contributor known as 'nia':
Successful Testing of Solaris running mhvtl via iSCSI:
http://mhvtl.nimsa.us/viewtopic.php?f=23&t=47
Successful Testing of Sun NDMP working with mhvtl via iSCSI
http://mhvtl.nimsa.us/viewtopic.php?f=20&t=53
Successful Testing of Linux ZFS-fuse file-system with "dedup on" for /
opt/mhvtl http://mhvtl.nimsa.us/viewtopic.php?f=13&t=31
Successful Testing of HP Data Protector Linux media agent
http://mhvtl.nimsa.us/viewtopic.php?f=29&t=51
Successful Testing for Backup Exec 2010 on Windows Server 2008 R2.
Tested version of mhvtl is 0.18.2 via iSCSI http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159
Successful Testing for NetBackup 7 Master on Windows 2008 R2 via iSCS
http://mhvtl.nimsa.us/viewtopic.php?f=24&t=37#p159
EMC/Legato NetWorker
Is no longer causing headaches !
Don't use 'STK/L700' Use 'SPECTRA/PYTHON' instead.
Install mhvtl-2009-12-16.tgz (mhvtl-0.16-11).
Build kernel module from source
# rpm -Uvh mhvtl-0.16-11.src.rpm
# cd /usr/src/packages/BUILD
# tar xvfz ../SOURCES/mhvtl-2009-12-16.tgz
# cd mhvtl-0.16/kernel
# make
# make install
Start the daemons
/etc/init.d/mhvtl start
Configure NetWorker as normal via jbconfig.
An ex colleague who has some very useful/helpful tips with NetWorker
(If your interested in NetWorker, his blogs & tips are well worth
following)
http://nsrd.info/blog/2009/07/13/carry-a-jukebox-with-you-if-youre-using-linux/
Also a follow up article
http://nsrd.info/blog/2009/11/14/networker-and-linuxvtl-redux/
Bakbone Netvault
(2010-01-09) Thanks to Nick Couchman who has reported NetVault 8.x and
mhvtl-2009-12-16 (mhvtl-0.16-11) do indeed work together.
To quote Nick "I have a NetVault 8.x installation here that I'm using,
and I just installed LinuxVTL2 and am successfully backing up data
from NetVault to LinuxVTL2. I'm not sure if any issues exist for
restores and the link, but I'll let you know if I run into anything.
All appears to be working fine at this point! FWIW I'm running my
NetVault server on RHEL5."
TSM
Thanks to Bernardo Clavijo, TSM now works with mhvtl - Need at least
0.18-4 release.
Bacular
A report from Beat Rubischon to advise Bacular works also.
Two changes were needed on Bacula's
bacula-sd.conf:
Device {
Name = Drive-1
Drive Index = 0
Media Type = LTO-4
Archive Device = /dev/nst0
AutomaticMount = yes;
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
AutoChanger = yes
Alert Command = "sh -c 'tapeinfo -f %c |grep TapeAlert|cat'"
# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No
}
Multiple tape drives are supported too.
New: 2010-05-21
Report by Albert Pauw.. Many thanks.
# Needed for mhvtl tape drives
Hardware End of Medium = No
Fast Forward Space File = No
I have used it this way for the last two months, but due to some
questions on the mhvtl forum I tried
bacula without these additions. Seems the latest versions of mhvtl
work out of the box with bacula,
so these additions are not needed anymore.
Getting Started
To install and get the package running:
1. Install both src.rpm & x86/x86_64 (depending on your CPU/OS version
- check using uname -p)
2. Build kernel module from source you just installed:
cd /usr/src/packages/BUILD
tar xvfz ../SOURCES/mhvtl-2009-12-16.tgz
cd mhvtl-0.16/kernel
make
make install
3. Start the package using the rc script.
/etc/init.d/mhvtl start
4. Check it's all running:
lsscsi -g
ps -ef|grep vtl (default with 5 daemons - 1 vtllibrary & 4 vtltape)
5. Configure your backup software ;)
mark_...@symantec.com
Reply all
Reply to author
Forward
0 new messages