Speed up spooling 153,61 MB/s - help needed!

[Gil Hausner]

Hi,

I am right now in a configuration phase of a Bareos system with an Autoloader + 2 LTO-8 drives (attached over SAS,hardware compression Enabled) using LTO-8 Type M tapes.

i made a d2d2t test using data spooling and tested the system with a backup set of 9.7TB/778,294 files.

I am wondering about the spooling speed - it took me 18 hours to spool this nearly 10 TB of files, 

during that time - while spooling - the bareos-client used only one core nearly 100% all the time an i hat a throughput of about 1,3-1,4 GBit/Sek aka 153,61 MB/s.

So - ist this the top speed i can accomplish?

I thought maybey the usage of only one Thread/Core on the client could be the thing that slows me down - if this ist true - how to use more than one core?

Is there something else that i could do to speed up?

Is maybe the MD5 calculation the bottleneck?

Thanks for any idea/help!

...and btw i got also this two warnings although the whole process finaly worked:

"bareos-sd JobId 95: Error: backends/generic_tape_device.cc:146 Unable to open device "lto8-0" (/dev/tape/by-id/scsi-35000e111c23a90b5-nst): ERR=No medium found" ?!?!

"bareos-sd JobId 95: 3997 Bad alert command: sh -c 'tapeinfo -f /dev/tape/by-id/scsi-35000e111c23a90b8 | grep TapeAlert': ERR=Child exited with code 1." ??!?!?

Specs:

*Bareos Server: Dual Xeon E5-2697 v2 (24Cores,48 Threads); 128GB Memory; SSD for boot, system, DB; 

Spooldrive: Hardware RAID-5  (writecache enabled) out of 12x12TB Seagate Exos x14, Intel 10 GBit Network conncetion

Ubuntu 18.04.2; Bareos 18.2.5; PostgreSQL version 10.10 

*Autoloader with 160 Slots, Barcode reader, filled with LTO-7 Tapes labeled and initialized als LTO-8 Type M, 2 IMB LTO-8 drives (attached over SAS, hardware compression Enabled)

*Testclient: Dual Xeon CPU E5-2603 v4 (12 Cores,12 Threads), 64GByre Memory, SSD boot, system; 

Datadrive to backup is on a hardware RAID 6 (writecache enabled) out of 12 SATA III WD 3,5" 7.2k (512e) drives. Intel 10 GBit Network conncetion 

Debian 9.7 Stretch

Installed the System, optimized PostgreSQL according to pgtune (https://pgtune.leopard.in.ua), labeling and initializing worked; 

Benchmarks shows that all harddrives are quiet fast anf should be way faster than the above speed!

The 10Gbit Connection brings with iperf 9.4 GBits/s

Testconfiguration:

/bareos-sd.d/

/autochanager/

autochanger-0.conf

BOF

Autochanger {

  Name = "neoxl80"

  Changer Device =  /dev/tape/by-id/scsi-35000e111c23a90b8

  Device = lto8-0, lto8-1

  Changer Command = "/usr/lib/bareos/scripts/mtx-changer %c %o %S %a %d"

}

EOF

/device/

lto8-0.conf

BOF

Device {

    Name = "lto8-0"

    DeviceType = Tape

    DriveIndex = 0

    ArchiveDevice = /dev/tape/by-id/scsi-35000e111c23a90b5-nst

    MediaType = LTO-8

    Check Labels = yes

    AutoChanger = yes                       # default: no

    AutomaticMount = yes                    # default: no

    Maximum File Size = 50GB                # default: 1000000000 (1GB)

    Always Open = yes

    Alert Command = "sh -c 'tapeinfo -f %c | grep TapeAlert'"

    Removable Media = yes                   # Media can be removed from the device - like tapes    

    Random Access = no

    Spool Directory = /120TB/bareosspool

    Maximum Spool Size = 1109951162777600   # 100TB 

}

EOF

lto8-1.conf

BOF

Device {

    Name = "lto8-1"

    DeviceType = Tape

    DriveIndex = 1

    ArchiveDevice = /dev/tape/by-id/scsi-35000e111c23a90bf-nst

    MediaType = LTO-8

    Check Labels = yes

    AutoChanger = yes                       # default: no

    AutomaticMount = yes                    # default: no

    Maximum File Size = 50GB                # default: 1000000000 (1GB)

    Always Open = yes

    Alert Command = "sh -c 'tapeinfo -f %c | grep TapeAlert'"

    Removable Media = yes                   # Media can be removed from the device - like tapes

    Random Access = no

    Spool Directory = /120TB/bareosspool

    Maximum Spool Size = 1109951162777600   # 100TB 

}

EOF

/bareos-dir.d/

/client/

demeter.conf

BOF

Client {

  Name = demeter

  Address = xxx.xxx.xxx.xxx

  Password = xxx

}

EOF

/director/

bareos-dir.conf

BOF

Director {                            

  Name = bareos-dir

  QueryFile = "/usr/lib/bareos/scripts/query.sql"

  Maximum Concurrent Jobs = 10

  Password = "xxxxxxxxxx"         # Console password

  Messages = Daemon

  Auditing = yes

  Optimize For Speed = yes

}

EOF

/fileset/

FSetDemeterSharedfolders.conf

BOF

FileSet {

  Name = "FSetDemeterSharedfolders"

  Description = "Backup all regular filesystems, determined by filesystem type."

  Include {

    Options {

      Signature = MD5 # calculate md5 checksum per file

      One FS = No     # change into other filessytems

      FS Type = btrfs

      FS Type = ext2  # filesystems of given types will be backed up

      FS Type = ext3  # others will be ignored

      FS Type = ext4

      FS Type = reiserfs

      FS Type = jfs

      FS Type = xfs

      FS Type = zfs

            # noatime - if supported by the OS, the read time won't be adapted

            # this would generate a bunch of writes for no reason on the client machine.

      noatime = yes

    }

    File = /sharedfolders # Datafolder on Demeter

  }

  # Things that usually have to be excluded

  # You have to exclude /var/lib/bareos/storage

  # on your bareos server

  Exclude {

    File = /var/lib/bareos

    File = /var/lib/bareos/storage

    File = /proc

    File = /tmp

    File = /var/tmp

    File = /.journal

    File = /.fsck

  }

}

EOF

/job/

DemeterTestJob.conf

BOF

Job {

  Name = "DemeterTestJob"

  Type = Backup

  Level = Full                

  Client = "demeter"

  FileSet = "FSetDemeterSharedfolders"

  Storage = "Tape"

  Pool = "Archiv2d-2"  #2days pool :)

  Messages = "Standard"

  Spool Data = yes

}

EOF

/pool/

Archiv2d-2.conf

BOF

Pool {

  Name = Archiv2d-2

  Pool Type = Backup  

  Recycle = yes                           # Bareos can automatically recycle Volumes

  AutoPrune = yes # Prune expired volumes

  Recycle Pool = Scratch

  Scratch Pool = Scratch

  Maximum Block Size = 1048576

  Volume Retention = 2 days

# Maximum Volume Bytes = 15 TB           # Limit Volume size to something reasonable

  Maximum Volumes = 100  

}

EOF

/storage/

Tape.conf

BOF

Storage {

  Name = Tape

  Address = lemgol                # N.B. Use a fully qualified name here (do not use "localhost" here).

  Password = "xxxxxxxxxxxxxx"

  Device = neoxl80

  Media Type = LTO-8

  Auto Changer = yes

}

EOF