LTO-6 inconsistent write performance (from 12M/s to 153M/s)

[TDL]

Hi

I encounter very variable write speeds - example during one backup:

Writing spool 1: 25.38M/s (7:03)

Writing spool 2: 149.1M/s (1:12)

Writing spool 3: 153.3M/s (1:10)

Writing spool 4: 12.66M/s (14:08)

Writing spool 5: 120.6M/s (1:29)

Writing spool 6: 153.3M/s (1:10)

Writing spool 6: 44.37M/s (4:02)

I have an IBM LTO-6 drive - SAS.

I am using a tmpfs as spool disk with a spool size of 10G (so no disk involved)

The CPU is at 0% for most of the time (I see storage daemon taking 3% cpu for a few seconds then go back to sleep)

The max block size of the media is 1048576 (1M)

The min block size is set to 0 (Do I need to change it to 1M too? if yes, can I change it without problem?) -I do not believe as it has to write 10G, I suppose it will always fill the block to the max size.

I encounter this with all the tapes I have (so likely not a defective tape).

Data written to tape is encrypted so likely not (highly) compressible by the drive that could explain potentially burst of write speed when writing highly compressible data.

Any idea what could cause those random slow downs (and how to fix it ;-) )?

Thanks a lot!

Thierry

[Bruno Friedmann (bruno-at-bareos)]

A few remarks:

- You shouldn't forget tmpfs is fundamentally a (virtual) memory-based file system; its contents live in memory only, but since they’re swappable the kernel can store them in swap instead of physical memory if necessary. So what precaution you're using to ensure, there's no swap involved in that case?

- The cpu usage you're seen by the sd look like the few seconds needed by spool file attributes sent to the database.

- block size min 0 and normal set to 1M is good.

You didn't explain your numbers, is there all happening at the same time? or sequentially. 

Also you didn't explain from where the data come, network ? then what type of, speed, hardware, family used and so on.

Did you let interleaving of job occurs on the same tape device? 

You didn't precise what M/s is it right to assume MB/s? 

So what is the speed of despooling only memory stored data with only one job writing to the tape drive, is this constant job by job, during the whole size of the tape.

[TDL]

Thanks for your answer Bruno.

To start with your last question, if I am not wrong, the (very variable) figures I gave are the speed of despooling only from memory one job to the tape drive. It is happening for all my backups (at the beginning of the tape and at the end - so tape position does not appear to have a noticeable impact).

Here are some clarifications:

- This is a backup from local disks, that is spooled by chunks of 10GB in a 12GB tmpfs (I ensured that no swap is used - server is only running bareos and has 24G of RAM). So bottom line, if I am not misunderstadnding completely the spooling, Bareos prepares a file of 10GB then, send it to the tape in one go (avoiding "shoeshining")

- Only one backup job running at a time.

- The numbers are in MB/s and are the numbers of consequents spool files being written for the same backup (Sppoling data, then spool 1 is sent to tape at 25.38MB/s, then spooling again, then spool 2 is sent  to tape at 149.1 MB/s and so on)

I had the impression that attributes are spooled too and only sent to the DB at the end of the backup (https://docs.bareos.org/TasksAndConcepts/DataSpooling.html - "When data spooling is enabled, Bareos automatically turns on attribute spooling. In other words, it also spools the catalog entries to disk. [...] Attribute despooling occurs near the end of a job.").

Do you mean that I should change the min size to 1MB? What is the best way of doing that with tapes that are already in use? Any risk during restore?