Multithreading in bareos-fd compression

[d.car...@i2tic.com]

Hello,

Is there any compression method with Multithreading support?.

I want to backup an SQL file of +300GB, and the problem is that GZIP and LZ4HC at least are single threaded. With one core the compression takes about three hours and I want to know if there's any way to use the rest of cores to be faster.

The machine has 26 cores, and using 12 cores it takes less than 30m in compress with pigz (gzip multithread) and compression level 6.

Thanks!!

[Douglas K. Rand]

On 09/21/18 06:20, d.car...@i2tic.com wrote:
> Is there any compression method with Multithreading support?.
>
> I want to backup an SQL file of +300GB, and the problem is that GZIP and
> LZ4HC at least are single threaded.

You can hack it up by having a ClientRunBeforeJob script that uses pbzip2 (or
another tool) to compress the file before the backup. You may want a
ClientRunAfterJob script that then deletes the file after the backup.

We do something kinda like this, and we built a specific backup job just for
the thing we needed to backup. So we have the general backup for that system
that then specifically ignores the directory, and then another job comes along
and does just that one directory.

Checkout the Catalog backup in the Bareos docs, it has a lot of the same features.

I personally like that the Bareos FD only uses one core to do compression.
Yes, it is slower, but it controls the impact that backups have on systems.

[Daniel Carrasco]

Thanks for your response.

Yeah, that's what I thought for the SQL backup file on that machine, but I've a MsSQL data file that is backup using VSS because is in use, and that option is not valid.

I also think that don't use the full power is good to don't slowdown the other services, but also having an option to select how many cores to use is a good idea. For example, this server is the most of time at less than 50% of CPU, and at night is just waiting on less than 5%, so you can use the 40% at day and 90% at night without problem and will work at x10 and x22 the single core speed.

Greetings.

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/ihBUmKx5eAE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Browning]

Why not use a task to backup the database to the local drive first then the script to compress it before backing it up with bareos and then deleting the backup on the local drive.

You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.

To post to this group, send email to bareos...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Eric Browning
Systems Administrator
801-984-7623

Skaggs Catholic Center
Juan Diego Catholic High School
Saint John the Baptist Middle
Saint John the Baptist Elementary

Twitter: @SCCMrB

[Daniel Carrasco]

Hello,

Thanks!!, but I've already have it, and what I thought is exactly this, compress with pigz before the backup and copy the file to several sites (bareos, other server...).

My problem comes because I want to also backup the raw file in bareos to be able to recover the file directly without import a dump, because the dump takes too much to be recovered, and raw file less than 30m.

Greetings!

[Bruno Friedmann]

There's perhaps another tricks that could help but need testing
especially the restore and how mssql run afterward.

Create a fileset only for your database and use the sparse = yes attribute
Because most of those db files are just fill up with empty space.

check and see how much real space is used (seems 30m) with sparse activated
you will have to compress (perhaps even not) a very small part of those
300Go file

Work well with almost empty file for virtual machine, so could be also
a way with db files.

Once again, test, test, test before using it for production.

--

Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch
Bareos Partner, openSUSE Member, fsfe fellowship
GPG KEY : D5C9B751C4653227
irc: tigerfoot

openSUSE Tumbleweed
Linux 4.16.9-1-default x86_64 GNU/Linux, nvidia: 390.59
Qt: 5.10.0, KDE Frameworks: 5.46.0, Plasma: 5.12.5, kmail2 5.8.1

[Daniel Carrasco]

Finally I've tried LZ4 and the size is the double, but at least it takes only 35m to be completed and compression ratio is 80%.

I'll try to do some tests to see if backup is right restoring to a test server.

Thanks, and greetings!.

_________________________________________

      Daniel Carrasco Marín

      Ingeniería para la Innovación i2TIC, S.L.
      Tlf:  +34 911 12 32 84 Ext: 223
      www.i2tic.com
_________________________________________