File compression

[Udo Ullrich]

Dear group, dear development team,

thank you for helping others with this very amazing piece of software!

Running the community version (18.2.5), entirely on Linux System (Client, too).

I want to conserve disk space, because of that I would like to use file compression, however, no matter which algorithm I choose from (LZ4, GZIP, ...) the volume size does

not differ. Now I suspect that there is no file compression at all. How can I find out, whether, the file backup is done with compression? Are there any logs telling this?

In bconsole I do not get any error messages, the backup Jobs work very fine (result OK).

What is the usual effect (ratio, e. g. using gzip) when using file compression?

For clarification this is the configuration file:

---------------------

FileSet {
  Name = "BackupRoot"
  Description = "Backup all regular filesystems"
  Include {
    Options {
      signature = MD5
      fstype = ext4
      compression = gzip
    }
    File = /
  }
  Exclude {
    File = /var/lib/bareos
    File = /var/lib/bareos/storage
    File = /tmp
    File = /var/tmp
  }
}

---------------------

Could it be that the option "compression" is ignored? Could it be there is a syntax error? I am puzzled, in the documentation I see the options with and without capital letters, all in capital letter, and again with no capital letters, with spaces and with quotation, e. g. compression, Compression, gzip, GZIP, compression=gzip, compression = gzip, etc.

Thank you for your help,

Udo Ullrich

[Spadajspadaj]

The question is whether the job output shows Compression. A volume is a storage unit. It may be a fixed file, it may be a tape. We don't know your configuration here. I, for example, have fixed-size 40G file-based volumes so the volumes size don't change but the jobs can be bigger or smaller depending on a compression used.

Sample output from a job utilizing compression from my installation:

*list joblog jobid=2403
 2020-01-21 02:07:05 backup1-dir JobId 2403: Start Backup JobId 2403, Job=backup_srv2_MySQL.2020-01-21_01.00.00_11
 2020-01-21 02:07:05 backup1-dir JobId 2403: Connected Storage daemon at backup1:9103, encryption: PSK-AES256-CBC-SHA
 2020-01-21 02:07:05 backup1-dir JobId 2403: Using Device "vchanger-1-0" to write.
 2020-01-21 02:07:05 backup1-dir JobId 2403: Connected Client: srv2-fd at 172.16.2.193:9102, encryption: PSK-AES256-CBC-SHA
 2020-01-21 02:07:05 backup1-dir JobId 2403:  Handshake: Immediate TLS  2020-01-21 02:07:05 backup1-dir JobId 2403:  Encryption: PSK-AES256-CBC-SHA
 2020-01-21 02:07:08 srv2-fd JobId 2403: Extended attribute support is enabled
 2020-01-21 02:07:08 srv2-fd JobId 2403: ACL support is enabled
 2020-01-21 02:07:06 bareos-sd JobId 2403: Connected File Daemon at 172.16.2.193:9102, encryption: PSK-AES256-CBC-SHA
 2020-01-21 02:07:08 bareos-sd JobId 2403: Volume "vchanger-1_2_0076" previously written, moving to end of data.
 2020-01-21 02:07:08 bareos-sd JobId 2403: Ready to append to end of Volume "vchanger-1_2_0076" size=19859852960
 2020-01-21 02:07:08 srv2-fd JobId 2403: python-fd: Starting backup of /_mysqlbackups_/mysql.sql
 2020-01-21 02:07:09 srv2-fd JobId 2403: python-fd: Starting backup of /_mysqlbackups_/mail.sql
 2020-01-21 02:07:09 srv2-fd JobId 2403: python-fd: Starting backup of /_mysqlbackups_/gts.sql
 2020-01-21 02:07:19 srv2-fd JobId 2403: python-fd: Starting backup of /_mysqlbackups_/epsilone_rcube.sql
 2020-01-21 02:07:20 bareos-sd JobId 2403: Releasing device "vchanger-1-0" (/var/spool/vchanger/vchanger-1/0).
 2020-01-21 02:07:20 bareos-sd JobId 2403: Elapsed time=00:00:12, Transfer rate=1.065 M Bytes/second
 2020-01-21 02:07:20 backup1-dir JobId 2403: Insert of attributes batch table with 4 entries start
 2020-01-21 02:07:20 backup1-dir JobId 2403: Insert of attributes batch table done
 2020-01-21 02:07:20 backup1-dir JobId 2403: Bareos backup1-dir 19.2.4~rc1 (19Dec19):
  Build OS:               Linux-5.3.14-200.fc30.x86_64 redhat CentOS Linux release 7.7.1908 (Core)
  JobId:                  2403
  Job:                    backup_srv2_MySQL.2020-01-21_01.00.00_11
  Backup Level:           Incremental, since=2020-01-20 01:00:05
  Client:                 "srv2-fd" 18.2.5 (30Jan19) Linux-4.4.92-6.18-default,redhat,CentOS Linux release 7.6.1810 (Core) ,CentOS_7,x86_64
  FileSet:                "MySQL - all databases" 2019-04-10 01:00:00
  Pool:                   "Offsite-eSATA" (From Job resource)
  Catalog:                "MyCatalog" (From Client resource)
  Storage:                "vchanger-1-changer" (From Pool resource)
  Scheduled time:         21-Jan-2020 01:00:00
  Start time:             21-Jan-2020 02:07:08
  End time:               21-Jan-2020 02:07:20
  Elapsed time:           12 secs
  Priority:               10
  FD Files Written:       4
  SD Files Written:       4
  FD Bytes Written:       12,790,416 (12.79 MB)
  SD Bytes Written:       12,791,390 (12.79 MB)
  Rate:                   1065.9 KB/s
  Software Compression:   82.7 % (gzip)
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         vchanger-1_2_0076
  Volume Session Id:      35
  Volume Session Time:    1579171330
  Last Volume Bytes:      19,872,665,728 (19.87 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Bareos binary info:     pre-release version: Get official binaries and vendor support on bareos.com
  Termination:            Backup OK

Remember that compression takes place on FD on a per-file basis so to verify that a job is indeed compressed, apart from the compression: field of job log is to just create a file with a known given size (let's say - 1GB), fill it up with zeros and then create a job to back up just this one file with compression. If it does work as it should, you should see a job with a very low "FD bytes written" and "SD bytes written" values since empty files compress very well.

And about the compression ratio - it all depends on the entropy of the input data. It's impossible to know without making some assumptions regarding the input data to tell how much said data will compress. As you can see, my job has about 83% of compression ratio and that's pretty typical for text data. Other types of data may compress much worse or even grow a bit (already compressed data).

But most important thing is that the job size doesn't have to (unless you're creating a volume-per-job media) correspond to volume sizes.

Regards,

MK

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/7656aca0-d310-4a97-8508-3d8192435feb%40googlegroups.com.

[Udo Ullrich]

Dear MK,

your answer was very helpful. Now I do understand that compression works (at my site) and how to verify.

No questions left (issue resolved).

Martin