Enabling compression on the existing setup

[Martin Prochazka]

Hi,

i am validating possibility to enable compression for space saving (currently 24 TB data). 

Version 25.03 Community, Always Incremental mode. Turning it on the bareos (storage daemon) side.

- what will happen if compression is enabled - will bareos still works with previous uncompressed volumes and just new volumes will be compressed? 

- opposite scenario, disabling compression if there will be issue - will bareos process combination plain + compressed + plain volumes?

Thanks.

[Bruno Friedmann (bruno-at-bareos)]

Compression is per block and if is detected, so if you do compression on the storage, it will work transparently (well more cpu / ram etc .. but you get it).

Also if a block is compressed, you can also send it compressed back to the FD which do the uncompression.

Just be sure to have SD and FD supporting the same compression algo.

Regards

[Martin Prochazka]

Because i probably sent message somewhere...Repeated post.

So i configured compression on the filesets for our Always Incremental scenario. Daily backups (based on the comparision between uncompressed and compressed file content) are compressed. But Consolidate job doesn't have Fileset variable, so it looks uncompressed. And continually, longterm (VirtualFull) volumes with Fileset configured seems uncompressed too. Any idea, where is something missing?

Thanks.

[Sebastian Sura]

Hello Martin,

if you configure compression in the fileset, then this means that the **filedaemon** will compress data it sends to the storage.
The storage itself does not know about this option and will neither compress nor decompress it on its own, unless you have autoinflation / autodeflation set up.

In your case this means that new data that is sent from the filedaemon will indeed be stored compressed, but old data stays uncompressed.
This is the same for consolidation: files that are contained in newer (incremental) backups will be compressed inside the new virtual full;
files that come from the old virtual full that were not compressed, stay uncompressed in the new virtual full.

If you wish your virtual full to be fully composed of compressed data, then the easiest way to achieve this is to run a new full backup on the client.
As far as I know its also possible to do this without requiring a new backup, but its a bit complicated.  You would have to define a new storage with autodeflation (but not autoinflation) setup, and copy/migrate the full job to the that new storage/device, and then copy them back; but I have not tested this approach.

It is also possible to test if your volume contains compressed data:
# bscan --list-records -u <user> storage/Full-0001
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=1 Stream=1 len=165
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=1 Stream=2 len=8626
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=1 Stream=1998 len=81
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=1 Stream=40 len=16
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=2 Stream=1 len=166
bscan: stored/bscan.cc:503-0 Record: SessId=1 SessTim=1781525444 FileIndex=2 Stream=2 len=23587
...
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=33 Stream=1 len=163
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=33 Stream=29 len=67
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=33 Stream=1998 len=81
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=33 Stream=40 len=16
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=34 Stream=1 len=192
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=34 Stream=29 len=72
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=34 Stream=1998 len=81
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=34 Stream=40 len=16
bscan: stored/bscan.cc:503-0 Record: SessId=2 SessTim=1781527561 FileIndex=35 Stream=1 len=180

If you see Stream=2, then the data for that record (basically a unit of data that the storage daemon uses) is uncompressed; if you see Stream=29, then the data is compressed.

Kind regards
Sebastian Sura

Am 15.06.26 um 10:22 schrieb Martin Prochazka:

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/2e9a0b6d-849d-42d8-ae73-77263e7df2e4n%40googlegroups.com.

-- 
 Sebastian Sura                  sebasti...@bareos.com
 Bareos GmbH & Co. KG            Phone: +49 221 630693-0
 https://www.bareos.com
 Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
 Komplementär: Bareos Verwaltungs-GmbH
 Geschäftsführer: Stephan Dühr, Jörg Steffens, Philipp Storz

[Martin Prochazka]

Thanks for enlightening.

So tldr;

short term solution: initiate full backup

long term solution: just let time work - note: consolidation imports old (expiring) backups first, so there is time delay before compressed files are in the consolidate/long term volumes

Thank you.