failed to handle the OVA files using sparse option

[levindecaro]

Hi,

I'm using bareos 18.2, to backup bunch of ova files exported from RHEV, the ova file size is the actual used size of the image after export(thin image). However bareos still allocate the real size of image during backup or restore. The sparse=yes options seems cannot handle ova files.

Anyone has experience to workaround that?

Thank you!

[Spadajspadaj]

Are you sure the OVA file is a sparse one? AFAIR, thin provisioning
means that the file size should sum up to already provisioned chunks of
data.

In other words, if inside the virtual machine you use 4G and have a 4G
file even though the maximum disk size is 30G you'd have a 4G file. But
it wouldn't be a sparse file.

If it were a sparse file, the filesystem would report a file size of 30G
but with only 4G of actual contents.


I hope I'm not making this overly confusing :-)

[levindecaro]

It is a sparse file i believe, by confirm that

[root@server export-domain]# du -hs VM1.ova
20G VM1.ova <--- actual size on disk

[root@server export-domain]# ls -l VM1.ova
-rw-------. 1 root root 179142214656 Jul 13 02:56 VM1.ova <--- real file size

After backup on bareos, it shown 167GB is backed up

After restore

[root@server export-domain]# ls -l ../restored/VM1.ova

-rw-------. 1 root root 177794008064 Jul 12 15:07 ../restored/VM1.ova

[root@mgnt21 export-domain]# du ../restored/VM1.ova

166G ../restored/VM1.ova <--- reverted to full allocated file.

[Spadajspadaj]

The question that comes to mint is "how sparse is the sparse file". I
know it sounds a bit strange but the bareos sparcity logic, as I see in
the docs is that it checks whether a 32k block is made entirely of
zeros. So if your sparse file doesn't have continuous 32k blocks
(properly aligned I suppose) it won't get treated as sparse.

[Andreas Rogge]

Am 13.07.19 um 08:19 schrieb levindecaro:

> It is a sparse file i believe, by confirm that
>
> [root@server export-domain]# du -hs VM1.ova
> 20G VM1.ova <--- actual size on disk
>
> [root@server export-domain]# ls -l VM1.ova
> -rw-------. 1 root root 179142214656 Jul 13 02:56 VM1.ova <--- real file size
>
> After backup on bareos, it shown 167GB is backed up

That is correct. Even if bytes have been skipped on backup still the
whole size of the file is recorded.

> After restore
>
> [root@server export-domain]# ls -l ../restored/VM1.ova
>
> -rw-------. 1 root root 177794008064 Jul 12 15:07 ../restored/VM1.ova
>
> [root@mgnt21 export-domain]# du ../restored/VM1.ova
>
> 166G ../restored/VM1.ova <--- reverted to full allocated file.

That's also correct. Bareos does not punch holes in files during restore.

The problem is the logic for sparse files: Bareos does detect contiguous
streams of zeroes, but it doesn't check whether the region was sparse or
not (I'm not aware of a portable way to do so right now).
So we don't know what regions of a backed up file are sparse and
therefore cannot punch the correct holes into the file during restore.

Sorry.

Best Regards,
Andreas
--
Andreas Rogge andrea...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221-630693-86
http://www.bareos.com

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Komplementär: Bareos Verwaltungs-GmbH
Geschäftsführer: S. Dühr, M. Außendorf, J. Steffens, Philipp Storz

[Spadajspadaj]

On 15.07.2019 09:05, Andreas Rogge wrote:

[root@server export-domain]# ls -l ../restored/VM1.ova

-rw-------. 1 root root 177794008064 Jul 12 15:07 ../restored/VM1.ova

[root@mgnt21 export-domain]# du ../restored/VM1.ova

166G	../restored/VM1.ova <--- reverted to full allocated file.

That's also correct. Bareos does not punch holes in files during restore.

The problem is the logic for sparse files: Bareos does detect contiguous
streams of zeroes, but it doesn't check whether the region was sparse or
not (I'm not aware of a portable way to do so right now).
So we don't know what regions of a backed up file are sparse and
therefore cannot punch the correct holes into the file during restore.

The docs are misleading then.

"By turning on the sparse option, Bareos will specifically look for empty space in the file, and any empty space will not be written to the Volume, nor will it be restored."  (https://docs.bareos.org/Configuration/Director.html#fileset-resource )

[Andreas Rogge]

> "By turning on the *sparse* option, Bareos will specifically look for

> empty space in the file, and any empty space will not be written to the
> Volume, nor will it be restored." 

> (https://docs.bareos.org/Configuration/Director.html#fileset-resource )Sorry. My bad.
The docs is correct. Bareos will puch a hole during restore if the
sparse option was set in the backup fileset.
The misleading behaviour was that this is done whether or not the
original file was actually sparse or just contained regions of 32K
contigous zeros.

[Spadajspadaj]

On 16.07.2019 08:30, Andreas Rogge wrote:
>> "By turning on the *sparse* option, Bareos will specifically look for
>> empty space in the file, and any empty space will not be written to the
>> Volume, nor will it be restored."
>> (https://docs.bareos.org/Configuration/Director.html#fileset-resource
>> )Sorry. My bad.
> The docs is correct. Bareos will puch a hole during restore if the
> sparse option was set in the backup fileset.
> The misleading behaviour was that this is done whether or not the
> original file was actually sparse or just contained regions of 32K
> contigous zeros.

You're getting me confused now :-)

First you wrote that fd doesn't write the file as sparse when restoring.
Now you say that fd "punches a hole" which - IMHO - suggests creating a
sparse file.

Maybe it's a matter of wording but in this context "punch a hole" I'd
interpret as creating a "holed" file. Which suggests strongly a sparse file.

I understand of course that sparsiness logic just looks for contiguous
regions of zeroes (do they need to be aligned to 32k boundary or do they
need to just be contiguous 32k zeros, BTW?).

But from what I understood from writing the sparse option description
(and I believe the original poster inferred the same) was that if the
sparse option is set, the fd looks for blocks of zeroes (regardless of
whether the file is indeed a sparse file on the underlying filesystem or
is just a plain fully-alocated file filled with zeros), writes the file
as sparse to the sd and then on restore creates a "holed" file without
those zeroed blocks. Which, for me, meant sparse.

It's a bit confusing since indeed the word "sparse" can be understood
differently on different levels so maybe it's worth to add a word or two
to the docs that the sparsiness of the backup is meant only in regard to
the method of storing the backup on backup media, not to the sparsiness
of the file itself in terms of the underlying filesystem?