Deduplication Storage Daemon Plugin

[Sébastien Marchal]

Hi,

In a previous post, I indicated that I was working on a plugin for Bareos Storage Daemon. During my tests, I noted that bsdEventWriteRecordTranslation is not emitted during a Virtual Full Backup. Have you got a patch to resolv this problem ?

For information, I published a first alpha release of the plugin on github : https://github.com/Sherlock221B/bareos/blob/master/src/plugins/stored/dedup-sd.c
I did some tests (backup of 30Go and 400Go) and the first results are interesting.

The plugin allows to deduplicate the data using block of fixed size (64k) or of variable size (avg 8k).
Varaiable size need more cpu and memory, but gives a better result.

The plugin also allows to deduplicate each volume (only the volumes whose name begins with a prefix defines) or a group of volumes (all the data stored on a set of volumes are deduplicated and stored in the same DB).

For best results, it is preferable that a volume is used only for a job. In my case, the name of a volume is the name of the job to which I add the prefix for deduplication.

You will find in attach a small example of the settings I use.

Best regards.

[Philipp Storz]

Hello Sébastien,

Am 06.04.2017 um 12:03 schrieb Sébastien Marchal:
> Hi,
>
> In a previous post, I indicated that I was working on a plugin for Bareos Storage Daemon. During
> my tests, I noted that bsdEventWriteRecordTranslation is not emitted during a Virtual Full
> Backup. Have you got a patch to resolv this problem ?
>
> For information, I published a first alpha release of the plugin on github :
> https://github.com/Sherlock221B/bareos/blob/master/src/plugins/stored/dedup-sd.c I did some tests
> (backup of 30Go and 400Go) and the first results are interesting.

Nice. Thanks for your work.

> The plugin allows to deduplicate the data using block of fixed size (64k) or of variable size
> (avg 8k). Varaiable size need more cpu and memory, but gives a better result.
>
> The plugin also allows to deduplicate each volume (only the volumes whose name begins with a
> prefix defines) or a group of volumes (all the data stored on a set of volumes are deduplicated
> and stored in the same DB).
>
> For best results, it is preferable that a volume is used only for a job. In my case, the name of
> a volume is the name of the job to which I add the prefix for deduplication.
>
> You will find in attach a small example of the settings I use.

Thank you very much for your work. That all sounds very interesting.

Regarding your requirements for the bsdEventWriteRecordTranslation calls
during virtual backup, I think that we can work on that together.

We can either discuss this on the list or in direct mail.

best regards,

Philipp

--
Mit freundlichen Grüßen

Philipp Storz philip...@bareos.com
Bareos GmbH & Co. KG Phone: +49 221 63 06 93-92
http://www.bareos.com Fax: +49 221 63 06 93-10

Sitz der Gesellschaft: Köln | Amtsgericht Köln: HRA 29646
Geschäftsführer: Stephan Dühr, M. Außendorf,
J. Steffens, P. Storz

[kune...@gmail.com]

is there any new information about this SD-plugin?