bareos and snapshots on linux

[Stefan Klatt]

Hello,

I have a xfs volume on lvm.

Is it better to use xfs or lvm snapshots?
On Centos/REL 7 there is the tool ssm, can it be used for this (I didn't
found information wich kind of snapshots - xfs or lvm - it generates)?
Or is it better to use xfsdump?
Which options I need to configure at bareos?
Has anybody scripts for this kind of backup?
Are there special tips for this?

A lot of questions, I know. Probably there are more, but I have no
experience with bareos and snapshots on Linux.

Regards

Stefan

[Jacob Keyes]

Hello,

did you ever get this to work? Because Filesystem Snapshots are on my agenda now too. We have a few clients with huge filesystems, where backups are becoming a bit inconsistent because they are taking so long. It's not a problem right now, but it might be in the future.

Regards,
Jacob

[Damiano Verzulli]

2017-08-07 23:30 GMT+02:00 Stefan Klatt <stefan...@cac-netzwerk.de>:

[...]

A lot of questions, I know. Probably there are more, but I have no
experience with bareos and snapshots on Linux.

[....]

 

Sorry but.... I think that what's written in the "Who Needs Bareos?" section of official bareos documentation is really wise:

".... if you are new to Unix systems or do not have offsetting experience with a sophisticated backup package, the Bareos project does not recommend using Bareos as it is much more difficult to setup and use than tar or dump..."

This is not to say that you can't use BareOS. What I want to say is that... you cannot expect to get proper (and detailed) answers to such an high number of questions, each one requiring _A_LOT_ of time to be properly answered.

You have two choices:

• save your money, by taking the time to complete your homework, by studying and trying and --only afterwards-- asking the mailing-list for (free) support on very specific issues;
• save your time, by paying for consultancy services, being able to heavily accelerate your learning-path.

Having said the above, let me add that it took to me _LOTS_ of time to:

• setup an LVM-based storage archive that....
• ...mantained 7 snapshot (one for each day of the week);
• ...and with the "last" snapshot being read by the bareos-fd (the bareos "client") to properly write backups to tapes

Getting snapshots from LVM LVs (so, at "block device" level) is quite straightforward. Rotating them (on a 7 day time-window) is also not so difficult (and easy scriptable).

Taking snapshot on the XFS level (so, at "filesystem level") seems not supported (I haven't deeply searched, but I bet there aren't builtin snapshot features in XFS)

In addition, as I've spent a couple of month dealing with ZFS (OpenZFS) I can _DEFINITELY_ assure you that snapshot-support that comes builtin with ZFS are _BY_FAR_ superior to related snapshot support of LVM (to give you an idea: it was not easy to let 7 snapshot to be concurrently "active" over an LVM LV; while I had _NO_ trouble in having more than 130 snapshot properly working over a single ZFS sub-pool). Unfortunately that means that ZFS performance are definitely "poor" as for my workload (a huge set of small files).

So, in the end: you posted _LOTS_ of questions, that I answered with _LOTS_ of "inputs". Now it's your turn: do your homework and.... get back here for specific, further, BareOS related, questions :-)

Cheers,
DV

[Damiano Verzulli]

2018-05-30 13:41 GMT+02:00 Jacob Keyes <rak...@gmail.com>:

[...]

Hello,
did you ever get this to work? Because Filesystem Snapshots are on my agenda now too. We have a few clients with huge filesystems, where backups are becoming a bit inconsistent because they are taking so long. It's not a problem right now, but it might be in the future.

Hi Jacob,

let me step-in in this discussion.

I'm a _LONG_ user of "snapshot" technologies. We used LVM snapshots for more than 7 year with _GREAT_ results. We used snapshots as a first-layer of "on-disk" backup: we have an LV that receive an "rsync" of data from several remote servers, on a daily basis. And we keep 7 snapshot of such LV, so to have 7 days of backup retention, easily "searchable" and "restorable" without the need to deal with "tape-restore". Afterwords, data goes to tapes, via BareOS usual approach.

Even tough we didn't took care about the filesystem, we never had consistency problems at EXT3 layer.

Several times we had issues due to the fact that LVM snapshots have a predefined "size" and... if you exceed such size, the snapshot become inconsistent.

Due to the increase of the amount of data, one year ago we started investigating for an "upgrade". We ended switching from LVM snapshots to OpenZFS ones. Snapshot support for ZFS is simply amazing: they can be created much quickier w.r.t. LVM (<5 second vs. >30) and can be chained limitless (we currently have a daily snapshot from Nov. 30 2017, with no issue!!!!).

Unfortunately the performance price that we pay for such a complexity is really important. So important that we decided to _NOT_ use the ZFS backend as the source of our FD (as it limited our backup to no more than few tens MB per second). By moving to something different (a common MDADM/DM stripe with some "tuning") we are able to have an FD continuosly reading from disk around 110/120 MB/s.

If you need further details, don't hesitate to ask (even tough I wonder if these topics are off-topic, on this ML).

In the end: snapshots are definitely production-ready, both for LVM and, expecially, ZFS. I really suggest you in "investigating" their adoption in your specific workload, even tough there could be performance-issues that you will need to address.

Cheers,

DV

 

[Stefan Klatt]

Hello Jacob,

sorry for late reply...

Yes, it's easier I thought.
ssm with lvm (I use xfs, ext4 works too) handles it really fine.
You need only enough room for the lvm snapshot and if needed a exclusion for the original mount point to backup only the snapshot.

Regards,

Stefan

--
CaC, Computer and Communication
Inhaber Stefan Klatt
End-2-End Senior Security Consultant
CISSP / CISM / ISO27001 Lead Implementer / TOGAF9 /
VdS-anerkannter Berater für Cyber-Security
Badges: https://www.youracclaim.com/users/stefan-klatt
Triftstrasse 9
60528 Frankfurt
Germany
USt-IdNr.: DE260461592

Tel.: +49-(0)172-6807809
Tel.: +49-(0)69-67808-900
Fax: +49-(0)69-67808-837
Email: stefan...@cac-netzwerk.de
Profil: http://www.cac-netzwerk.de/profil

[diamaunt]

On Thursday, May 31, 2018 at 5:21:13 PM UTC-5, Damiano Verzulli wrote:
> BareOS.

If you want to play with capitalization, wouldn't BAReOS be more appropriate?

Backup And Recovery, Open Source.

Or maybe BaReOS

[Stefan Klatt]

Probably he thinks it's a complex backup Operating System?

:-)