Backup strategy for large dataset

[a.br...@mail.de]

Hello everybody,

I'm in the process of developing a regular backup-strategy and found that I need some assistance. Here are the parameters in short:

- 35TB of medical imaging data

- daily increment of 50-60GB

- one site,  10Gb/s Backbone

- Overland NEOs LTO7 Storageloader, 8-bay.

attached to

- dedicated backupserver with 20TB RAID6, will be enhanced as needed

I have already backed up all data on LTO7 tape (WORM, for legal reasons) as of Dec'19.

A 2nd generation, also LTO7 WORM,  is currently in progress (VERY! slow, ~12MB/s, different story). Tapes are/will be stored offsite.

After that I'm planning to do a 3rd generation on disk inhouse and amend the 1st and 2nd gen on tape so that I end up with three generations identical to a certain cutoff date.

Then, what to do next? How could a daily routine look like?

The radiologists are very concerned about their data and would like to see a daily tape change with the ejected tape being taken offsite in the evening. A 1-bay LTO8 drive could be purchased then, the daily tape change would be done by an apprentice or so...

So I thought about an Always-Incremental-B2D2T-strategy starting with the above cutoff day. But I still have too less experience and so I'm struggeling hard to develop a clear structure in my head - WHAT is WHEN being copied WHERE - let alone transform that into a working bareos configuration.

Do my thoughts appear reasonable up to that point?

BTW: can a daily tape change be realized at all, where you just push the button to eject the TUE-tape, insert the WED-tape, and so on without having to stop the storage-daemon in order to unlock the tape?

Thanks for helping me with the first steps to bring this under way.

Andreas

[Brock Palen]

You will have some complexity with the size of your data and the size of your loader. Unless your data compresses really well.
Does it have more than one tape drive? Your total loader capacity is 48 TBytes raw, and you need 2x your full size to do Consolidations or new Fulls or you have gaps in your protection.

If I’m reading this right you want an off site copy.

If that’s correct I would go about this two different ways,

* Get a much bigger loader with 2 drives
or
* Expand backups server raid6 to have Full + growth*Growth Window capacity

I would then use migrate+Archive jobs to make my off site and copy to tape.

In the first case you can avoid the extra migrate, just do an archive to a pool of tapes you eject.

Workflow case 1 Bigger Tape loader 2 or more tape drives.
* Spool to Disk
* AI-Consolidated Pool Tape
* AI-Incremental Pool Disk
* Offsite Pool Tape

Fulls and Consolidations backups go to AI-Consolidated Tape pool,
Your daily go to disk until they are consolidated into tape.

To create your off sites you can use a copy job of whatever full you want.
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#copy-jobs

I personally for offsite to avoid issues with Always Incremental jobs use an Archive job
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html#virtual-full-jobs

This avoids the offsite tapes being upgraded to the primary copy when the Consolidate job prunes the older jobs.

To do a VirtualFull archive job like this though you need enough tape for 2x the data otherwise your swapping tapes every 5.5 hours.
I would then use a script called at the end of the archive job to eject the volumes and make volumestatus=Used from that Offsite pool. Load new tapes label and put back in Offsite for the next archive.

To do AI jobs this way and the VirtualFull archive job you need 2 tape drives to read from one pool/tape and write to the other. It will be fast as you get LTO speeds. No second Full backup.



Workflow case 2 Bigger Disk in Backup Server
* Spool if you want to avoid failed jobs cluttering disk volumes, maybe skip.
* AI-Consolidated Pool Disk
* AI-Incremental Pool Disk
* Offsite Pool Tape

This would work similar, but your Fulls and Consolidations all happen on Disk. This now means your disk needs to be 2x Full + Incremental. As your Full or VirtualFull consolidation will read all the data and make a second copy before purging the old copy from the catalog. So you won’t need the disk space long, but you will need it.

As for Offsite I would do the archive job again, Make a VirtualFull mark it as archive to the Offsite pool in the Tape loader. Eject all 7-8 tapes and load new ones.

Assuming you don’t want to be doing a new full Offsite every time, you can do copy jobs from your pools but I have found Bareos to behave in an unintuitive way when it comes Consolidations and offline media Copies. You end up having to run the consolidate job twice (once primary copy, Copy job then becomes primary and is picked up in next consolidation, where it wants to read from that media now) When you really just want the Copy jobs to be purged in most cases along with the primary copy on a Consolidate.

So I hoped you wanted just a full offsite say 1/month.

Another option if you wanted is to setup a second SD with disk off site, and send just your incremental there as part of a Copy job. Sneakernet the tapes with Full less often.


In general Always Incremental with copies for off site needs some work unless you are ok having a full snapshot every so often with an Archive job. I really hope Bareos improves on this in the future.


Lastly if you do use the archive job. It’s it’s own stand alone job, As your using WORM and cannot reuse your tapes, you probably want to crank up their retention so you can easly restore files form any batch of tapes. I would also for all of these run an extra backup of your catalog and append it to the offsite copies so you can pull catalog data in the event everything is lost. If you are encrypting (you are right?) be sure to keep a second copy of all your keys, I keep mine in OnePassword. I also upload a dump of the catalog to a cloud storage provider (Secure as you will). Doing a bscan of 35 TBytes will take a while, so please please please keep extra copies of your catlog and keys.

Also do a lot of testing with the archive jobs. How to turn them back into backup jobs to make a new VirtualFull for the real job, etc. If your Full’s ever screw up, you will want to do this (what I do for road warriors where Fulls are not really possible) to avoid the long time to do a new full from the source. Putting an Archive job back into a backup job, running a manual virtual full of a job will suck the files defined in the archive job in (won’t if it’s an archive, thus the change to update job type=B) and rebuild a full, then do an incremental.


I personally do a version of case 2, but I use Migrate jobs to move the AI-Consolidated to a tape pool, but my jobs full’s are much smaller than yours. I also only do an offsite archive job once a month. Otherwise Incremental and AI-Consolidated on Disk, Migrate to tape pool purge AI-Consolidated jobs quickly. So when I consolidate it’s Tape + Disk —> Disk —> Migrate Tape —> Archive Tape

# copy job to long term tape
Job {
Name = "Migrate-To-Offsite-AI-Consolidated"
Client = myth-fd
Type = Migrate
Purge Migration Job = yes
Pool = AI-Consolidated
Level = Full
Next Pool = Offsite
Schedule = WeeklyCycleAfterBackup
Allow Duplicate Jobs = no
Priority = 4 #before catalog dump
Messages = Standard
Selection Type = PoolTime # 7 days
Spool Data = No
Selection Pattern = "."
RunAfterJob = "/usr/local/bin/prune.sh”
}

The script forces the volumes to truncate/prune after I move the jobs off.


Brock Palen
1 (989) 277-6075
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

> --
> You received this message because you are subscribed to the Google Groups "bareos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/64fe4cfa-6c87-4d28-aed9-9e30de285ee1n%40googlegroups.com.

[Oleg Volkov]

This system does not looks like suits any usual backup system. Your FULL is terrible and restore will be awful.

Use the storage ability. Make DR site and replicate the storage. Take daily snapshots.

If you have very dumb storage, you can use ZFS. It has space efficient snapshots and ability to replicate to another ZFS storage.

[Spadajspadaj]

Two things.

RAID (or any other replication) is not a backup solution!

Archiving is not backup.

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/515ef17b-be7d-426b-94db-706b394a907fo%40googlegroups.com.

[Spadajspadaj]

As I wrote earlier, this looks more like archiving plan, not a backup one (or a combination of backup and archiving). But more to the point - in case of backups you have to have a verification plan and periodical restore tests. In case of archiving you need to have a verification plan (i.e. every half a year you read each archive unit, check whether it's readable and if its checksum is correct; in case of more critical data you might want to even have some kind of functional tests like trying to read the data into appropriate software and check whether the data is still parseable and loadable; If any copy fails you'd want to re-create it from other existing copies).

On 07.08.2020 09:49, a.br...@mail.de wrote:

Thanks a lot, Brock, for your comprehensive post and also to the others. I haven't fully worked through your example cases yet, but it  will certainly help me to get my head around it all. Maybe it helps if I provide a few more details about how the data/images are organized:

I run a Linux based virtualization cluster on RAID6-hosts with Windows VMs.

The images are organised in windows-folders of 2TB size each, like "E:\img\img01\" to currently "E:\img\img17\".

Once a folder is full, it's contents will never change again. They're like archives that will be read from but no more written to.

So I thought I'd proceed like this:

1. Backup "img01" to "img17" to tape, store the tapes offsite.

2. Do this a second time and store the tapes offsite, seperate from the first generation.

3. Do this a third time to disk, for quick access if needed.

4. Make sure the catalog of at least 1. and 2. is in a very safe place.

5. Develop a daily backup strategy - starting with "img18".

As for  (1.) - (3.) I have created seperate Full-jobs for each imgXX-folder. (1.) has already completed successfully, (2.) is currently in progress.

I thought that once (1.) and (2.) are completed successfully I'm safe what "img01-17" is concerned and never have to consider these folders for backup again. Right or am I missing something?

What I'd like to discuss here is (5.) - under consideration of a few parameters:

- the daily increment of image data is roughly 50 GB. BTW: The images (DICOM, JPEG2000) don't compress at all :).

- for legal reasons we have to store the images on WORM-media. So I need a daily job that writes to tape.

- the doctors want best possible protection against fire, supernova, Borg-attack etc. They want a daily tape change routine with the latest WORM-tape taken offsite.

For the daily tape change I could buy another LTO drive. I can also expand my backup-server to fit above (3.) and the daily increment.

So, here's what I thought I need to develop:

- Backup the daily increment to disk.

- Backup the same daily increment to a WORM tape (in a new 1-slot drive) that is part of a "daily change" pool of tapes (MON-SAT or so...)

- Append the same daily increment to another WORM tape in the 8-slot loader. Once the tape is full, take it offsite and add a fresh tape in the loader.

If that strategy doesn't sound to weird I need to transfer this into a working bareos config.

Sorry if it all sounds confusing but for me it's still really, really complex.

Thanks

Andreas

To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/4aa25985-fd74-4e2b-bb71-95ed37154bc1n%40googlegroups.com.

[a.br...@mail.de]

Thanks a lot, Brock, for your comprehensive post and also to the others. I haven't fully worked through your example cases yet, but it  will certainly help me to get my head around it all. Maybe it helps if I provide a few more details about how the data/images are organized:

I run a Linux based virtualization cluster on RAID6-hosts with Windows VMs.

The images are organised in windows-folders of 2TB size each, like "E:\img\img01\" to currently "E:\img\img17\".

Once a folder is full, it's contents will never change again. They're like archives that will be read from but no more written to.

So I thought I'd proceed like this:

1. Backup "img01" to "img17" to tape, store the tapes offsite.

2. Do this a second time and store the tapes offsite, seperate from the first generation.

3. Do this a third time to disk, for quick access if needed.

4. Make sure the catalog of at least 1. and 2. is in a very safe place.

5. Develop a daily backup strategy - starting with "img18".

As for  (1.) - (3.) I have created seperate Full-jobs for each imgXX-folder. (1.) has already completed successfully, (2.) is currently in progress.

I thought that once (1.) and (2.) are completed successfully I'm safe what "img01-17" is concerned and never have to consider these folders for backup again. Right or am I missing something?

What I'd like to discuss here is (5.) - under consideration of a few parameters:

- the daily increment of image data is roughly 50 GB. BTW: The images (DICOM, JPEG2000) don't compress at all :).

- for legal reasons we have to store the images on WORM-media. So I need a daily job that writes to tape.

- the doctors want best possible protection against fire, supernova, Borg-attack etc. They want a daily tape change routine with the latest WORM-tape taken offsite.

For the daily tape change I could buy another LTO drive. I can also expand my backup-server to fit above (3.) and the daily increment.

So, here's what I thought I need to develop:

- Backup the daily increment to disk.

- Backup the same daily increment to a tape (in a new 1-slot drive) that is part of a "daily change" pool of tapes (MON-SAT or so...)

- Append the same daily increment to another WORM tape in the 8-slot loader. Once the tape is full, take it offsite and add a fresh tape in the loader.

If that strategy doesn't sound to weird I need to transfer this into a working bareos config.

Sorry if it all sounds confusing but for me it's still really, really complex.

Thanks

Andreas

[Brock Palen]

Yeah I don’t know any backup system that works the way you laid it out here. All I know of have to eventually roll up the backups into a consolidated version. This is done two ways.

1. Periodic Full backup (Classic method, pulling a full new copy of the data from the backup target, does not read back the old Full in the backup system)
2. Always Incremental (new method where it’s done on the backup server, made possible by accurate jobs, reads the old Full and applies incrementals like patch diffs overtop creating a single new job)

Think about it, if you didn’t merge your full + incrementals over time, to do a restore in 1 year you would need your full tapes, which have not been checked in say 1 year, plus 365 incremental jobs, applying one on top of the other till you get the current state.

The documentation for the Always Incremental has some really good plots showing the behavior.
https://docs.bareos.org/TasksAndConcepts/AlwaysIncrementalBackupScheme.html

What this means though is you have to have enough capacity to have
Old Full + Incrementals + New Full

Thus you always need capacity of 2x + Incrementals the data volume you are backing up.



Now if you do option 1. you have a big load on your network and systems you are taking Fulls say 1x/month, but you can in theory eject your volumes used for your Full backup, because you won’t read them back except on restore (and periodic testing that restores work).

You can also use classic Copy jobs without the wonky behavior I mentioned before because you have no Consolidate jobs.
https://docs.bareos.org/TasksAndConcepts/MigrationAndCopy.html#

Bareos doesn’t like having more than one Copy of a job in my experense, so in your case where you want 3 copies of your data, I would do:
Primary Job to Disk
Copy Jobs to Tape (Full and Incrementals)
Periodic VirtualFull Job to Tape (Different Pool than above)

You can then eject the tapes safely. In this setup if you have a Full + Incrementals on disk always for quick access, you in theory can run your Copy of your Full and your VirtualFull archive job, out of sync and you could get away with your single loader.

You will though still need 2x the disk as you will have that old full + new full overlapping for some period of time. You could have your disk volumes expire before your tape volumes, but again you have a gap that for a restore you would need to go get your tapes from off site load them in the loader etc. before you could do a restore of any file that only resdies on those tape volumes.


Ah Dicom, there is a project to replace our Pax at my day job, it’s no small job.



If you go with Option #2, you likely will have VirtualFulls created more ofthen and thus burn through a lot more WORM media, Though you could put off setting Always Incremental Max Full Age to 1 Month or something and other values so you have 1 month of incrementals, and once a month roll them up into a new full.

Doing this though with WORM you will have piles and piles of media. For 2 off sites copies of 1 Full a month and 5 Tapes/35TByte, you are looking at 120 Tapes/year.

Double check, can you take an archive snapshot once a month that’s WORM? You could do that with the Archive job with the VirtualFull, set retention time, and it’s a frozen snapshot of how everything looked at that time. But do your Backup Planning using traditional media you can recycle as jobs expire and are replaced with new ones.


Your idea of the second drive with the daily incremental, would probably work, then do the Copy job to the pool in the loader.

Still though you need to sort out how your going to get your periodic fresh “Here if a full of the current state of my system”


You could do the wonky setup I do, where I always consolidate back to disk, Migrate to tape, and expire on disk quickly to free space. Your Full in case #1 would read all the data form your servers to disk, in Consolidation in #2, you would read from WORM in the loader to disk, but then when you migrate from Disk to Tape the new full you won’t have enough space in your loader so you will have to eject the tapes and load new ones. That may be fine for you.

Brock Palen
1 (989) 277-6075
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

> To view this discussion on the web visit https://groups.google.com/d/msgid/bareos-users/414d9ef4-b729-43fb-abb2-1cd4cc5a2a7cn%40googlegroups.com.