bareos for archiving

[Otto Brandstaetter]

Hi,

I have been using bareos for more than 8 years or so as a backup solution for my home servers.

Now, I am thinking about a different thing.

I want to archive to tapes - meaning that I want to backup and delete stuff afterwards from the disks to save space. If needed, I can recover it from tapes.

Did anyone do this already? How would the approach be for that? I mean, does it make sense to move stuff to a "special location" prior to backup? Where? How to recover in case of emergency?

Best regards and thanks in advance,

Otto

[Spadajspadaj]

It is possible. Tape is just another medium and bareos is just another solution that stores data on the tape. There are some issues to tackle though.

Firstly, as with normal backup jobs - if you prune records from database you need to read tape contents manually. But that's nothing special, just worth remembering.

Secondly, and that's more important - usually if you want to archive the data you might want to keep more than one copy of a given file. And that's where things get tricky. If you have a static set of data to archive, you might just want to run several copies of a full backup job, store them offline and be done with it. That's fairly easy.

It's getting way more difficult if your data set grows and you want to make sure that each file is copied at least X times onto different media. You could do that with dynamic fileset contents and some clever scripting querying database to see which files have already been backed up and how many times. But it's nowhere near something standard and easy. But can be done. Did something like that some 8 years ago with bacula but my use case was fairly limited.

And - as with any archiving, not just limited to bareos - I assume that while your set of data to archive might be growing, meaning that you get new files, their contents don't change over time.

TL;DR - Bareos can be used in archiving process but it might not be straightforward.

MK

--
You received this message because you are subscribed to the Google Groups "bareos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/a605c134-f676-40fc-92c4-55a17e6a372an%40googlegroups.com.

[Brock Palen]

Similar to what other have said it could probably work, but you need though is volumes and settings where things never get purged.

A better option IMHO is LTFS https://github.com/LinearTapeFileSystem/ltfs/wiki/ltfs

This basically lets you fuse mount a tape and write to it like a disk you can eject. Downside you have no catalog of waht files are on what tape. Upside nothing gets pruned and when you mount the tape you can just ‘ls’ or ‘find’ your hearts content. This is popular in media where you often have single things like 4k raw that are huge so a whole tape might be ‘one thing’.

At my day job we use IBM Spectrum Archive which uses LTFS + DMAPI + GPFS.
LTFS does the open layout on the tape, DMAPI https://en.wikipedia.org/wiki/DMAPI supported on GPFS (a parallel filesystem)
Let us ’stub’ files. That is we wrote them to a tape, the inode has in an extended attribute a pointer to the file on tape. So you can ls in the directory and a file might say it’s 10GB, but if you use ‘du’ it says it’s 2MBytes. If you try to read it Specturm Archive triggered by a DMAPI event tells our pair of TS4500 libraries what tape to mount and it reads it back to disk.

In our case we have 26,000TB on tape (x2 replicated) front ended by a 2PB (x2 replicated) cache filesystem.

So in this case the filesystem is the database/catalog and automates getting files back. Effectively turns tape into a NAS.

IMHO use HSM systems like this for archive, use backup systems for recovery. Each can kinda do the other but it’s often not done that well.

If your small scale the open source LTFS + text file is probably enough.


Brock Palen
bro...@mlds-networks.com
www.mlds-networks.com
Websites, Linux, Hosting, Joomla, Consulting

> To view this discussion visit https://groups.google.com/d/msgid/bareos-users/0b75a0ba0766212f5ca37554512513d90d66ae1d.camel%40gmail.com.