S3QL backup restore

[Sven Martin]

I have an Amazon S3 bucket mounted on an Ubuntu 14.04 server, which serves as my ownCloud server (EFSS). This bucket is configured as the data location for ownCloud. So all the users' files end up here.

What is the best way to:

- backup this mounted S3 bucket

- restore this backup in the event of data loss

Remark:
I have installed S3QL using these instructions https://bitbucket.org/nikratio/s3ql/wiki/installation_ubuntu. I do not see the /contrib folder anywhere.

Best regards, Sven

[Brian Pribis]

On Tuesday, October 7, 2014 7:41:09 AM UTC-4, Sven Martin wrote:

I have an Amazon S3 bucket mounted on an Ubuntu 14.04 server, which serves as my ownCloud server (EFSS). This bucket is configured as the data location for ownCloud. So all the users' files end up here.

What is the best way to:

- backup this mounted S3 bucket

- restore this backup in the event of data loss

Sven,

You could use duplicity to do this (https://help.ubuntu.com/community/DuplicityBackupHowto).   It is fairly trivial actually.  You could either backup to a local storage medium or to another s3 bucket mounted by s3ql.   Duplicity will allow you to do incremental restores of single files as well as entire backup sets. 

Just a note, if you use duplicity you'll need to be careful where you store its cache files.  I actually use a bucket for this.  They can get really big, especially if you are backing up a lot of data.

brian

[Nikolaus Rath]

On 10/07/2014 04:41 AM, Sven Martin wrote:
> I have an Amazon S3 bucket mounted on an Ubuntu 14.04 server, which
> serves as my ownCloud server (EFSS). This bucket is configured as the
> data location for ownCloud. So all the users' files end up here.
>
> What is the best way to:
> - backup this mounted S3 bucket

I'd suggest following the steps described at
http://www.rath.org/s3ql-docs/contrib.html#s3-backup-sh.

> - restore this backup in the event of data loss

You can just mount the file system and copy everything back using cp,
rsync or (for slightly better performance), pcp.py
(http://www.rath.org/s3ql-docs/contrib.html#pcp-py).

> Remark:
> I have installed S3QL using these instructions
> https://bitbucket.org/nikratio/s3ql/wiki/installation_ubuntu. I do not
> see the /contrib folder anywhere.

Did you look at /usr/share/doc/s3ql/README.Debian?

Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Sven Martin]

Thx!

I decided to use s3ql_backup.sh and backup the owncloud data folder to the same bucket (different folder). Since the high durability of Amazon S3 this should be ok, I hope.

My only concern is: "Will I allways be able to mount the s3ql file system? Can this get corrupted somehow?" During testing I ran into situations where I was unable to mount the s3ql file system sitting in an Amazon S3 bucket. Can't really remember the errors, and since I'm a newbie with Ubuntu & s3ql I guess it was probably due my lack of knowledge then anything else.

Sven

Op woensdag 8 oktober 2014 01:32:32 UTC+2 schreef Nikolaus Rath:

[Nikolaus Rath]

Sven Martin <spro...@gmail.com> writes:
> My only concern is: "Will I allways be able to mount the s3ql file system?
> Can this get corrupted somehow?"

I'm pretty sure there still a number of bugs in S3QL. And even if there
were none, there's always the danger of broken hardware on your computer
or the remote backend. So yeah, things can get corrupted.

That said, there a number of people using S3QL regularly without having
their file system corrupted.

[Sven Martin]

Hi Nikolaus,

The server I mounted a S3QL file system on is still in test. I installed owncloud and configured its data directory to be on a mounted S3 volume using S3QL. Rebooting automounts the volume. I was on holiday for a week and when I came back the system had no free disk space. It turned out the volume was not mounted and the mountpoint got filled up with owncloud data. I don't knwo what caused the volume to unmount. I deleted the data (user data, but it's test so no issue) to free up space, rebooted the server. Now I am unable to mount the S3 filesystem. It keeps saying "Mounting filesystem..." and does not return to the prompt. Running fsck.s3ql --force works but does not help.

Any tips & tricks?

Best regards, Sven

--
You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/D3jiaHrqLxs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nikolaus Rath]

Hi Sven,

When replying to emails on this list, please do not put your reply
above the quoted text, and do not quote the entire message you're
answering to. This makes it unnecessarily hard for other readers to
understand the context of your email. Instead, please cut quoted parts
that are not relevant to your reply, and insert your responses right
after the points you're replying to (as I have done below). Thanks!

Sven Martin <spro...@gmail.com> writes:
> Hi Nikolaus,
>
> The server I mounted a S3QL file system on is still in test. I installed
> owncloud and configured its data directory to be on a mounted S3 volume
> using S3QL. Rebooting automounts the volume. I was on holiday for a week
> and when I came back the system had no free disk space. It turned out the
> volume was not mounted and the mountpoint got filled up with owncloud data.
> I don't knwo what caused the volume to unmount. I deleted the data (user
> data, but it's test so no issue) to free up space, rebooted the server. Now
> I am unable to mount the S3 filesystem. It keeps saying "Mounting
> filesystem..." and does not return to the prompt. Running fsck.s3ql --force
> works but does not help.
>
> Any tips & tricks?

What S3QL version are you using? Note the following entries in the
Changelog:

2014-09-04, S3QL 2.11.1

* By popular demand, mount.s3ql is now able to daemonize again
and also does so by default.

2014-08-27, S3QL 2.11

* mount.s3ql no longer daemonizes on its own. With a modern init
system this should no longer be necessary, and when running
mount.s3ql from the command line the shell can be used to put the
process into background.


Best,
Nikolaus

PS: Please remember the first paragraph when replying.