Backup and restore metadata of s3ql

[Peter Auyeung]

Is there a way to backup and restore metadata of s3ql?

e.g for xfs xfs_dump and xfs_mdrestore.

I wonder if it is possible to restore metadata of a particular path of s3ql

Thank you

Peter

[Nikolaus Rath]

On Jun 29 2016, Peter Auyeung <pauy...@connexity.com> wrote:
> Is there a way to backup and restore metadata of s3ql?

You could copy/restore the s3ql_metadata object in your storage backend,
or the *.db file in your --cachedir.

> I wonder if it is possible to restore metadata of a particular path of
> s3ql

Not without writing code.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Peter Auyeung]

I did a copy and restore of the backend of s3ql on local storage and getting the following error:

# mount.s3ql local:///ntap4/restore/ /s3ql/restore/

Using 10 upload threads.

Autodetected 4034 file descriptors available for cache entries

Enter file system encryption passphrase: 

ERROR: Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/bin/mount.s3ql", line 9, in <module>

    load_entry_point('s3ql==2.18', 'console_scripts', 'mount.s3ql')()

  File "/usr/lib/s3ql/s3ql/mount.py", line 129, in main

    (param, db) = get_metadata(backend, cachepath)

  File "/usr/lib/s3ql/s3ql/mount.py", line 374, in get_metadata

    param = backend.lookup('s3ql_metadata')

  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 77, in lookup

    return self._verify_meta(key, meta_raw)[1]

  File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 137, in _verify_meta

    raise CorruptedObjectError('HMAC mismatch')

s3ql.backends.common.CorruptedObjectError: HMAC mismatch

[Nikolaus Rath]

Hi Peter,

A: Because it confuses the reader.
Q: Why?
A: No.
Q: Should I write my response above the quoted reply?

..so please quote properly, as I'm doing in the rest of this mail:

On Jun 29 2016, Peter Auyeung <pauy...@connexity.com> wrote:
> On Wednesday, June 29, 2016 at 11:53:06 AM UTC-7, Nikolaus Rath wrote:

>> On Jun 29 2016, Peter Auyeung <pauy...@connexity.com <javascript:>>

>> wrote:
>>> Is there a way to backup and restore metadata of s3ql?
>>
>> You could copy/restore the s3ql_metadata object in your storage backend,
>> or the *.db file in your --cachedir.
>

> I did a copy and restore of the backend of s3ql on local storage and
> getting the following error:
>
> # mount.s3ql local:///ntap4/restore/ /s3ql/restore/
> Using 10 upload threads.
> Autodetected 4034 file descriptors available for cache entries
> Enter file system encryption passphrase:
> ERROR: Uncaught top-level exception:
> Traceback (most recent call last):
> File "/usr/bin/mount.s3ql", line 9, in <module>
> load_entry_point('s3ql==2.18', 'console_scripts', 'mount.s3ql')()
> File "/usr/lib/s3ql/s3ql/mount.py", line 129, in main
> (param, db) = get_metadata(backend, cachepath)
> File "/usr/lib/s3ql/s3ql/mount.py", line 374, in get_metadata
> param = backend.lookup('s3ql_metadata')
> File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 77, in lookup
> return self._verify_meta(key, meta_raw)[1]
> File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 137, in _verify_meta
> raise CorruptedObjectError('HMAC mismatch')
> s3ql.backends.common.CorruptedObjectError: HMAC mismatch

Well.. you did something wrong. Maybe you copied one of the
s3ql_metadata_bak objects to s3ql_metadata? This will give you a
checksum error.

That said, I think you should explain what problem you are trying to
solve. I don't think what you're trying to do is the right solution.

[Peter Auyeung]

Hi Nikolaus

I am trying to backup a s3ql to google clould either on s3ql or not

I was trying to rsync s3ql locally to the one on google storage but that created too many engress traffic as for the rsync compare

I like looked up gsutil rsync but that doesnt keep permission

I then looking to just rclone the s3ql to google cloud

Please kindly advise
Thanks
Peter

>
> Best,
> -Nikolaus
>
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
> »Time flies like an arrow, fruit flies like a Banana.«
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/DBmekkt-Syk/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nikolaus Rath]

This doesn't make sense. If you're trying to backup to Google cloud,
what are you trying to do above? Clearly you're trying to mount
something from local storage.

> I was trying to rsync s3ql locally to the one on google storage

You can do that, but then you won't be able to mount the file system
using the Google Storage backend (the backends use different file
formats). contrib/clone_fs.py can do the conversion though.


Best,
-Nikolaus

PS: Please don't CC me on replies, I am reading the list.

[Peter Auyeung]

That would be s3ql on local storage
Trying to backup locally to s3ql and snapshots and replicate to google

>> I was trying to rsync s3ql locally to the one on google storage
>
> You can do that, but then you won't be able to mount the file system
> using the Google Storage backend (the backends use different file
> formats). contrib/clone_fs.py can do the conversion though.
>

Does clone_fs.py incrementally sync two s3ql?

I am trying to reduce the cloud put get traffic during incremental

Thanks

> Best,
> -Nikolaus
>
> PS: Please don't CC me on replies, I am reading the list.
>
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
> »Time flies like an arrow, fruit flies like a Banana.«
>

[Nikolaus Rath]

So why are you talking about metadata then? You should be replicating
the entire folder that you pass to the local backend.

Also, if you correctly backup and restore then by definition S3QL can't
even tell that anything happened. If after the restore S3QL doesn't work
anymore, then that's not a problem with S3QL but with your
backup/restore procedure.

>>> I was trying to rsync s3ql locally to the one on google storage
>>
>> You can do that, but then you won't be able to mount the file system
>> using the Google Storage backend (the backends use different file
>> formats). contrib/clone_fs.py can do the conversion though.
>>
> Does clone_fs.py incrementally sync two s3ql?

No.

> I am trying to reduce the cloud put get traffic during incremental

Then you should be working with two mounted S3QL file systems (one
using the local backend and one using the Google Storage backend) and do
the synchronization using e.g. rsync.


Best,
-Nikolaus

[Peter Auyeung]

You are correct, i was messed up with the db and param file. The restore is now working.

>>>> I was trying to rsync s3ql locally to the one on google storage
>>>
>>> You can do that, but then you won't be able to mount the file system
>>> using the Google Storage backend (the backends use different file
>>> formats). contrib/clone_fs.py can do the conversion though.
>> Does clone_fs.py incrementally sync two s3ql?
>
> No.
>
>> I am trying to reduce the cloud put get traffic during incremental
>
> Then you should be working with two mounted S3QL file systems (one
> using the local backend and one using the Google Storage backend) and do
> the synchronization using e.g. rsync.
>

I was using rsync. As I asked in a different thread. Due to the amount of small files in s3ql. The rsync generated lots of put get traffic during incremental.

>
> Best,
> -Nikolaus
>
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
> »Time flies like an arrow, fruit flies like a Banana.«
>

[Nikolaus Rath]

On Jul 02 2016, Peter Auyeung <pauy...@connexity.com> wrote:
>>> I am trying to reduce the cloud put get traffic during incremental
>>
>> Then you should be working with two mounted S3QL file systems (one
>> using the local backend and one using the Google Storage backend) and do
>> the synchronization using e.g. rsync.
>
> I was using rsync. As I asked in a different thread. Due to the amount
> of small files in s3ql. The rsync generated lots of put get traffic
> during incremental.

There's no way to avoid that. At the moment, S3QL uses at least one
object per stored file. As a workaround, you can store zip/tar the files
before putting them on the S3QL file system.

But again, this is the case no matter what backend you use and no matter
how you synchronize. I'm still at a loss what your actual
question/problem/intention is.