Problems to mount s3 bucket after using aws cli sync

[Marcelo Terres]

Hi.

I was using my S3 bucket in standard region and I'm having a lot of problems with data.

So, I created a new S3 bucket in Oregon region, and, to avoid send all data again, I used aws cli sync to "copy" data from one bucket to another.

The problem now is that I can't mount this new s3 bucket in s3ql. Is there something that I can do to solve this problem?

This is the output after trying to mount it:

root@backup:~# mount.s3ql --cachedir /var/.s3ql --log /var/.s3ql/mount.log --allow-other --upstart --metadata-upload-interval 600 --authfile "/etc/authinfo2" "s3://mybackup" "/backups/s3qlmount/"

Using 4 upload threads.

Autodetected 4052 file descriptors available for cache entries

Uncaught top-level exception:

Traceback (most recent call last):

  File "/usr/local/bin/mount.s3ql", line 9, in <module>

    load_entry_point('s3ql==2.15', 'console_scripts', 'mount.s3ql')()

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/mount.py", line 129, in main

    (param, db) = get_metadata(backend, cachepath)

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/mount.py", line 377, in get_metadata

    param = backend.lookup('s3ql_metadata')

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/comprenc.py", line 69, in lookup

    meta_raw = self.backend.lookup(key)

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/common.py", line 107, in wrapped

    return method(*a, **kw)

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 276, in lookup

    return self._extractmeta(resp, key)

  File "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 715, in _extractmeta

    raise CorruptedObjectError('Invalid metadata format: %s' % format_)

s3ql.backends.common.CorruptedObjectError: Invalid metadata format: raw

Thanks in advance.

Regards,

Marcelo Terres

[Nikolaus Rath]

On Oct 21 2015, Marcelo Terres <mhte...@gmail.com> wrote:
> Hi.
>
> I was using my S3 bucket in standard region and I'm having a lot of
> problems with data.
>
> So, I created a new S3 bucket in Oregon region, and, to avoid send all data
> again, I used aws cli sync to "copy" data from one bucket to another.
>
> The problem now is that I can't mount this new s3 bucket in s3ql. Is there
> something that I can do to solve this problem?
>
> This is the output after trying to mount it:
>

[...]

> "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py",
> line 715, in _extractmeta
> raise CorruptedObjectError('Invalid metadata format: %s' % format_)
> s3ql.backends.common.CorruptedObjectError: Invalid metadata format:
> raw

Sounds like you copied only the objects, but not the associated
metadata. You can check that using the aws management console.


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[Marcelo Terres]

Thanks for your answer.

I have thousands of files in the bucket.

Metadata have an specific filename or extension, like .db?

Regards,
Marcelo H. Terres
mhte...@gmail.com
IM: mar...@jabber.mundoopensource.com.br
http://www.mundoopensource.com.br
http://twitter.com/mhterres

> --
> You received this message because you are subscribed to a topic in the Google Groups "s3ql" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/s3ql/dEyPBJsUhn0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to s3ql+uns...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Nikolaus Rath]

Hi Marcelo,

When replying to emails on this list, please do not put your reply
above the quoted text, and do not quote the entire message you're
answering to. This makes it unnecessarily hard for other readers to
understand the context of your email. Instead, please cut quoted parts
that are not relevant to your reply, and insert your responses right
after the points you're replying to (as I have done below). Thanks!

On Oct 22 2015, Marcelo Terres <mhte...@gmail.com> wrote:
> On Wed, Oct 21, 2015 at 5:17 PM, Nikolaus Rath <Niko...@rath.org> wrote:
>> On Oct 21 2015, Marcelo Terres <mhte...@gmail.com> wrote:
>>> Hi.
>>>
>>> I was using my S3 bucket in standard region and I'm having a lot of
>>> problems with data.
>>>
>>> So, I created a new S3 bucket in Oregon region, and, to avoid send all data
>>> again, I used aws cli sync to "copy" data from one bucket to another.
>>>
>>> The problem now is that I can't mount this new s3 bucket in s3ql. Is there
>>> something that I can do to solve this problem?
>>>
>>> This is the output after trying to mount it:
>>>
>> [...]
>>> "/usr/local/lib/python3.4/dist-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py",
>>> line 715, in _extractmeta
>>> raise CorruptedObjectError('Invalid metadata format: %s' % format_)
>>> s3ql.backends.common.CorruptedObjectError: Invalid metadata format:
>>> raw
>>
>> Sounds like you copied only the objects, but not the associated
>> metadata. You can check that using the aws management console.
>>
>

> I have thousands of files in the bucket.

A bucket doesn't contain files, it contains "storage objects". If you
think of them as files, you'll most likely make some incorrect
assumptions (like you do below).

> Metadata have an specific filename or extension, like .db?

Metadata is data associated with a storage object, it's not a separate
storage object. Open the AWS Management Console, and click on any one
storage object, then on the "Properties" button on the right, and then
expand the "Metadata" section. I bet that in the original bucket, it
will show some metadata, while in the new bucket there will be nothing.

[Marcelo Terres]

Hi.

> Hi Marcelo,
>
> When replying to emails on this list, please do not put your reply
> above the quoted text, and do not quote the entire message you're
> answering to. This makes it unnecessarily hard for other readers to
> understand the context of your email. Instead, please cut quoted parts
> that are not relevant to your reply, and insert your responses right
> after the points you're replying to (as I have done below). Thanks!

Sorry. It will not happen again.

>
> Metadata is data associated with a storage object, it's not a separate
> storage object. Open the AWS Management Console, and click on any one
> storage object, then on the "Properties" button on the right, and then
> expand the "Metadata" section. I bet that in the original bucket, it
> will show some metadata, while in the new bucket there will be nothing.

No, the same storage object in both buckets has metadata info. I'm
sending the screenshots.

Any other ideas?

Thanks.

[Nikolaus Rath]

On Oct 22 2015, Marcelo Terres <mhte...@gmail.com> wrote:

>> Metadata is data associated with a storage object, it's not a separate
>> storage object. Open the AWS Management Console, and click on any one
>> storage object, then on the "Properties" button on the right, and then
>> expand the "Metadata" section. I bet that in the original bucket, it
>> will show some metadata, while in the new bucket there will be nothing.
>
> No, the same storage object in both buckets has metadata info. I'm
> sending the screenshots.
>
> Any other ideas?

Are you sure that the two buckets are 100% identical? There is no
possitibility that e.g. the new one may have *additional* objects under
the same prefix (that were already present before you copied the data)?

[Marcelo Terres]

Hi.

>
> Are you sure that the two buckets are 100% identical? There is no
> possitibility that e.g. the new one may have *additional* objects under
> the same prefix (that were already present before you copied the data)?
>
>
> Best,
> -Nikolaus

No, both buckets have the same number of objects. I listed it using aws cli.

Each storage object have exactly the same name and size. The only
things that changes are date and time.

[Nikolaus Rath]

On Oct 23 2015, Marcelo Terres <mhte...@gmail.com> wrote:
>> Are you sure that the two buckets are 100% identical? There is no
>> possitibility that e.g. the new one may have *additional* objects under
>> the same prefix (that were already present before you copied the data)?
>

> No, both buckets have the same number of objects. I listed it using aws cli.
>
> Each storage object have exactly the same name and size. The only
> things that changes are date and time.

I have no idea how this could possibly happen. Can you reproduce this
with a freshly created file system? Does it only happen if you copy from
one bucket to a different one, or also if you copy the files within the
same bucket (e.g. prepending them all with a different prefix).