Problems upgrading an old file system

[David Given]

I have an old S3 filesystem, last used June 2017 (according to the timestamps in the files in the bucket). I'm trying to upgrade this so it can be mounted with current S3QL versions. I still have the s3qlauth.txt file (timestamp October 2017).

If I use s3ql version 2.21, which looks about the right vintage for the timestamps in the bucket and is most likely the version I used back then because it's the one in Debian, then I get:

> s3ql.backends.s3c.S3Error: AllAccessDisabled: All access to this object has been disabled

If I use s3ql version 2.24, which looks about the right vintage for the timestamp in the s3qlauth.txt file, then it just spins on:

> Encountered BadDigestError (BadDigest: Meta MD5 for s3ql_passphrase does not match), retrying Backend.open_read (attempt 1)...

3.0 or any version fails with this message, which to me suggests a filesystem version mismatch:

> s3ql.backends.common.CorruptedObjectError: Object content does not match its key (s3ql_metadata vs s3ql_metadata_bak_10)

...but it does at least manage to read the metadata files from the bucket, which means that my credentials work.

Any suggestions how to get unstuck here? My suspicion is that Amazon's configuration changed somewhere, which is why 2.21 is failing. Is there any way to get the actual file system version from the metadata files?

--

┌─── http://www.cowlark.com ───
│ "I have always wished for my computer to be as easy to use as my
│ telephone; my wish has come true because I can no longer figure out
│ how to use my telephone." --- Bjarne Stroustrup

[Daniel Jagszent]

Hello David,
> I have an old S3 filesystem [...] I'm trying to upgrade this so it can
> be mounted with current S3QL versions.[...]

> If I use s3ql version 2.21, which looks about the right vintage for
> the timestamps in the bucket and is most likely the version I used
> back then because it's the one in Debian, then I get:
> > s3ql.backends.s3c.S3Error: AllAccessDisabled: All access to this
> object has been disabled

where do you get this error? (Full debug log, backtrace) The bucket you
are trying to access is locked down; see
https://stackoverflow.com/questions/53329023/allaccessdisabled-all-access-to-this-object-has-been-disabled-error-being-thr

You should be able to mount your file systems with S3QL versions up to
version 2.26. After version 2.26 the file system changed (REV 23 to REV
24; see
https://groups.google.com/forum/#!msg/s3ql/4TzCVIMkA4o/xDxoKWCDAgAJ;context-place=forum/s3ql
<https://groups.google.com/forum/#%21msg/s3ql/4TzCVIMkA4o/xDxoKWCDAgAJ;context-place=forum/s3ql>)
and you need to upgrade the file system (
https://www.rath.org/s3ql-docs/adm.html#upgrading-the-file-system )
before you can mount it.

The current version of S3QL still uses REV 24 of the file system ( see
https://github.com/s3ql/s3ql/blame/master/src/s3ql/__init__.py#L46 ) so
you should be able to just install version 3.3.2 and do a file system
upgrade before you first mount it (that's why you got the
CorruptedObjectError exceptions)

[David Given]

On Fri, 6 Mar 2020 at 01:43, Daniel Jagszent <dan...@jagszent.de> wrote:

Hello David,

[...] 

> If I use s3ql version 2.21, which looks about the right vintage for
> the timestamps in the bucket and is most likely the version I used
> back then because it's the one in Debian, then I get:
> > s3ql.backends.s3c.S3Error: AllAccessDisabled: All access to this
> object has been disabled
where do you get this error? (Full debug log, backtrace) The bucket you
are trying to access is locked down; see
https://stackoverflow.com/questions/53329023/allaccessdisabled-all-access-to-this-object-has-been-disabled-error-being-thr

Wordy log follows:

---snip---

$ sudo bin/s3qladm --debug --authfile /etc/s3qlauth.txt upgrade s3://eu-west-1/com.cowlark.pool/
2020-03-06 20:13:44.388 4736 DEBUG    MainThread s3ql.backends.common.get_ssl_context: Reading default CA certificates.
2020-03-06 20:13:44.389 4736 DEBUG    MainThread s3ql.backends.s3c._do_request: started with GET /com.cowlark.pool/s3ql_passphrase?None, qs=None
2020-03-06 20:13:44.389 4736 DEBUG    MainThread s3ql.backends.s3c._send_request: sending GET /com.cowlark.pool/s3ql_passphrase
2020-03-06 20:13:44.914 4736 ERROR    MainThread root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "bin/s3qladm", line 26, in <module>
    s3ql.adm.main(sys.argv[1:])
  File "/tmp/s3ql-release-2.21/src/s3ql/adm.py", line 90, in main
    with get_backend(options) as backend:
  File "/tmp/s3ql-release-2.21/src/s3ql/common.py", line 260, in get_backend
    getattr(options, 'compress', ('lzma', 2)), raw)()
  File "/tmp/s3ql-release-2.21/src/s3ql/common.py", line 340, in get_backend_factory
    backend.fetch('s3ql_passphrase')
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/common.py", line 354, in fetch
    return self.perform_read(do_read, key)
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/common.py", line 317, in perform_read
    fh = self.open_read(key)
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/common.py", line 108, in wrapped
    return method(*a, **kw)
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/s3c.py", line 329, in open_read
    resp = self._do_request('GET', '/%s%s' % (self.prefix, key))
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/s3c.py", line 537, in _do_request
    self._parse_error_response(resp)
  File "/tmp/s3ql-release-2.21/src/s3ql/backends/s3c.py", line 571, in _parse_error_response
    raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'), resp.headers)

s3ql.backends.s3c.S3Error: AllAccessDisabled: All access to this object has been disabled

---snip---

I'm not sure the bucket is locked down; later versions appear to read files from the bucket fine (but then fail with object format mismatches). Looking at the file in the Amazon S3 browser shows that the owner account has full permissions.

You should be able to mount your file systems with S3QL versions up to
version 2.26.

2.26 fails too, but differently. This is the one which looks like it's correctly reading the file from the bucket, but then failing to parse it.

$ sudo bin/s3qladm --debug --authfile /etc/s3qlauth.txt upgrade s3://eu-west-1/com.cowlark.pool/
2020-03-06 20:17:30.353 4973 DEBUG    MainThread s3ql.backends.common.get_ssl_context: Reading default CA certificates.
2020-03-06 20:17:30.353 4973 DEBUG    MainThread s3ql.backends.s3c._do_request: started with GET /s3ql_passphrase?None, qs=None
2020-03-06 20:17:30.353 4973 DEBUG    MainThread s3ql.backends.s3c._send_request: sending GET /com.cowlark.pool/s3ql_passphrase
2020-03-06 20:17:30.534 4973 WARNING  MainThread s3ql.backends.s3c._extractmeta: MD5 mismatch in metadata for s3ql_passphrase
2020-03-06 20:17:30.534 4973 DEBUG    MainThread s3ql.backends.common.wrapped: Average retry rate: 0.02 Hz
2020-03-06 20:17:30.534 4973 DEBUG    MainThread s3ql.backends.common.wrapped: Encountered BadDigestError (BadDigest: Meta MD5 for s3ql_passphrase does not match), retrying Backend.open_read (attempt 1)...

[...and repeat ad nauseam...]

Do S3 credentials ever expire?

[Nikolaus Rath]

On Mar 06 2020, David Given <d...@cowlark.com> wrote:
> s3ql.backends.s3c.S3Error: AllAccessDisabled: All access to this object has
> been disabled
> ---snip---
>
> I'm not sure the bucket is locked down; later versions appear to read files
> from the bucket fine (but then fail with object format mismatches).

S3QL switched from using v3 authentication to using v4 authentication
for Amazon S3 at some point. Maybe AWS no longer supports v3?


Best,
-Nikolaus

--
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

[David Given]

On Sat, 7 Mar 2020 at 13:54, Nikolaus Rath <Niko...@rath.org> wrote:

[...] 

> I'm not sure the bucket is locked down; later versions appear to read files
> from the bucket fine (but then fail with object format mismatches).

S3QL switched from using v3 authentication to using v4 authentication
for Amazon S3 at some point. Maybe AWS no longer supports v3?

I verified the credentials with both the aws tool and s3fs. I also tried creating brand new credentials, and they behave in precisely the same way. So I'm confident it's not a credential problem.

s3fs was very unhappy about mounting a bucket with dots in the name. I know s3ql used to have this problem, but I thought it'd been fixed. If I rename my bucket, will s3ql still mount it, or are there internal references to the bucket name in the metadata?

 

[Nikolaus Rath]

On Mar 08 2020, David Given <d...@cowlark.com> wrote:
> On Sat, 7 Mar 2020 at 13:54, Nikolaus Rath <Niko...@rath.org> wrote:
> [...]
>
>> > I'm not sure the bucket is locked down; later versions appear to read
>> files
>> > from the bucket fine (but then fail with object format mismatches).
>>
>> S3QL switched from using v3 authentication to using v4 authentication
>> for Amazon S3 at some point. Maybe AWS no longer supports v3?
>>
>
> I verified the credentials with both the aws tool and s3fs. I also tried
> creating brand new credentials, and they behave in precisely the same way.
> So I'm confident it's not a credential problem.

Right you are. There is nothing wrong your credentials, what has changed
is the way in which S3QL uses them to authenticate you to AWS.

>
> s3fs was very unhappy about mounting a bucket with dots in the name. I know
> s3ql used to have this problem, but I thought it'd been fixed. If I rename
> my bucket, will s3ql still mount it, or are there internal references to
> the bucket name in the metadata?

I don't think there are, but I also don't think that renaming would help.

[Daniel Jagszent]

Hi David,

• Install the latest version of S3QL
• UPGRADE THE FILE SYSTEM (that's what you have missed): https://www.rath.org/s3ql-docs/adm.html#upgrading-the-file-system
• now you should be able to mount the file system, again