s3backer: can't read block zero meta-data: Operation not permitted

[James Royal-Lawson]

Hi there!

I'm a bit stumped as to why I'm suddenly getting...

2017-06-26 08:12:48 ERROR: rec'd 403 response

s3backer: can't read block zero meta-data: Operation not permitted

Everything has been working fine for years. I recently managed to expand the size of my s3backer disk (see separate thread in this group) and that was working fine. But I had to restart the server recently Since booting it, I can't get s3backer to start. Restarting has never been a problem in the past on the (few) occasions I've had to restart the server.

I've not changed any AWS IAM settings or the permissions for the s3 bucket - it's the same user and the same key and the same bucket that was working perfectly OK. 

Any suggestions as to what might be causing me to get a 403 response now?

/James

[Archie Cobbs]

Which --authVersion are you using? It's possible they have stopped accepting version 2 (just a guess).

-Archie

[James Royal-Lawson]

aws2, but I've just tried it with --authVersion=aws4 and got the same result. a 403.

/James

[Archie Cobbs]

Weird.

Try running in the foreground with -f --debug-http so you can see if there is any information in the HTTP payload of the error response.

Also make sure that s3backer is actually trying to authenticate, i.e., properly provisioned with credentials and passing the appropriate headers.

-Archie

[James Royal-Lawson]

Here's the bit of the output after > HEAD (with a few things redacted).

Doesn't really say much. 

User-Agent: s3backer/1.3.2/r451
Host: xxxxx.s3.amazonaws.com
Accept: */*
Date: Fri, 30 Jun 2017 09:33:28 GMT
Authorization: AWS xxxxx

* The requested URL returned error: 403

* Closing connection #0

* HTTP response code said error

2017-06-30 11:33:28 ERROR: rec'd 403 response: HEAD https://xxxxx.s3.amazonaws.com/s300000000

I'm running the command as root. As I mentioned earlier, I'm using the same ID and key pair that I've been (successfully) for year. The fact that it stopped after I rebooted is really puzzling me. I've rebooted the server several times over the years and had no problem mounting the disk again. 

/James

[Archie Cobbs]

On Friday, June 30, 2017 at 7:21:00 AM UTC-5, James Royal-Lawson wrote:

I'm running the command as root. As I mentioned earlier, I'm using the same ID and key pair that I've been (successfully) for year. The fact that it stopped after I rebooted is really puzzling me. I've rebooted the server several times over the years and had no problem mounting the disk again.

Did you upgrade s3backer without restarting it sometime before the reboot? If so then the new version may have an issue relative to the old version, but you wouldn't have noticed it until the reboot and consequent restart of s3backer.

-Archie

[James Royal-Lawson]

I rebooted in April and definitely haven't updated s3backer since then. This reboot was though the first since I successfully increased the size of my s3backer disk from 250g to 500g.

/James 

[Archie Cobbs]

On Friday, June 30, 2017 at 7:21:00 AM UTC-5, James Royal-Lawson wrote:

User-Agent: s3backer/1.3.2/r451

I just realized you are running version 1.3.2, which is really old (2011 vintage).

There have been lots of bug fixes since then. Try upgrading to at least 1.4.0 (using version 4 authentication).

-Archie

[James Royal-Lawson]

 Blimey! That is old... I've just grabbed and installed 1.4.3

Hasn't helped. I've tried with both authversions aws4 and aws2. No luck :-(

> HEAD /s300000000 HTTP/1.1
User-Agent: s3backer/1.4.3/
Host: xxxxx.s3.amazonaws.com
Accept: */*
x-amz-date: 20170630T133109Z
x-amz-content-sha256: xxxxx
Authorization: AWS4-HMAC-SHA256 Credential=AKIAIPVQVLCZ32X3PHXQ/20170630/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=xxxxx

* The requested URL returned error: 403
* Closing connection #0
* HTTP response code said error

2017-06-30 15:31:09 ERROR: rec'd 403 response: HEAD https://xxxxx.s3.amazonaws.com/s300000000
s3backer: can't read data store meta-data: Operation not permitted

[Archie Cobbs]

OK, last idea... check if issue #69 (fixed in version 1.4.4) applies to you.

Also, try adding the --vhost flag if not already using it.

-Archie

[James Royal-Lawson]

I'm already using the --vhost flag. My don't think that issue applies to me (I've got a prefix of s3 (no slash). I tried omitting --listblocks but it didn't make any difference.

For reference here's the full command I'm (normally) running.

s3backer --blockSize=1024k --size=500g --listblocks --vhost --ssl --timeout=250 --maxUploadSpeed=35k --blockCacheThreads=3 --blockCacheFile=/tmp/s3backerCache --accessFile=/etc/passwd-s3backer --prefix=s3 xxxxx /s3

/James

/James