Somewhat frequent Excon::Errors::Forbidden exceptions

[Trevor Turk]

I'm not sure if this is the right place to ask, but I've been seeing
somewhat frequent errors when trying to use the Carrierwave/Fog/Excon
combo like so:

https://gist.github.com/6e03558b08a80b2bd3e5

I can't think of any particular reason that this might be happening,
or why it seems to be sporadic.

I'm wondering if anyone else with more S3 experience has seen this
kind of thing before.

Any ideas would be very much appreciated - I'm at a loss!

Thanks!
- Trevor

[Trevor Turk]

On Dec 3, 11:52 am, Trevor Turk <trevort...@gmail.com> wrote:
> I can't think of any particular reason that this might be happening,
> or why it seems to be sporadic.

I've come across a few mentions of a problem with system clocks that
can result in 403 errors. See this FAQ:

http://aws.amazon.com/articles/1109?_encoding=UTF8&jiveRedirect=1#04

I'm hosting this app on Heroku, and I'm not sure why the times would
be off... anyway, I'm still investigating, but thought it worth
mentioning this.

- Trevor

[geemus]

It looks like the error you posted in the gist was missing part of the
text (in particular it appears to be a partial description of the
request, but not the response). Could you get me the response info?
Frequently the body of that will have a more specific explanation for
the error and in this case it ought to tell us if it is clock skew or
not, and/or give me something else to dig in to.
Thanks,
wes

[Trevor Turk]

On Dec 3, 6:52 pm, geemus <wbe...@engineyard.com> wrote:
> It looks like the error you posted in the gist was missing part of the
> text (in particular it appears to be a partial description of the
> request, but not the response).  Could you get me the response info?
> Frequently the body of that will have a more specific explanation for
> the error and in this case it ought to tell us if it is clock skew or
> not, and/or give me something else to dig in to.
> Thanks,
> wes

Thanks very much for the reply. That was an error from Hoptoad, which
seems to be cutting off the error message at a certain length. I'll
see if there's another way I can figure out to get the entire
exception and report back here if I'm successful. I've also enabled
logging in S3, so perhaps that will show some useful info as well.

Thanks again,
- Trevor

[Trevor Turk]

On Friday, December 3, 2010 6:52:21 PM UTC, geemus wrote:

It looks like the error you posted in the gist was missing part of the text (in particular it appears to be a partial description of the request, but not the response). Could you get me the response info? 

I enabled logging on the relevant S3 bucket, and I'm seeing "403 RequestTimeTooSkewed" errors. This app is on Heroku, so I'm following up with them. Perhaps there's something wrong with the clocks somewhere in their system...? I'm not sure what else to do.

I've also forked excon and made a change in the hopes of getting more info into Hoptoad:

https://github.com/trevorturk/excon

Perhaps that'll help. I wasn't sure if I should open a pull request for excon, though, since this is kind of a funny problem that might be specific to me. Still, it might be useful to have the response info first, since it may be more interesting than the request.

I'll report back if/when I resolve this. Thanks!

- Trevor 

[Jon Crosby]

As an extra point of data, I can reproduce this exact error on any new EC2 instance (not on Heroku) that attempts to write to S3 using Fog as the last line of its user-data bash script. Logging into that same instance and re-running the same line of code works without the error.

I'll provide more info if I have time to explore this weekend.

-Jon

[geemus (Wesley Beary)]

I just added something that should at least provide a work around for you guys.  A new method/request on s3 in particular.

storage = Fog::Storage.new(:provider => 'AWS', ...)

storage.sync_clock

It will try to list your buckets, and rescue if necessary.  Either way it will pull the date back that amazon reports and set an offset so that further requests will match against aws.  Unless your skew is happening in realtime (and quickly) this should work, albeit be a bit slow (ideally you would reset the clock on the machine itself once rather than doing this regularly).  Anyway, hopefully should have this released later today.  Let me know what you think and/or if you have issues trying it.

Thanks,

wes

[Trevor Turk]

On Tuesday, December 21, 2010 8:16:12 PM UTC, geemus wrote:

It will try to list your buckets, and rescue if necessary.  Either way it will pull the date back that amazon reports and set an offset so that further requests will match against aws.  Unless your skew is happening in realtime (and quickly) this should work, albeit be a bit slow (ideally you would reset the clock on the machine itself once rather than doing this regularly).  Anyway, hopefully should have this released later today.  Let me know what you think and/or if you have issues trying it.

Hmm... maybe putting this into an initializer would be smart...? I'll look out for a release and give it a try. Thank you! 

[geemus (Wesley Beary)]

I considered putting it in the initializer, but haven't yet for a couple reasons:

1) I'm not entirely sure it will solve the problem (though it seems likely)

2) it may have other repercussions that I haven't realized yet

3) it would cause initialization to require a sync which is slower

3 isn't a deal breaker but 1 and 2 might be, figured I would release it into the wild and see how it sticks/works and then maybe add it to initialization if it looks good/worth it.

[Trevor Turk]

On Tuesday, December 21, 2010 9:39:51 PM UTC, geemus wrote:

I considered putting it in the initializer, but haven't yet for a couple reasons...

I'm sorry - I should have been more clear. I was thinking of putting this into an initializer in my Rails app. So, when the app started up, it'd sync up the clocks, which seems like a smart idea (since I'm having this problem). I don't think it should be done automatically by Fog... not that I've given it much thought. But, if the problem is widespread and/or if we have great success, then some kind of documentation recommending it might be good, but it seems weird to do it automatically (you shouldn't have to, really, you know?)

Anyway - I'm sleepy and forgot to thank you properly. Thank you!!! I'll give it a shot ASAP and report back. 

- Trevor

[geemus (Wesley Beary)]

Got it, yeah a rails initializer seems like a great place for this (because you know you have had the issue).  Once it is a bit more field tested I definitely think a doc update is in order.

Also you are very welcome, hope it makes the problem go away for you.

[Trevor Turk]

On Tuesday, December 21, 2010 9:52:53 PM UTC, geemus (Wesley Beary) wrote:

Got it, yeah a rails initializer seems like a great place for this (because you know you have had the issue).  Once it is a bit more field tested I definitely think a doc update is in order.

Also you are very welcome, hope it makes the problem go away for you.

Just reporting back -- I haven't seen this error since adding an initializer that calls sync_clock. I can't be 100% sure that this is what made the problem go away, but it seems somewhat likely. I'll report back if the problem shows up again, but I'm feeling quite happy at the moment. Thanks again. Working with Fog has been a real treat so far. 

- Trevor