Strategy to retry on HTTP failures

[Naitik Shah]

I'm using a couple of libraries which wrap third party HTTP APIs (for example goamz/s3) which use the net/http package underneath. They occasionally fail with errors like "remote error: handshake failure" or "connection reset by peer" which are to be expected. But rather than failing I want to get them to retry. I was planning on patching my various call sites to deal with the retry logic at the high level API, but was wondering if anyone has advice on ways to generalize this. Most of them end up using http.DefaultClient, and I figured maybe something that automatically retries for GETs might work and get me a long way there.

--
-Naitik

[Dave Cheney]

Hello,

I'm a user of goamz and it looks like the aws service endpoint isn't
as reliable as we'd like. I think the best way to handle this would be
to raise a feature request on the goamz project. You can find it here
https://launchpad.net/goamz

As an aside, are you just trying the default region, or have you tried others ?

Cheers

Dave

[Naitik Shah]

My data is in the "US Standard" (US East?) region, but the small amount of connection failures seems reasonable imo, and I think I'm just going to handle this at each call site independently for now.

-Naitik

--
-Naitik

[Kyle Lemons]

I'm not sure about these libraries, but if they allow you to control their underlying http.Client you could put in your own custom Dial function that returns a special net.Conn with retry-on-error semantics.

[Naitik Shah]

I'm attempting to do it at the Transport layer, though it is still pretty rough and incomplete so far: 

https://github.com/nshah/go.httpcontrol/blob/master/httpcontrol.go

--
-Naitik

[roger peppe]

this is a known issue with goamz. Gustavo Niemeyer is planning
to fix it soon.

wouldn't it be nice if AWS worked more reliably?!

[John Beisley]

However you do it, do include exponential backoff on retry, website admins appreciate that sort of thing!