http.Client timeouts

[Sanjay Menakuru]

Hey golang-nuts,

With the new SetDeadline-style methods on net.Conn, there seems to be no way to make a http request that has a timeout, taking into account redirects, and keep-alive connections.

Previously, I would do this by providing my own Dial function and setting the Conn's timeout, which, as I understand it, was a flawed technique anyways.

But now, there doesn't seem to be a clean way to do it.

Please let me know if there is a clean way to handle this that I am overlooking.

Thanks,

Sanjay

[Kyle Lemons]

The same technique seems like it would work.  The difference is that the timeout applies to all reads as a whole, not to each individual read call.  I don't think, however, this plays nicely with the client pooling.

[Sanjay Menakuru]

It doesnt play nicely with connection pooling or redirects as far as I can tell. That is to say, if you set a timeout 10 seconds in the future, with a RedirectFunc that allows 3 redirects, then the three HTTP requests are sharing the timeout. There's no way to say *each* request gets 10 seconds.

In my case, I'm making internal calls, so I want to set the timeouts to extremely low values. So more like 20ms per redirect, max 3 redirects as a policy. This could be handled by setting Deadline to 60ms in the future (not strictly the same thing, but close enough), but connection pooling breaks...

Sanjay

[Kyle Lemons]

The solution probably involves wrapping the connection in a net.Conn of your own devising.

[Brad Fitzpatrick]

Let's always _a_ solution, but likely not the best one.

Sanjay, could you flie a bug so I remember to look into this at some point?

[Sanjay Menakuru]

I thought of returning my own net.Conn, but its hard to tell when you're on boundaries of requests, short of actually parsing stuff. I could parse, I suppose, but the redundant work would be annoying. Its probably easier to copy the code from net/http and add a Timeout field (of type time.Duration) to Transport that sets that as the ReadDeadline after calling req.Write.

Sanjay

[Johann Höchtl]

Am Dienstag, 20. März 2012 18:20:40 UTC+1 schrieb Brad Fitzpatrick:

Let's always _a_ solution, but likely not the best one.

Sanjay, could you flie a bug so I remember to look into this at some point?

Has the former solution been a security issue for GAE? Otherwise why not restore it, provided more sensible tailoring.
 

[Sean Talts]

Here's what I've come up with, with tests: http://gist.github.com/seantalts/11266762

Involves making your own Transport and overriding RoundTrip with a function that sets a goroutine that calls the original RoundTrip and selects between that and a timeout. This is the technique they seem to use internally to set a timeout for other things in transport.go. Please let me know if I'm overlooking something or this is not good style; for example I'm aware that the goroutines will only exit once they have finished waiting for the response.

[ivan.v...@gmail.com]

I have a post request which uploads a video file to drive using a long running PUT request.
In your solution the request will be broken after the timeout, instead I would like to break it and create new one to continue uploading if there was no bytes sent during certain time.

[Sean Talts]

Good point. I think even the new http.Client Timeout in 1.3 will have the same issue, if I'm reading this right: https://codereview.appspot.com/70120045/patch/120001/130001

Doesn't affect us at the moment but good to keep track of this behavior.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/-DE6-aLttM4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Camilo Aguilar]

I'm have exactly the same issue using http.Client to download a big file. Timeout just fires up ignoring the ongoing transfer. I'm using Go 1.3, this used to work fine using conn.SetDeadline in Go 1.2.1. The issue happens as well when setting the Timeout property in http.Client.

[Camilo Aguilar]

I just filed a bug to track this in https://code.google.com/p/go/issues/detail?id=8465

[James Bardin]

Both methods of timing out have nothing to do with if there is activity on the connection or not. They are doing what they are supposed to do, hitting a deadline or closing the connection after the specified time.

If this worked before, it was a bug.

[Camilo Aguilar]

That makes no sense to me, would you please elaborate more? If I'm setting a timeout in the http.Client, as a user, I'm expecting it to kick in if the connection is unable to be established for some reason, ie: DNS taking too long to resolve or server taking to long to accept the connection. I wouldn't expect it to kick in in the middle of a data transfer.

[James Bardin]

On Sat, Aug 2, 2014 at 12:10 PM, Camilo Aguilar <camilo....@gmail.com> wrote:
>
> That makes no sense to me, would you please elaborate more? If I'm setting a timeout in the http.Client, as a user, I'm expecting it to kick in if the connection is unable to be established for some reason, ie: DNS taking too long to resolve or server taking to long to accept the connection. I wouldn't expect it to kick in in the middle of a data transfer.

The http client documentation is pretty clear with "and will interrupt reading of the Response.Body."

        // Timeout specifies a time limit for requests made by this
        // Client. The timeout includes connection time, any
        // redirects, and reading the response body. The timer remains
        // running after Get, Head, Post, or Do return and will
        // interrupt reading of the Response.Body.


As is net.Conn

        // A deadline is an absolute time after which I/O operations
        // fail with a timeout (see type Error) instead of
        // blocking. The deadline applies to all future I/O, not just
        // the immediately following call to Read or Write.

The proper way to implement an idle timeout is to walk a deadline forward on each read or write operation. If you're looking for an IdleTimeout, you need to create your own net.Conn which sets deadlines on each operation. You can look at how we implemented the ReadWriteTimeout in github.com/mreiferson/go-httpclient (which is mostly surpassed by go1.3 except for this situation).

[Camilo Aguilar]

Thanks for clarifying. IMHO, just because the documentation explains the behavior doesn't mean Go's API in this regard is totally unexpected. I will be very useful to understand the reasoning behind this implementation, especially in http.Client.

[James Bardin]

On Sat, Aug 2, 2014 at 2:55 PM, Camilo Aguilar <camilo....@gmail.com> wrote:

I will be very useful to understand the reasoning behind this implementation, especially in http.Client.

I'm not certain I understand; are you looking for more clarification on this? If so, what aspect of it don't you understand?

[Camilo Aguilar]

I'm understand now, I'm just saying it is a weird and unexpected API behavior in http.Client, despite being documented.