Stalling http.Get - wireshark full of [RST, ACK]

[Juli an]

Hi,

I am struggling with a very annoying problem. My application does a lot of http GET requests to some .NET Application on an IIS 7.5.

For a random number of requests, the connections return, a status code is delivered, body read, everything's fine. But at some point, it does not depend on the URL as it seems, the Server sends a lot of packets of the form:

 

114719 3898.085719000 62.x.x.x 192.168.0.14 TCP 60 80→50748 [RST, ACK] Seq=413 Ack=391 Win=0 Len=0

I did try to set timeouts to catch this. The default one and also via custom transports like described here: http://stackoverflow.com/questions/16895294/how-to-set-timeout-for-http-get-requests-in-golang

Yes I close the response body, this is no issue. (The socket count in the running process is low)

None of them were successful. Do you have an idea how to work around this or get it fixed?

Thank you!

Cheers

Julian

[Jesse McNelis]

On Thu, Oct 29, 2015 at 10:07 AM, Juli an <julian...@googlemail.com> wrote:
> Hi,
>
> I am struggling with a very annoying problem. My application does a lot of
> http GET requests to some .NET Application on an IIS 7.5.
>
> For a random number of requests, the connections return, a status code is
> delivered, body read, everything's fine. But at some point, it does not
> depend on the URL as it seems, the Server sends a lot of packets of the
> form:
>
> 114719 3898.085719000 62.x.x.x 192.168.0.14 TCP 60 80→50748 [RST, ACK]
> Seq=413 Ack=391 Win=0 Len=0
>

Looks like the server is closing the connection.

> I did try to set timeouts to catch this. The default one and also via custom
> transports like described here:
> http://stackoverflow.com/questions/16895294/how-to-set-timeout-for-http-get-requests-in-golang
> Yes I close the response body, this is no issue. (The socket count in the
> running process is low)
>
> None of them were successful. Do you have an idea how to work around this or
> get it fixed?
>

Perhaps the server has some kind of rate limiting that prevents the
client from re-establishing a connection.

[Dave Cheney]

There may be an intermediary, or several, who is interrupting the request. 

Your client is on an rfc 1918 address range, so there is some nat going on, probably at your gateway router -- if that is overloaded or malfunctioning, that could cause an issue. Similarly, there is likely to be one or more layers of transparent caching and proxying between you and the origin server, any of those can interpose themselves on the tcp flow.

Does the target server offer https ? Does the problem occur when using https ?

[Juli an]

Thanks Dave for your answer,

On Thursday, October 29, 2015 at 1:28:28 AM UTC+1, Dave Cheney wrote:

There may be an intermediary, or several, who is interrupting the request. 

Your client is on an rfc 1918 address range, so there is some nat going on, probably at your gateway router -- if that is overloaded or malfunctioning, that could cause an issue. Similarly, there is likely to be one or more layers of transparent caching and proxying between you and the origin server, any of those can interpose themselves on the tcp flow.

Yes there is NAT between me and the router and further (carrier grade) NAT due to DS-Lite. I'll try running the tool on a directly attached machine with a public-v4 address.

What surprises me anyway is that none of the set timeouts seem to catch the request. It just hangs infinitely.

I also thought that some goroutines which do further processing of the response data could be the problem but if I understand correctly, goroutines do not interfer with the "main" command-flow.

 

Does the target server offer https ? Does the problem occur when using https ?

The 62.x.x.x Server only speaks IPv4 and HTTP-only. No HTTPS.

[Juli an]

On Thursday, October 29, 2015 at 1:07:38 AM UTC+1, Jesse McNelis wrote:

On Thu, Oct 29, 2015 at 10:07 AM, Juli an <julian...@googlemail.com> wrote:
> Hi,
>
> I am struggling with a very annoying problem. My application does a lot of
> http GET requests to some .NET Application on an IIS 7.5.
>
> For a random number of requests, the connections return, a status code is
> delivered, body read, everything's fine. But at some point, it does not
> depend on the URL as it seems, the Server sends a lot of packets of the
> form:
>
> 114719 3898.085719000 62.x.x.x 192.168.0.14 TCP 60 80→50748 [RST, ACK]
> Seq=413 Ack=391 Win=0 Len=0
>

Looks like the server is closing the connection.

That was my thought too. I then tried explicitly sending "Connection:close" but that didn't help.

Do you have any clue why those RST packets aren't recognized as fatal to the opened socket? I don't understand at which level the Information of that RST is "dropped"...

> I did try to set timeouts to catch this. The default one and also via custom
> transports like described here:
> http://stackoverflow.com/questions/16895294/how-to-set-timeout-for-http-get-requests-in-golang
> Yes I close the response body, this is no issue. (The socket count in the
> running process is low)
>
> None of them were successful. Do you have an idea how to work around this or
> get it fixed?
>

Perhaps the server has some kind of rate limiting that prevents the
client from re-establishing a connection.

There are indeed ready solutions for that in the .NET MVC communiy indeed. But those provide decent error codes in HTTP, not TCP-level closing of the connection.

[Juli an]

It all came down to insufficient error handling in my application. It accessed an http Response object which was nil eventually and then hang. This happened in simple innocent-looking variable-lookups like "response.ContentLength".

Adding proper error handling that does retries even if the request fails (not just bad HTTP status code) fixes the problem.