No error go side: "connection reset by peer" not working as expected

[Peter Waller]

Hi All,

Mysteriously, I'm not getting "connection reset by peer" when I expect it in go.

The problem I'm trying to solve is to await something listening on a TCP socket inside a docker container. Docker proxies connections to the underlying server and so is already listening on the target port - it gives a "connection reset by peer" message if you try to read from the socket before the client is ready.

I have seen this with curl reproducibly every time if the target server is not ready:

$ curl 0.0.0.0:49252

curl: (56) Recv failure: Connection reset by peer

So my strategy is to try connecting repeatedly and then when I get a connection without error, declare the server is "ready for connections".

Here's an excerpt of the go code: http://play.golang.org/p/Mk_GMfDrdi

(It's not how I would write it properly, it's been heavily bastardized to try and understand this problem)

The problem is, when the server isn't ready, there is no error when I try contacting the TCP port with go. It writes without error, and it reads zero bytes without error:

Sent req: 35 <nil>

Listening!

bytes, err: 0 <nil>

err: <nil>

Copied!

Hopefully I've not missed any error checks anywhere, but that's one reason I supplied the code so you can see for yourselves.

If I were only supporting HTTP, I could check that a HTTP request succeeds without error and be done with it, but I want to support arbitrary TCP protocols.

The only difference I see between go and curl is that curl is using the send/recv syscalls and go read/write.

My question is: should I expect to see a "Connection reset by peer" error message, and if not why not?

Thanks,

- Peter

[James Bardin]

Yeah, I've never been thrilled with this docker userland proxy. Last I saw though, the team was working on removing it (or atleast only making it optional), so we may not have to deal with it much longer.

On Tuesday, November 18, 2014 5:27:23 PM UTC-5, Peter Waller wrote:

The problem is, when the server isn't ready, there is no error when I try contacting the TCP port with go. It writes without error, and it reads zero bytes without error:

Sent req: 35 <nil>

Listening!

bytes, err: 0 <nil>

err: <nil>

Copied!

What's happening here is this: 

  You send your request, which fits in one packet, the server can't proxy it to the backend, so it closes the connection. If you sent multiple packets, you would get an RST, causing the "Connection reset by peer". If you tried to send again, you'd get a "broken pipe".

  You then do a recv, which returns 0 bytes (this is equivalent to an EOF on a socket), so io.Copy returns 0, nil.

At the HTTP protocol level, an empty response is analogous to an error, so it's up to you to treat it as such. Ping a healthcheck endpoint until it returns a 200.

 

[Peter Waller]

Thanks James.

Here are links for further reading for anyone watching. Looks like the
proxy may be removed imminently with any luck.

https://github.com/docker/docker/issues/8356
https://github.com/docker/docker/pull/9078

However there is some chance it may hang around to support some
non-linux systems.

Sadly, the userland proxy is less magical than I thought it might be,
as you said:

https://github.com/docker/docker/blob/master/pkg/proxy/tcp_proxy.go

What should a TCP proxy do when upstream is unavailable?

Is there any way in theory a TCP proxy can signal to a connectee that
upstream is unavailable? Ultimately when the initiator connects and
upstream is not alive, I would like the initator's read or write
syscalls to fail with error.

I can think of a few partial solutions which all have drawbacks.

1. Make the upstream connection in advance of accepting
Drawbacks:
* Results in idle connection to upstream for an arbitrarily long time
* If upstream doesn't accept, the proxy doesn't accept and the
inbound client blocks

2. If it's not possible to connect to upstream, send an out-of-band
exception packet
Drawbacks:
* Requires the initiator to be aware of this protocol extension

3. Send a garbage packet and hope that this causes the initiator to
fail with errors.
Drawbacks:
* Eww. (Even more than the other ideas)

If there is no workable solution, ultimately it looks like I will have
to live with being somewhat protocol-aware or have to have a way of
negotiating with the target listener that they are ready to receive
connections, which seems unfortunate.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Peter Waller]

I have found a solution which works on Linux at least!

I reliably get ECONNRESET (Connection reset by peer) from the read syscall now.

It's possibly not ideal behaviour - it might be even better if the
client got an ECONNREFUSED on connect. It is unclear if it's possible
to arrange this. It might be with a clever arrangement of multiple
listens with a queue length of one and SO_REUSEADDR.

Here is the protocol-agnostic proxy which returns ECONNRESET in the
case where upstream is unavailable (without the proxy implementation):

package main

import "net"

func main() {
l, err := net.Listen("tcp", ":9999")
if err != nil {
panic(err)
}

for {
c, err := l.Accept()
if err != nil {
panic(err)
}
go func() {
defer c.Close()
c.(*net.TCPConn).SetLinger(0)
}()
}
}

Here is a client which just does a read - it does actually see a
connection reset from the read syscall.

package main

import (
"log"
"net"
)

func main() {
c, err := net.Dial("tcp", "localhost:9999")
if err != nil {
log.Fatal(err)
}

b := make([]byte, 1024)
n, err := c.Read(b)
log.Fatalf("%v %s", n, err)

[James Bardin]

On Wed, Nov 19, 2014 at 3:47 AM, Peter Waller <pe...@scraperwiki.com> wrote:

Is there any way in theory a TCP proxy can signal to a connectee that
upstream is unavailable? Ultimately when the initiator connects and
upstream is not alive, I would like the initator's read or write
syscalls to fail with error.

Nope, this is just an inherent drawback of using a TCP proxy. There's no way to replicate the connection state directly on both sides at the userland level. For example, you can't finish the SYN/ACK handshake, then pretend you didn't. If you go low enough down the stack to do this, you just made a NAT service which iptables is already very good at.

Unfortunately, none of these improve the situation universally:

 

I can think of a few partial solutions which all have drawbacks.

1. Make the upstream connection in advance of accepting
  Drawbacks:
  * Results in idle connection to upstream for an arbitrarily long time
  * If upstream doesn't accept, the proxy doesn't accept and the
    inbound client blocks

The upstream service may not tolerate idle connection, causing more problems when timeouts occur around the time you want to splice the two sides together.

 

2. If it's not possible to connect to upstream, send an out-of-band
   exception packet
  Drawbacks:
  * Requires the initiator to be aware of this protocol extension

Exactly.

 

3. Send a garbage packet and hope that this causes the initiator to
   fail with errors.
  Drawbacks:
  * Eww. (Even more than the other ideas)

Not even a consideration.

On Wed, Nov 19, 2014 at 5:48 AM, Peter Waller <pe...@scraperwiki.com> wrote:

            c.(*net.TCPConn).SetLinger(0)

I proposed adding SetLinger(0) in a couple key places (mostly to reduce port exhaustion on the upstream side; clients do not want every connection ending with an RST), but they were rightly conservative about altering the proxy behavior. Plus, getting an RST when the upstream can't connect is no more valid than closing the connection. Having the connection close early is always an error state you have to deal with anyway.

[Peter Waller]

Thanks for the discussion, this has all improved my understanding significantly.

It's unfortunate though because I find myself trying to stack these
proxies - I want to have a service which conditionally forwards one
public port to docker different containers - how can one implement that?

Maybe I shall invest time in considering iptables, but that looks like
it would have a slew of other problems, for example what happens if my
orchestrator crashes? Does it leave random iptable rules lingering? I
have some learning to do..

[James Bardin]

On Wed, Nov 19, 2014 at 9:44 AM, Peter Waller <pe...@scraperwiki.com> wrote:

Thanks for the discussion, this has all improved my understanding significantly.

It's unfortunate though because I find myself trying to stack these
proxies - I want to have a service which conditionally forwards one
public port to docker different containers - how can one implement that?

Based on what conditions? At the time you accept the connection you don't have any more information that port number and ip address.

 

Maybe I shall invest time in considering iptables, but that looks like
it would have a slew of other problems, for example what happens if my
orchestrator crashes? Does it leave random iptable rules lingering? 

Yes, you would have to handle that in some way. This is why they're trying to integrate it into the docker daemon directly.

Really the easiest solution here *is* a proxy. The only situation you presented that you had a problem with was the connection closing when the backend is unavailable. Yes, depending on what your client is doing, this can cause a "reset by peer", or a "broken pipe", or a closed connection with no response, but these are all errors you have to deal with anyways.