How to detect TCP connection closed by remote peer with non blocking manner?

[Naoki INADA]

Hi, network programmers.

When writing request-response protocol client, I don't want to have separated reader goroutine.

In case of mysql client, not having separate reader goroutine makes:

* We can reuse same buffer for sending query and reading response.

* We can implement read timeout (from query sent) easily with conn.SetReadDeadline()

* We can parse read buffer directly.  It minimize allocations.

But we have one problem about it: sending query is succeeded even if MySQL server

had shut down the connection.  And when conn.Read() returns io.EOF, there are no way

to determine we can safely retry sending query which is not idempotent with new connection.

See these issue and pull request.

https://github.com/go-sql-driver/mysql/issues/295

https://github.com/go-sql-driver/mysql/pull/302

I know sending query from client and closing connection from server may happen at same time.

But in most case, server initiate close is happen while client connection is in connection pool.

So I want to check the connection is closed by server or not before sending query.

In C, I can do it with select().  But in Go, I don't know straight way to do it.

I think this problem is not only for mysql, but same to many request-response protocols like HTTP/1.1.

How do you think about this problem?

[Jesse McNelis]

On 10 May 2016 12:55 p.m., "Naoki INADA" <songof...@gmail.com> wrote:
>
>
> But we have one problem about it: sending query is succeeded even if MySQL server
> had shut down the connection.  And when conn.Read() returns io.EOF, there are no way
> to determine we can safely retry sending query which is not idempotent with new connection.

A Write() to a TCP connection will return an error if the kernel knows the connection is already closed(just like select() does)
But if the other end of the connection didn't properly close the connection then you have to wait for TCP timeout.

The most straight forward thing to do is just Write() to the connection, wait for a response and handle the error.
It looks like the mysql protocol has a PING command (https://dev.mysql.com/doc/internals/en/com-ping.html) which does give you something to send to test a connection before sending an actual sql query. Perhaps that would be better for your usecase.

But in general if your application level protocol (HTTP, mysql) doesn't allow you to reconnect and ask whether a request completed then there is no way to be certain whether it did or not.

[INADA Naoki]

On Tue, May 10, 2016 at 1:51 PM, Jesse McNelis <jes...@gmail.com> wrote:

On 10 May 2016 12:55 p.m., "Naoki INADA" <songof...@gmail.com> wrote:
>
>
> But we have one problem about it: sending query is succeeded even if MySQL server
> had shut down the connection.  And when conn.Read() returns io.EOF, there are no way
> to determine we can safely retry sending query which is not idempotent with new connection.

A Write() to a TCP connection will return an error if the kernel knows the connection is already closed(just like select() does)

No.  TCP has half-close.  There are no way to know received FIN is half-close or complete-close.

Kernel can't close the connection when just received FIN.

https://play.golang.org/p/QVA57iddia

 

But if the other end of the connection didn't properly close the connection then you have to wait for TCP timeout.

The most straight forward thing to do is just Write() to the connection, wait for a response and handle the error.
It looks like the mysql protocol has a PING command (https://dev.mysql.com/doc/internals/en/com-ping.html) which does give you something to send to test a connection before sending an actual sql query. Perhaps that would be better for your usecase.

Sending ping before each query makes twice roundtrip... 

But in general if your application level protocol (HTTP, mysql) doesn't allow you to reconnect and ask whether a request completed then there is no way to be certain whether it did or not.

I'm not protocol designer.

--

INADA Naoki  <songof...@gmail.com>

[Jakob Borg]

2016-05-10 7:03 GMT+02:00 INADA Naoki <songof...@gmail.com>:
>> But in general if your application level protocol (HTTP, mysql) doesn't
>> allow you to reconnect and ask whether a request completed then there is no
>> way to be certain whether it did or not.
>
> I'm not protocol designer.

You will need application level responses to determine this. If you
perform an operation over TCP and get a successful response back (from
the application on the other side), it was successful. If you get a
connection error it may have been successful or not, you can't tell
and should retry. There is usually no point in verifying that the
connection is still alive before performing your operation, as it may
equally well close just after your verification. "PING" commands and
similar can be useful to keep a connection from timing out, and making
sure that the server was alive and well at least a short while ago. :)

//jb

[ma...@joh.to]

On Tuesday, May 10, 2016 at 1:02:53 PM UTC+2, Jakob Borg wrote:

2016-05-10 7:03 GMT+02:00 INADA Naoki <songof...@gmail.com>:
>> But in general if your application level protocol (HTTP, mysql) doesn't
>> allow you to reconnect and ask whether a request completed then there is no
>> way to be certain whether it did or not.
>
> I'm not protocol designer.

You will need application level responses to determine this. If you
perform an operation over TCP and get a successful response back (from
the application on the other side), it was successful. If you get a
connection error it may have been successful or not, you can't tell
and should retry.

 

Sure.  You can't be absolutely sure in all cases.  But that's not the point.
 

> There is usually no point in verifying that the
connection is still alive before performing your operation, as it may
equally well close just after your verification.

If the connection has been idle for a while (say, 10 minutes), it's not unreasonable to ask the operating system whether the connection was closed in the meantime.  But AFAIK there's currently no nice way of doing that in Go.
 

> "PING" commands and
similar can be useful to keep a connection from timing out, and making
sure that the server was alive and well at least a short while ago. :)

I don't like the idea of pinging the server routinely.  On the other hand, if the connection has been idle for a significant amount of time, perhaps the round-trip caused by the ping is not such a big deal as your app is clearly not too busy.


.m

[Jakob Borg]

2016-05-10 13:11 GMT+02:00 <ma...@joh.to>:
> Sure. You can't be absolutely sure in all cases. But that's not the point.
>
>>
>> > There is usually no point in verifying that the
>> connection is still alive before performing your operation, as it may
>> equally well close just after your verification.
>
>
> If the connection has been idle for a while (say, 10 minutes), it's not
> unreasonable to ask the operating system whether the connection was closed
> in the meantime. But AFAIK there's currently no nice way of doing that in
> Go.

I think my point is that even if you could do that, the best the OS
can offer is "I think so, at least I haven't heard otherwise" and you
might still get a "connection reset" error one millisecond later when
actually trying to use the connection. Hence it's usually best to not
ask and just use the connection instead, then handle the error that
might come along. Regular PINGs on an otherwise unused connection are
of course useful to detect it going down and to keep it alive to begin
with.

//jb

[ma...@joh.to]

On Tuesday, May 10, 2016 at 1:16:38 PM UTC+2, Jakob Borg wrote:

I think my point is that even if you could do that, the best the OS
can offer is "I think so, at least I haven't heard otherwise" and you
might still get a "connection reset" error one millisecond later when
actually trying to use the connection.

Yeah, and I addressed your point above.  Yes, you can't do this perfectly in all cases.  But that doesn't mean you shouldn't try to improve on the current situation, which is clearly suboptimal.

Consider that the connection might have TCP keepalives enabled and the OS actually already figured out that the connection is dead.  Or the other end just closed the connection normally in the meanwhile.  There are plenty of cases where it'd be useful to just check if the OS knows that the connection is clearly gone, without entering the realm of "I don't really know".

 

Regular PINGs on an otherwise unused connection are
of course useful to detect it going down and to keep it alive to begin
with.

I don't think there's any point in making the application layer involved in this.  TCP keepalives are a much better way of solving this problem.


.m

[Jakob Borg]

2016-05-10 13:26 GMT+02:00 <ma...@joh.to>:
> Consider that the connection might have TCP keepalives enabled and the OS
> actually already figured out that the connection is dead. Or the other end
> just closed the connection normally in the meanwhile. There are plenty of
> cases where it'd be useful to just check if the OS knows that the connection
> is clearly gone, without entering the realm of "I don't really know".

Read() will have returned an error immediately when the connection was closed.

>> Regular PINGs on an otherwise unused connection are
>> of course useful to detect it going down and to keep it alive to begin
>> with.
>
>
> I don't think there's any point in making the application layer involved in
> this. TCP keepalives are a much better way of solving this problem.

I'd love to see TCP keepalives be more generally useful. In practice
things get platform specific and annoying if you don't like the
default two hours timeout, and they don't work through proxies or load
balancers. For me it's usually simpler to do the check at the
application level. YMMV, I guess. :)

//jb

[Naoki INADA]

On Tuesday, May 10, 2016 at 9:06:19 PM UTC+9, Jakob Borg wrote:

2016-05-10 13:26 GMT+02:00  <ma...@joh.to>:
> Consider that the connection might have TCP keepalives enabled and the OS
> actually already figured out that the connection is dead.  Or the other end
> just closed the connection normally in the meanwhile.  There are plenty of
> cases where it'd be useful to just check if the OS knows that the connection
> is clearly gone, without entering the realm of "I don't really know".

Read() will have returned an error immediately when the connection was closed.

Yes.  But when connection is alive, it blocks forever since I want to check connection

**before** sending request.

Slightly offtopic: 0 byte read means EOF.  When I tried conn.Read() to 0 byte buffer,

Go returns io.EOF while peer hasn't closed the connection yet.

I think TCPConn.Read() should check receive buffer length is more than 0 byte...

[Jakob Borg]

2016-05-10 16:38 GMT+02:00 Naoki INADA <songof...@gmail.com>:
>> Read() will have returned an error immediately when the connection was
>> closed.
>>
>
> Yes. But when connection is alive, it blocks forever since I want to check
> connection
> **before** sending request.

I feel that we might be talking past each other here, but just to be
sure - if you're going to Write() on the conn, there is no need to
check it before that. The check will be performed as part of the
Write() and you will get an error back or not. Any check you do before
the Write() will at best answer whether the Write() *might* succeed.

> Slightly offtopic: 0 byte read means EOF. When I tried conn.Read() to 0
> byte buffer,
> Go returns io.EOF while peer hasn't closed the connection yet.
> I think TCPConn.Read() should check receive buffer length is more than 0
> byte...

From what I can tell, a zero byte Read() blocks even when there is
nothing to read. So combined with SetReadDeadline() it can do what you
want, not that I know if this guaranteed to be the case, and I
certainly don't recommend doing it.

That is, taking your example from above: https://play.golang.org/p/p65CnSwDvV

//jb

[Naoki INADA]

From what I can tell, a zero byte Read() blocks even when there is
nothing to read. So combined with SetReadDeadline() it can do what you
want, not that I know if this guaranteed to be the case, and I
certainly don't recommend doing it.

That is, taking your example from above: https://play.golang.org/p/p65CnSwDvV

//jb

Wow.  Different behavior between playground and my linux machine.

Maybe, it's difference between fake network stack in playground and real network stack for linux.

My environment is:

$ go version

go version go1.6.2 linux/amd64

$ go env

GOARCH="amd64"

GOBIN=""

GOEXE=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOOS="linux"

GOPATH="/home/inada-n/local/gopath"

GORACE=""

GOROOT="/home/inada-n/local/go1.6.2"

GOTOOLDIR="/home/inada-n/local/go1.6.2/pkg/tool/linux_amd64"

GO15VENDOREXPERIMENT="1"

CC="gcc"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"

CXX="g++"

CGO_ENABLED="1"

One byte read with short timeout can be used to check.

But how timeout can be short?

I want more straight way to non blocking read (or non blocking EOF detection).