sending http request after receiving [FIN, ACK]

[Tal Shemesh]

Hi,

In our project we use OrientDB and vertx 2.1.5

after querying the DB using restful request for several seconds, we started seeing the following error:

java.nio.channels.ClosedChannelException

        at org.vertx.java.core.http.impl.ClientConnection$2.handle(ClientConnection.java:256)

        at org.vertx.java.core.http.impl.ClientConnection$2.handle(ClientConnection.java:246)

        at org.vertx.java.core.net.impl.ConnectionBase.handleClosed(ConnectionBase.java:137)

        at org.vertx.java.core.http.impl.ClientConnection.handleClosed(ClientConnection.java:367)

        at org.vertx.java.core.net.impl.VertxHandler$3.run(VertxHandler.java:120)

        at org.vertx.java.core.impl.DefaultContext$3.run(DefaultContext.java:175)

        at org.vertx.java.core.impl.DefaultContext.execute(DefaultContext.java:135)

        at org.vertx.java.core.net.impl.VertxHandler.channelInactive(VertxHandler.java:118)

        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)

when looking at the http packages (using wireshak), it looks like the OrientDB is asking for connection colsing [Fin, Ack], vertx return an ACK for that request and then sends another POST request on the same connection, couple of millisecond before it send the ACK for the connection closing.

it looks like a bug,

why does vert.x send a request on a connection that just got a close request?

 

[Tim Fox]

If you don't want keep alive or pipe-lining you can disable it on the client.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tal Shemesh]

Thanks for the quick answer.

because we are sending many request to the OrientDB, we do want to use Keep-Alive.

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/ERfQ6p1DduU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vertx+un...@googlegroups.com.

[Tim Fox]

If keep alive is enabled then Vert.x will attempt to reuse connection.

Vert.x gets notification from Netty when a connection is closed. We don't get this notification until the connection has been closed.

In your sequence of events below, the connection doesn't get closed until after the POST has been sent, so I'm not sure there is anything Vert.x could have done differently.

[Tal Shemesh]

so you are saying it's probably a netty bug?

[Tim Fox]

I'm not sure it's a bug at all.

I think Netty is correct in not sending a close event until the connection is actually closed, and that happens _after_ the POST is sent.

It might be worth checking on the Netty list though...

[Tal Shemesh]

could it be that vert.x shouldn't send a post request on a channel that already got a close request?

[Tim Fox]

On 27/01/15 17:20, Tal Shemesh wrote:

could it be that vert.x shouldn't send a post request on a channel that already got a close request?

I'm not sure there is any way we can know that the channel got the close request. Remember... the TCP close handshake is handled by the OS, not by Vert.x or even Netty. We just get events when the connection has been closed.

[Jez P]

Why is OrientDB closing the connection? How can you reliably use keep-alive under those circumstances?

[Alexander Lehmann]

http keep-alive should indicate in the http response if keep alive can be used on this connection or not,the header Connection: either contains keep-alive or close, if it contains keep-alive it can also indicate a timeout and a maximum connection count left.

The server doesn't have to and the client doesn't have to honour the header, but it should be as an indication what can be done (I think, I should check with the http spec).

If the server says Connection: close, the client should not reuse the connection and if the keep-alive is allowed, it should account for the fact that a timeout may occur if the connection is kept and the server closes the connection in that case.

If the connection is closed even though the server has indicated that keep-alive is possible, it would be a bug in the server, if the server sends the header, its the clients resposibility, e.g. by handling a connection reset response gracefully and opening a new request or using another connection if available.

I haven't tried the behaviour in vert.x 3 so please take this with a grain of salt (I would like to write a few http compliance tests but I didn't get around to this yet).


One thing I would like to add is that the behaviour of the packages on the network (with wireshark) are not really an indication if the client or the server does everything right or not, if the request is sent out even though the connection has been reset on the other side may still be correct depending on the timing of the connection (i.e. if the server closes the connection after the client has started sending the next request, the client will get a socket error that has to be handled in some reasonable way).

[Alexander Lehmann]

Please note that a close request is a request meaning that the client may not have the correct status of that connection notified to the client yet.

[Alexander Lehmann]

Come to think of it, the client may not get the information that the socket was closed before it tries to write something to the socket, e.g. in database pools, the client will usually do a dummy select like select sysdate from dual to check if the socket is still connected and the server is still available.

[Alexander Lehmann]

Could you do me a favour and check if Orientdb is in fact implementing http keep-alive correctly. I have run a few tests requests on the server and it looks like it is returning Connection: keep-alive and then closes the connection, which would break the keep-alive feature, so there is no point in turning it on.

I am currently using Orientdb community 2.0.1 and I try to execute the following request:

GET /query/GratefulDeadConcerts/sql/select+from+V+where+type='artist'+and+name='Garcia' HTTP/1.1
Host: localhost
Connection: keep-alive
Authorization: Basic YWRtaW46YWRtaW4=

GET /query/GratefulDeadConcerts/sql/select%20from%20V%20where%20type='artist' HTTP/1.1
Host: localhost
Connection: keep-alive
Authorization: Basic YWRtaW46YWRtaW4=

this should execute two queries with keep-alive but it only returns one.

[Alexander Lehmann]

using ncat I should add, this is not related to vert.x it seems

[Tal Shemesh]

Hi Alexander, 

 Sorry for the late response.

 we tried a couple of things and got no success.

 It appears that this problem happens in all the apis we are integrating to (~50).

 the problem disappear when we disable the use of "keep-alive"

 I would expect vertx not to send a request on a connection that just got a "close connection" request. 

 Is it possible to implement this behavior?

[Tim Fox]

On 01/03/15 19:16, Tal Shemesh wrote:

Hi Alexander, 

 Sorry for the late response.

 we tried a couple of things and got no success.

 It appears that this problem happens in all the apis we are integrating to (~50).

 the problem disappear when we disable the use of "keep-alive"

 I would expect vertx not to send a request on a connection that just got a "close connection" request.

Hi Tal,

According to the information you provided before, Vert.x sends the request *before* it knows that the connection has been closed.

All the low level TCP protocol handling (sending ACKs etc) is not something that Vert.x does (or even that Netty does) - it's handled by the OS TCP implementation, so I'm not sure there is anything we can do here.

As Alex says, if your server is closing connections anyway, it seems pretty pointless you enabling keep alive.

[Alexander Lehmann]

I think it might be possible to handle the close request more gracefully and not pass an exception to the client but send the request on a new connection instead.

Haven't tried anything regarding this, but I wrote a unit test to show the behaviour.

I can put that up if you like

[Mik Quinlan]

Hi Alexander

I'm not sure if you have that unit test around, but I wouldn't mind seeing it.  We're having some strange ChannelClosed exceptions that we are investigating and what you describe could be a workaround while we figure out the issues.

Thanks in advance.