dead sockets between proxy and service

[steven....@gmail.com]

Hi,

To whom can help.

I have a membrane service proxy(4.1.0) running on Linux box and have a REST web service running on tomcat8 behind the proxy.

Everything works fine, the proxy can route request to the service without any problem, except:

I noticed that after running for several days, there will be some sockets between the proxy and service build up.(after a week, there will be about 200 sockets dead there and never go away until restart the proxy)

most of dead sockets are at following status:

Proxy side                 Tomcat side

--------------                 ----------------

FIN_WAIT_1             CLOSE_WAIT             

                                                               or

ESTAB                      ESATB

Linux "ss -nt" commands shows:

State         Recv-Q    Send-Q                                        Local Address:Port                                          Peer Address:Port

ESTAB      94622      0                                                  20.168.1.32:35057                                           20.168.1.65:8443
ESTAB      0              637056                                        20.168.1.65:8443                                             20.168.1.32:35057

I can reproduce the above "ESTAB" dead socket by following way:

-run a client, send a GET request to the proxy, and proxy will route the request to the REST service(TOMCAT), the response will be about 3MB in size.

-after the client start receiving content, kill the client.

-this will end up with the above "ESTAB" state sockets pair, and they will never go away until restart the proxy.(or tomcat).

My questions are:

Is this a known issue of the proxy? or there is a setting in the proxy can prevent this happen?

I could not found any clue in the doc and forum, is there anybody have this problem before?

Any help or thought is welcome.

Thanks in advance,

[Tobias Polley]

Hi Steven,

I am not sure whether this is an easily solvable problem: The basic problem is that there is no API call in the synchronous network (Java) API to discover whether a connection has been closed by the remote side.

That the connection as been closed by the remote side is only exposed to the program when in actually tries to use the connection and results in an IOException "Connection closed" iirc.

In the scenario you describe (Proxy side: FIN_WAIT_1, Tomcat side: CLOSE_WAIT), Tomcat has already closed the connection and Membrane has not.

How this is caused may depedend on your custom setup.

The easiest way to work around this might be to configure Tomcat to use the "Keep-Alive" header "timeout" parameter (see http://tools.ietf.org/id/draft-thomson-hybi-http-timeout-01.html ). This header is recognized by Membrane and triggers a connection close  a few seconds after the timeout has expired. In your case, this should enable the connection to be actually be closed.

Best,
Tobias

[steven....@gmail.com]

Hi

Thanks for Tobias' response and suggestion.

Yes I can do something like the "Keep-Alive" to deal with the "CLOSE_WAIT" status.

But, the most strange thing is the "ESTAB" state sockets which I can reproduce,

Seems when the client suddenly gone, the proxy took care and cleaned up sockets between the client and the proxy successfully, with no problem.

But, seems something wrong in the proxy, it even did not try to close the socket between the proxy and backend service, since both direction still in "ESTAB" status.

I use tcpdump and wireshark to see what happen for the "ESTAB" state dead socket,

it shows, the tomcat sent "TCP Window Full" to proxy and proxy replied "TCP ZeroWindow"

Afterwards, the tomcat kept sending "TCP Keep-Alive", and proxy kept replying "TCP ZeroWindow"

Any idea? suggestion?

Thanks in advance.

Steven.

[steven....@gmail.com]

Some updates.

I looked into the source code of HttpServerHandler.java, in run() method (line 79)

I can see when call process() (line 112), the boundConnection is NULL (since line 97 to 100 set it to NULL).

when the method process() still running, if I kill the client, that will trigger an exception, and reach the finally section(line 156)

since the local variable boundConnection is NULL, so will not try to close it, and current code does not check the  exchange.getTargetConnection().

to me seems it should check the exchange.getTargetConnection() as well, if not NULL, then try to close it.

I tried this fix, and it seems works, prevent me from reproducing the 2 "ESTAB" status dead socket when I kill the client.

But I am not clear if this is a bug in code or not, if there is a reason not checking the exchange.getTargetConnection() and if closing  exchange.getTargetConnection() in the finally section could introduce any other problem?, if there is a better way to handle this?

any idea?

Thanks.

here it the link to the source code on github:

https://github.com/membrane/service-proxy/blob/master/core/src/main/java/com/predic8/membrane/core/transport/http/HttpServerHandler.java#L79

[steven....@gmail.com]

Anybody have comments on this?

[steven....@gmail.com]

I checked the 4.2 code, there is no change in this area,

and I have the feeling that this is a bug,

since I played with all the setting of both tomcat and proxy side, can not avoid the reproducible problem of 2 "ESTAB" status dead socket as I mentioned in my previous post.

Should I raise a bug on github? is there any procedure I should follow?

Thanks,

[steven....@gmail.com]

I raised a issue on github for this problem:

https://github.com/membrane/service-proxy/issues/241