Remote client EJB call hangs when response length is greater or equal 64 KB

[Josef Straub]

Hi,
my remote EJB client using "remote+https" or "https-remoting" as protocols works fine with Wildfly versions 8 to 24 but not anymore with 25 or 26.
So, I removed the remote protocol and I see that Wildfly 25/26 starts using https protocol also for the EJB calls and at first sight seems to work fine, because I can see the HTTP requests/responses in my interceptor registered as Undertow filter and the client invocations succeed.
The problems start when the EJB response length is greater or equal than 64 KB (i.e 1024 * 64 bytes).
I have created an EJB accepting a string parameter and returning the same as result.
If I invoke this EJB with strings having length <= 65530, I see on server side that the HTTP response length is <=65535 bytes (5 bytes overhead).
This works fine even invoking many times the EJB
When I suddenly invoke the EJB with at least one more byte, so that response is reaches the 64 KB limit, the client freezes on reading the result.
I see that on server side the HTTP call is done and 65536 bytes has been returned, but the client hangs on reading them (unmarshalling the result).
In the client reading buffer I see exactly the expected result string, but still it hangs.
Does anyone know a solution for this?

[Josef Straub]

I have checked / debugged the Wildfly code and in my opinion there are several bugs in both the server and the client code of the HTTP2 protocol marshalling/unmarshalling implementation.

First bug: The server sends suddenly (mostly on bigger objects) in the middle of the data stream a FRAME_TYPE_RST_STREAM (i.e. see io.undertow.protocols.http2.Http2FrameHeaderParser:191 --> type = header[3] & 0xff;), 

which marks the stream as broken (see io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel:684 -->state |= STATE_STREAM_BROKEN;)

and afterwards causes "ClosedChannelExceptions" on stream reads in the client (see org.wildfly.httpclient.common.WildflyClientInputStream:58 -->int res = streamSourceChannel.read(pooled.getBuffer());)

Second bug (initiated by the above fact) : The client can't handle correctly this "ClosedChannelExceptions" (i.e reset stream), because it freezes. This should not happen under any circumstances!

So, client keeps reading from the channel and after each read it invokes the wait(0) on its own lock Object (see . org.wildfly.httpclient.common.WildflyClientInputStream:147 --> lock.wait();)

Normally this wait(0) is wake up by channel notifications on invoking the notifyAll() of same lock Object (in case of ClosedChannelExceptions it is the org.wildfly.httpclient.common.WildflyClientInputStream:94 --> lock.notifyAll();

Unluckily after the last read and therefore invoked wait(0), there is no more the notifyAll() invoked, so the EJB client call freezes for ever. This should not happen under any circumstances!

My solution I found is, and I have to admit I'm not happy with that, to disable the HTTP2 protocol on the server and use HTTP1.

<https-listener name="https" socket-binding="https"  ssl-context="CHA-SSL-CONTEXT" enable-http2="false" />

Making the EJB calls over HTTP/1 works then also for huge Objects.

Involved classes with line numbers are all taken from Wildlfy 26.0.0 Client jar:

I hope some Wildfly Guru will read this and react.

[Scott Marlow]

Please create a tracking WildFly issue via https://issues.redhat.com/browse/WFLY and be sure to update the forum link field to this conversation.  Please respond again here with the new WFLY issue link (so that we have links in both directions). 

You can also look at open (or recently closed) issues on https://issues.redhat.com/browse/WEJBHTTP (seems that you might need to search within various issue types there but some other issue types shown could be related).

Hope this helps,

Scott

[Josef Straub]

Thank you Scott, 

I have already reported it.

Link to the ticket [WFLY-15936] EJB over HTTP/2 doesn't work with huge objects - Red Hat Issue Tracker