"OutOfMemoryError: Direct Buffer" with WLFY 23

[Sebastian Schnabl]

The Wildfly 23.0.2.Final we are using is failing in the undertow component  with an "OutOfMemoryError: Direct buffer memory" after server was running for some time (< 4 weeks uptime).  Please see attached stacktrace: 

ERROR [io.undertow.request.io] (default I/O-2) UT005090: Unexpected failure: java.lang.OutOfMemoryError: Direct buffer memory
    at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
    at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
    at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
    at org.jbo...@3.8.4.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
    at org.jbo...@3.8.4.Final//org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
    at org.jbo...@3.8.4.Final//org.xnio.ByteBufferSlicePool.allocateSlices(ByteBufferSlicePool.java:162)
    at org.jbo...@3.8.4.Final//org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149)
    at io.under...@2.2.5.Final//io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:166)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:749)
    at io.under...@2.2.5.Final//io.undertow.conduits.HeadStreamSinkConduit.flush(HeadStreamSinkConduit.java:131)
    at org.jbo...@3.8.4.Final//org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
    at io.under...@2.2.5.Final//io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)
    at io.under...@2.2.5.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1793)
    at io.under...@2.2.5.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1763)
    at io.under...@2.2.5.Final//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:205)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http2.Http2UpgradeHandler.handleRequest(Http2UpgradeHandler.java:102)
    at io.under...@2.2.5.Final//io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:62)
    at io.under...@2.2.5.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:255)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
    at io.under...@2.2.5.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
    at org.jbo...@3.8.4.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
    at org.jbo...@3.8.4.Final//org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
    at org.jboss...@3.8.4.Final//org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.jboss...@3.8.4.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

I'm aware of https://issues.redhat.com/browse/XNIO-346 but this should be addressed in WLFY 23, since XNIO 3.8.4.Final is shipped .  Anyone an idea?

Regards,

Sebastian

[Felix Feisst]

We are encountering the exact same issue. We are also running on Wildfly 23.0.2.Final, after a few weeks of run time, some of our instances can encounter OutOfMemory: Direct buffer memory errors. In a default configuration, the direct buffer memory can grow the same size as the configured heap memory. We do monitoring of the direct memory now (via mbean), such that we are able to restart production instances over night whenever direct buffer memory gets into a critical region.

I'm pretty sure, that the direct buffer memory problem has started to show up in our application after we've introduced websockets. Do you use websockets, too? Can you run tests without websockets and verify that direct buffer memory is no longer growing?

Some hints how to further tackle this issue would be very much appreciated.

Regards,

Felix

[Cameron Rodriguez]

WildFly Core and most other components use the latest version of XNIO, but Elytron is still on 3.7.2. I attempted to reproduce the issue using the steps in XNIO-346, but was unable to. It's possible that it might appear in a different manner now that it doesn't affect the rest of the server.

Could you send the server configuration? I'll try to reproduce the issue on my end.

Cameron

[Felix Feisst]

I have attached the standalone-full.xml. Do you need further configuration?

Thanks,

Felix

[Cameron Rodriguez]

Hi Felix, thanks for sending the config over. I can't really verify if there are any issues with your specific configuration because I can't load some of the dependencies (mainly opus2).

I wasn't able to reproduce the issue, however it looks similar to UNDERTOW-1935, which was fixed in a later version. If you are able to, try upgrading WildFly and see if the issue reappears.

Best,
Cameron

[Felix Feisst]

Hi Cameron, thanks for the hint with the undertow issue. I have upgraded undertow to 2.2.10 in our wf23 installation. Monitoring on our test environments show that this fixes the issue. We will rollout the updated version to production this week end and we will further monitor direct buffer memory consumption. But I'm very confident that it will be back to normal usage.

I think this issue is resolved. Thanks again for your help!

Best,

Felix

[Sebastian Schnabl]

Hello.

thanks for the hint to UNDERTOW-1935. @Felix Good to hear that upgrading the undertow component would fix this issue. On our system we found another workaround for this: we switched in undertow from default  <buffer-cache name="default"/>  to the new <byte-buffer-pool> implementation, but with flag "direct=true":

/subsystem=undertow/byte-buffer-pool=BBP:add()

/subsystem=undertow/byte-buffer-pool= BBP:write-attribute(name=direct,value=true)

/subsystem=undertow/server=default-server/http-listener=default:write-attribute(name=buffer-pool,value=BBP)

@Cameron

In the docs i could read: 

>> NOTE

>> IO buffer pools are deprecated in JBoss EAP 7.2. While they are still set as the default in the current release, they will be replaced by Undertow byte buffer pools in a future release.

Do you have an idea, why this still not is not changed in default configuration files provided by WLFY?

Regards,

Sebastian

caro...@redhat.com schrieb am Freitag, 8. Juli 2022 um 22:55:04 UTC+2:

[Cameron Rodriguez]

Hey, glad it helped!

I reached out to the Undertow team, and they mentioned that there were some issues with the Undertow pools, and XNIO had become more stable in the meantime (here's the thread). If the team finds that things have become more stable, they'll make the upgrade in a future WildFly version.

Best,
Cameron

[serg.k...@gmail.com]

Hi All

We are doing migration to WFLY 26 (domain mode) and unfortunately got into same issue for the Undertow load balancer.

Configuration like : /profile=load-balancer/subsystem=io/buffer-pool=default:write-attribute(name=direct-buffers,value=false) is helped, but i am not sure it is proper solution.

Thanks,

BR, Sergii Kostenko

вівторок, 12 липня 2022 р. о 20:04:59 UTC+1 caro...@redhat.com пише: