28.0.0 Sporadic NPE in HttpServletRequest\getParameterMap() during multi-part upload

[Chris Palmer]

We upgraded from 27.0.1 to 28.0.0 a few days ago. Although the upgrade/testing went smoothly, in production we saw a small number of NPE during multi-part uploads, and have had to revert to 27.0.1.

The start of the traceback is:

2023-05-01 15:37:41,611 ERROR [stderr] (default task-2) java.lang.NullPointerException: Cannot invoke "io.undertow.connector.PooledByteBuffer.getBuffer()" because "this.pooled" is null
2023-05-01 15:37:41,611 ERROR [stderr] (default task-2)         at io.undert...@2.3.5.Final//io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:202)
2023-05-01 15:37:41,611 ERROR [stderr] (default task-2)         at io.undert...@2.3.5.Final//io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
2023-05-01 15:37:41,611 ERROR [stderr] (default task-2)         at io.under...@2.3.5.Final//io.undertow.server.handlers.form.MultiPartParserDefinition$MultiPartUploadHandler.parseBlocking(MultiPartParserDefinition.java:226)
2023-05-01 15:37:41,611 ERROR [stderr] (default task-2)         at io.undert...@2.3.5.Final//io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:877)
2023-05-01 15:37:41,611 ERROR [stderr] (default task-2)         at io.undert...@2.3.5.Final//io.undertow.servlet.spec.HttpServletRequestImpl.getParameterMap(HttpServletRequestImpl.java:829)

The uploads were 1-5 MB in size. The application servlet responds to the HTTP POST by extracting some values from the request, then calling getParameterMap().

HAProxy logs suggest that the requests may have been in progress for a little over 90 seconds.It is hard to tell if slow or unreliable uplinks from the clients may be aggravating the problem.

The application code was not changed across the upgrade apart from being recompiled. Other supporting non-wildfly libraries were not changed. The wildfly configuration was identical.

Any ideas?

Thanks, Chris

[Bartosz Baranowski]

Could you please file WFLY ticket with simple reproducer?
Im fairly sure this code has not changed recently( there is one PR that Im aware of).

[Chris Palmer]

Unfortunately we are unable to replicate this at will, and are still no nearer being able to provide any more information. I do suspect it is timing-dependent and potentially related to interrupted uploads. It's also possible that haproxy is somehow contributing. With a frequency in production of about one per day it is too frequent to leave live, but too infrequent to know how to diagnose. Despite numerous attempts I've been unable to provoke it myself which makes trying to create a simple reproducer very hard.

I've created a WFLY ticket with this info in  https://issues.redhat.com/browse/WFLY-18029