Crash on outgoing messages from initiating gateway.

[jason.co...@gmail.com]

Hello,

I've recently upgraded to ipf 5, in a relatively high throughput environment, from ipf 3.  Running on Tomcat 10.1 with Java 17.  

This process performs iti55, 56, 38 and 39, and some VERY limited hl7v2 mllps transactions.  Using Spring configuration, and the same basic code structure as the iheclient tutorial. 

Since the upgrade, anywhere from a few hours up to roughly a day, I eventually get a crash with the following stacktrace:

05-Mar-2025 14:23:04.583 SEVERE [https-jsse-nio-11080-exec-14] org.apache.coyote.AbstractProtocol$ConnectionHandler.process Failed to complete processing of a request
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.base/java.lang.Thread.start0(Native Method)
at java.base/java.lang.Thread.start(Thread.java:809)
at java.net.http/jdk.internal.net.http.HttpClientImpl.start(HttpClientImpl.java:338)
at java.net.http/jdk.internal.net.http.HttpClientImpl.create(HttpClientImpl.java:272)
at java.net.http/jdk.internal.net.http.HttpClientBuilderImpl.build(HttpClientBuilderImpl.java:135)
at org.apache.cxf.transport.http.HttpClientHTTPConduit.lambda$setupConnection$1(HttpClientHTTPConduit.java:428)
at org.apache.cxf.transport.http.HttpClientHTTPConduit$HttpClientCache.computeIfAbsent(HttpClientHTTPConduit.java:195)
at org.apache.cxf.transport.http.HttpClientHTTPConduit.setupConnection(HttpClientHTTPConduit.java:428)
at org.apache.cxf.transport.http.asyncclient.AsyncHTTPConduit.setupConnection(AsyncHTTPConduit.java:191)
at org.apache.cxf.transport.http.HTTPConduit.prepare(HTTPConduit.java:551)
at org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(MessageSenderInterceptor.java:47)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:530)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:441)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:356)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:314)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:334)
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:320)
at org.openehealth.ipf.platform.camel.ihe.ws.SimpleWsProducer.callService(SimpleWsProducer.java:70)
at org.openehealth.ipf.platform.camel.ihe.ws.AbstractWsProducer.process(AbstractWsProducer.java:137)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:65)
at org.apache.camel.impl.engine.SharedCamelInternalProcessor.processNonTransacted(SharedCamelInternalProcessor.java:156)
at org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:133)
at org.apache.camel.impl.engine.SharedCamelInternalProcessor$1.process(SharedCamelInternalProcessor.java:89)
at org.apache.camel.impl.engine.DefaultAsyncProcessorAwaitManager.process(DefaultAsyncProcessorAwaitManager.java:82)
at org.apache.camel.impl.engine.SharedCamelInternalProcessor.process(SharedCamelInternalProcessor.java:86)
at org.apache.camel.support.cache.DefaultProducerCache.send(DefaultProducerCache.java:178)
at org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:176)
at org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:148)
at org.apache.camel.impl.engine.DefaultProducerTemplate.send(DefaultProducerTemplate.java:131)
at com.medent.medentgateway.ITI38Outgoing.send(ITI38Outgoing.java:232)
at com.medent.medentgateway.ITI38Outgoing.send(ITI38Outgoing.java:186)
at com.medent.medentgateway.ITI38Outgoing.createXCARequest(ITI38Outgoing.java:149)
at com.medent.medentgateway.EdgeListener.performIncomingMessage(EdgeListener.java:398)
at com.medent.medentgateway.EdgeListener.processRequest(EdgeListener.java:69)
at com.medent.medentgateway.EdgeListener.doPost(EdgeListener.java:130)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:590)
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.logging.log4j.web.Log4jServletFilter.doFilter(Log4jServletFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:663)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Thread.java:840)

I've attempted a few things to try to ameliorate this with no luck.  This is a relatively high throughput gateway, but it's not even close to my responding gateway, which went through the same upgrade and has had no isses.  It's maybe 1/1000th the amount of traffic.

I'm working with my sysadmin to try to get jstatd up and running so I can monitor, but while I wait for that, is there anything anyone here can think of that could be the issue?  I have noticed that the crash seems to happen only after iti-38 transactions.

One thing I have noticed is that it appears that there are 2 reflections to the wsdl for every transaction, per thread.

05-Mar-2025 14:22:49.412 INFO [https-jsse-nio-11080-exec-25] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti39.wsdl
05-Mar-2025 14:22:49.413 INFO [https-jsse-nio-11080-exec-25] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti39.wsdl
05-Mar-2025 14:22:49.962 INFO [https-jsse-nio-11080-exec-17] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti55/iti55-raw.wsdl
05-Mar-2025 14:22:49.964 INFO [https-jsse-nio-11080-exec-17] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti55/iti55-raw.wsdl
05-Mar-2025 14:22:54.032 INFO [https-jsse-nio-11080-exec-24] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti56/iti56-raw.wsdl
05-Mar-2025 14:22:54.034 INFO [https-jsse-nio-11080-exec-24] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti56/iti56-raw.wsdl
05-Mar-2025 14:22:54.153 INFO [https-jsse-nio-11080-exec-21] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti55/iti55-raw.wsdl
05-Mar-2025 14:22:54.155 INFO [https-jsse-nio-11080-exec-21] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xcpd:2009}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-hl7v3-5.0.0.jar!/wsdl/iti55/iti55-raw.wsdl
05-Mar-2025 14:22:57.728 INFO [https-jsse-nio-11080-exec-24] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti39.wsdl
05-Mar-2025 14:22:57.729 INFO [https-jsse-nio-11080-exec-24] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti39.wsdl
05-Mar-2025 14:22:57.819 INFO [https-jsse-nio-11080-exec-5] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti38.wsdl
05-Mar-2025 14:22:57.820 INFO [https-jsse-nio-11080-exec-5] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti38.wsdl
05-Mar-2025 14:22:57.822 INFO [https-jsse-nio-11080-exec-26] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti38.wsdl
05-Mar-2025 14:22:57.824 INFO [https-jsse-nio-11080-exec-26] org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL Creating Service {urn:ihe:iti:xds-b:2007}RespondingGateway_Service from WSDL: jar:file:/usr/INITGATEWAY/tomcat_installs/current_tomcat/InitGatewayProd/webapps/InitGateway/WEB-INF/lib/ipf-commons-ihe-xds-5.0.0.jar!/wsdl/iti38.wsdl

Any help at all is appreciated,

Jason

[jason.co...@gmail.com]

FYI, if anyone else has a similar issue.

This is caused by a bug in openjdk 17.

[JDK-8308364] HttpClientImpl SelectorManager thread never dies if response body not retrieved - Java Bug System

[CXF-8885] HttpClient SelectorManager threads run indefinitely causing OOM - ASF JIRA

I'm going to attempt the "force.urlconnection.http.conduit = true" fix in my environment, to see if that clears the issue up.  Working on the best place to put that setting, probably in my applicationContext.xml.

Thanks

Jason

[jason.co...@gmail.com]

Quick question, from the apache cxf site, does anyone know what this means and how it would apply?

"The java.net.http.HttpClient is reported to be considerably slower than java.net.HttpURLConnection (see please https://bugs.openjdk.org/browse/JDK-8277519), so there is an option to revert the default to URLConnectionHTTPConduit by setting "force.urlconnection.http.conduit" contextual property to true  ."

I've tried setting that as a switch to the java vm launch, which didn't work, and tried adding as a property to the http:conduit in my applicationContext.xml as well as a couple other places there with no luck.  I don't really do any Spring other than in relation to ipf, so any guidance and how to set this is very much appreciated.

[jason.co...@gmail.com]

Just to share, in case anyone else has this issue, I used a similar solution to one found in the comments here: 

public class URLConnectionHTTPConduitFactory implements HTTPConduitFactory {     @Override     public HTTPConduit createConduit(HTTPTransportFactory f, Bus bus, EndpointInfo endpointInfo, EndpointReferenceType target) throws IOException {         return new URLConnectionHTTPConduit(bus, endpointInfo, target);     } }

With this bean added to my applicationContext.xml:

<bean id="org.apache.cxf.transport.http.HTTPConduitFactory" class="package.of.my.URLConnectionHTTPConduitFactory" />

Note...making this change made the gateway MUCH more responsive, as well.  Besides just fixing the threads issue.