Auto-forward (export) using STOW-RS fails after some time

[Marius S]

Hello,

We have two DCM4CHEE instances.
When studies received in first DCM4CHEE they are auto-forwarded/Exported to second DCM4CHEE instance.
For Export we have created stowrs2 device and configured STOW-RS network connection with Keylcoak user. We have created Export Rule and Exporter Descriptor.

Running on dockers:
dcm4chee-arc-psql:5.31.2-secure (we added more RAM -Xms512m -Xmx2048m)

dcm4che/keycloak:23.0.3

Export working fine if we send few studies to DCM4CHEE, but now we have sent many studies to first DCM4CHEE and there is 50000+ export tasks created as we send by Instances (Export Rule > Export Entity: Instance) and export fails to send to second DCM4CHEE instance.

Export task starts sending images to second DCM4CHEE, however after some time it stops. The logs show strange behavior. As indicated in log below there was an export task and it started on 09:48:48, and as you can see it stuck for 16 minutes and at 10:04:46 it showed the RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection timed out.

However the second DCM4CHEE is working fine at that time, I was able to access it.

And if we restart first DCM4CHEE docker container, it starts to send images again.

So the question where is it set to wait 16 minutes for a timeout and can we make it smaller?

Is this set in DCM4CHEE ui2, or Keycloak?
I have not found any setting with such parameter.

Is there any way to get more detailed log for this time out? 

Log:

2025-03-07 09:48:48,170 INFO  [org.dcm4chee.arc.qmgt.impl.TaskScheduler] (EE-ManagedExecutorService-default-Thread-6) Start processing Task{taskID=109557, deviceName='dcm4chee-arc', queueName='Export', type=EXPORT, status=IN PROCESS, scheduledTime=2025-03-06 15:49:55.739, numberOfFailures=0, exporterID=stowrs2exporter, studyInstanceUID=1.2.840.114350.2.487.2.798268.2.4552468031.1, seriesInstanceUID=1.3.46.670589.11.21359.5.0.7684.2023020308423942463, sopInstanceUID=1.3.46.670589.11.21359.5.0.7684.2023020308454287491, localAET=DicomSCP01}

.......

2025-03-07 10:04:46,713 WARN  [org.dcm4chee.arc.stow.client.impl.StowTaskImpl] (EE-ManagedExecutorService-default-Thread-6) org.jboss.resteasy.client.jaxrs.internal.ClientWebTarget@3c32fee: failed to send Instance[iuid=1.3.46.670589.11.21359.5.0.7684.2023020308454287491,cuid=1.2.840.10008.5.1.4.1.1.4] to WebApplication[name=stowrs2app,serviceClasses=[STOW_RS],path=/dcm4chee-arc/aets/DCM4CHEE/rs,aet=null,applicationClusters=[],keycloakClientID=stowuser,properties={},installed=null]:
: jakarta.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: java.net.SocketException: Connection timed out
at org.jboss.resteas...@6.2.6.Final//org.jboss.resteasy.client.jaxrs.engines.ManualClosingApacheHttpClient43Engine.invoke(ManualClosingApacheHttpClient43Engine.java:361)
at org.jboss.resteas...@6.2.6.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:427)
at org.jboss.resteas...@6.2.6.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:71)
at org.jboss.resteas...@6.2.6.Final//org.jboss.resteasy.client.jaxrs.internal.ClientInvocationBuilder.post(ClientInvocationBuilder.java:205)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear//org.dcm4chee.arc.stow.client.impl.StowTaskImpl.store(StowTaskImpl.java:181)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear//org.dcm4chee.arc.stow.client.impl.StowTaskImpl.runStoreOperations(StowTaskImpl.java:142)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear//org.dcm4chee.arc.stow.client.impl.StowTaskImpl.run(StowTaskImpl.java:122)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear//org.dcm4chee.arc.export.stow.StowExporter.export(StowExporter.java:97)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-export-mgt-5.31.2.jar//org.dcm4chee.arc.export.mgt.impl.ExportTaskProcessor.process(ExportTaskProcessor.java:117)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-export-mgt-5.31.2.jar//org.dcm4chee.arc.export.mgt.impl.ExportTaskProcessor$Proxy$_$$_WeldClientProxy.process(Unknown Source)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-qmgt-5.31.2.jar//org.dcm4chee.arc.qmgt.impl.TaskScheduler.processTask(TaskScheduler.java:214)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-qmgt-5.31.2.jar//org.dcm4chee.arc.qmgt.impl.TaskScheduler.processTasksSequential(TaskScheduler.java:149)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-qmgt-5.31.2.jar//org.dcm4chee.arc.qmgt.impl.TaskScheduler.process(TaskScheduler.java:129)
at deployment.dcm4chee-arc-ear-5.31.2-psql-secure.ear.dcm4chee-arc-qmgt-5.31.2.jar//org.dcm4chee.arc.qmgt.impl.TaskScheduler.lambda$process$0(TaskScheduler.java:118)
at org.jbo...@30.0.1.Final//org.jboss.as.ee.concurrent.ControlPointUtils$ControlledRunnable.run(ControlPointUtils.java:108)
at org.wildfly.secu...@2.2.2.Final//org.wildfly.security.auth.server.SecurityIdentity.runAsConsumer(SecurityIdentity.java:376)
at org.wildfly.secu...@2.2.2.Final//org.wildfly.security.auth.server.Scoped.runAsConsumer(Scoped.java:112)
at org.wildfly.secu...@2.2.2.Final//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:60)
at org.jbo...@30.0.1.Final//org.jboss.as.ee.concurrent.SecurityIdentityUtils.lambda$doIdentityWrap$1(SecurityIdentityUtils.java:50)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.glassfish.jakarta.enterprise.concurrent//org.glassfish.enterprise.concurrent.internal.ManagedFutureTask.run(ManagedFutureTask.java:119)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
at org.glassfish.jakarta.enterprise.concurrent//org.glassfish.enterprise.concurrent.ManagedThreadFactoryImpl$ManagedThread.run(ManagedThreadFactoryImpl.java:322)
Caused by: java.net.SocketException: Connection timed out

Thanks,
M.