org.apache.http.conn.HttpHostConnectException: Connection refused

[Kate Alpert]

Dear all,

Every so often, our pipelines fail because they suddenly cannot connect to our XNAT server (stack trace below).  Is there anything we can do about this?  Perhaps change the pipeline code to try connecting more than once before it fails?

Thanks,

Kate

Unable to store workflow with http://nunda.northwestern.edu:8080/nunda/

org.apache.http.conn.HttpHostConnectException: Connection to http://nunda.northwestern.edu:8080 refused

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)

        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)

        at org.nrg.xnattools.xml.WorkflowStore.store(WorkflowStore.java:71)

        at org.nrg.xnattools.xml.WorkflowStore.store(WorkflowStore.java:96)

        at org.nrg.pipeline.client.XNATPipelineLauncher.update(XNATPipelineLauncher.java:163)

        at java.util.Observable.notifyObservers(Observable.java:159)

        at org.nrg.pipeline.manager.EventManager.notify(EventManager.java:35)

        at org.nrg.pipeline.utils.AdminUtils.fireNotification(AdminUtils.java:44)

        at org.nrg.pipeline.manager.ExecutionManager.execute(ExecutionManager.java:134)

        at org.nrg.pipeline.task.StepManager.execute(StepManager.java:166)

        at org.nrg.pipeline.task.StepManager.execute(StepManager.java:153)

        at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:213)

        at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:193)

        at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:70)

        at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:292)

        at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:254)

Caused by: java.net.ConnectException: Connection timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)

        at java.net.Socket.connect(Socket.java:579)

        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)

        ... 20 more

Unable to store workflow with http://nunda.northwestern.edu:8080/nunda/

[Kate Alpert]

Any ideas?

[Herrick, Rick]

It’s not that I’ve been ignoring you but…

Well, yes it is that I’ve been ignoring you, but I’ve been ignoring you for a legitimate reason: I don’t have an answer for you :)

Basically, yes, you’re correct that the pipeline code should do something to connect more than once and/or recover from transient or intermittent failure conditions. The problem is simple on the face of it—keep trying until you connect—but not quite as straightforward as you’d like. Really the ability to recover, especially from failures in HTTP transactions, should be configurable and manageable somehow. This is somewhat complicated by the fact that the code that’s managing that is pretty old and uses an old and deprecated API for the HTTP transaction management.

In the short term, there really isn’t a way to work around this unfortunately (or if there is, I’m not privy to it). In the medium term, we need to update our code to use the newer code library and add some default retry behavior. Longer term, it’d be nice to have the various retry parameters (e.g. how many retries, how long between retries, what to do once you’ve failed out of retries) configurable. I’ve added an issue to our JIRA (https://issues.xnat.org/browse/XNAT-3586) to track this issue.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Kate Alpert]

Thanks, Rick, I understand.  

Do you (or Flavin) know how CNDA handles this?  I get the sense that they're running a good number of pipelines...