Problem with pipelines

Sven Prevrhal

unread,

Sep 4, 2014, 8:24:58 AM9/4/14

to xnat_di...@googlegroups.com

I installed the WebBasedQCImageCreator.xml pipeline according to the wiki https://wiki.xnat.org/display/XNAT16/Installing+Pipelines+in+XNAT (which, BTW, seems to inconsistently first guide towards the WebBasedQCImageCreator, but then talks on about the DicomToNifti pipeline), but when I triggered it on an MR dataset, I got the "queued" message, then nothing. So I looked in /home/xnat/pipelines/logs and found this (The full log file below):

FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - Couldnt search for queued workflows

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

My address is https (dev.pf-xchg.de/xnat), which I access through nginx to forward it as http to Tomcat, so I figure it might something to do with the security certificates. They are valid, and do work for the https, though. Do I need to register them with Tomcat, or is there something else going on?

Best regards

Sven

=============

root@rs210205:/home/xnat/pipelines# ls logs/

pipeline_2014_04_01_12_21_05.log pipeline_2014_04_02_08_53_34.log

pipeline_2014_04_02_08_46_59.log pipeline_2014_04_04_16_24_35.log

pipeline_2014_04_02_08_52_42.log pipeline_2014_04_04_16_37_25.log

pipeline_2014_04_02_08_52_50.log pipeline_2014_04_04_16_57_14.log

pipeline_2014_04_02_08_52_59.log pipeline_2014_04_04_17_03_23.log

pipeline_2014_04_02_08_53_08.log pipeline_2014_04_04_17_19_08.log

pipeline_2014_04_02_08_53_17.log pipeline_2014_09_04_12_51_54.log

pipeline_2014_04_02_08_53_25.log pipeline_2014_09_04_12_56_33.log

root@rs210205:/home/xnat/pipelines# less logs/pipeline_2014_09_04_12_56_33.log

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Initializing the pipeline

properties from the specified properties object

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Setting admin email to: x

natadmin

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Setting email ID to: xnat

admin

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Setting SMTP host to: loc

alhost

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Setting REST mail host to

: https://dev.pf-xchg.de/xnat/data/services/mail/send

FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - Couldnt search for que

ued workflows

javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.ja

va:397)

at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.jav

a:126)

at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFact

ory.java:572)

at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnect

ion(DefaultClientConnectionOperator.java:180)

at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedCli

entConnectionImpl.java:294)

at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)

at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)

at org.nrg.xnattools.SessionManager.createJSESSION(SessionManager.java:64)

at org.nrg.xnattools.SessionManager.getJSESSION(SessionManager.java:112)

at org.nrg.xnattools.service.WebServiceClient.connect(WebServiceClient.java:84)

at org.nrg.pipeline.client.XNATPipelineLauncher.isPipelineQueuedOrAwaitingOrOnHold(XNATPipelineLauncher.java:355)

at org.nrg.pipeline.client.XNATPipelineLauncher.isPipelineQueuedOrAwaitingOrOnHold(XNATPipelineLauncher.java:344)

at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:76)

at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:290)

at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:255)

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre https://dev.pf-xchg.de/xnat/ [https://dev.pf-xchg.de/xnat/]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre 24e03ff2-49dd-42db-b2aa-872e60a29122 [24e03ff2-49dd-42db-b2aa-872e60a29122]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre 1409828191947 [1409828191947]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre smtp.gmail.com [smtp.gmail.com]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre x.xnatadmin [x.xnatadmin]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre /home/xnat/build/TEST_datexch/20140904_125631 [/home/xnat/build/TEST_datexch/20140904_125631]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre DRTHERAPAT-DEV [DRTHERAPAT-DEV]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ad...@dev.pf-xchg.de [ad...@dev.pf-xchg.de]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre x...@xxx.com [x...@xxx.com]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre 202 [202]

:DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ANONYMIZE [ANONYMIZE]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre DRTHERAPAT_DEV_E00023 [DRTHERAPAT_DEV_E00023]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre TEST_datexch [TEST_datexch]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre 1 [1]

INFO main org.nrg.pipeline.utils.PipelineEngineUtils - Parameter Resolved

INFO main org.nrg.pipeline.utils.PipelineEngineUtils - Loop Resolved

INFO main org.nrg.pipeline.utils.PipelineEngineUtils - Output resolved

INFO main org.nrg.pipeline.task.StepManager - Step attributes resolved

INFO main org.nrg.pipeline.manager.ResourceManager - Loaded /home/xnat/pipelines/catalog/images/resources/WebBasedQCImageCreator.xml

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='sessionLabel']/values/unique/text()^ [ANONYMIZE]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='session']/values/unique/text()^ [DRTHERAPAT_DEV_E00023]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='xnat_project']/values/unique/text()^ [TEST_datexch]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^if (count(/Pipeline/parameters/parameter[name='aliasHost']/values) > 0) then /Pipeline/parameters/parameter[name='aliasHost']/values/unique/text() else /Pipeline/parameters/parameter[name='host']/values/unique/text()^ [https://dev.pf-xchg.de/xnat/]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='user']/values/unique/text()^ [24e03ff2-49dd-42db-b2aa-872e60a29122]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='pwd']/values/unique/text()^ [1409828191947]

INFO main org.nrg.pipeline.manager.ResourceManager - Loaded /home/xnat/pipelines/catalog/images/resources/WebBasedQCImageCreator.xml

INFO main org.nrg.pipeline.task.StepManager - Step attributes resolved

INFO main org.nrg.pipeline.manager.ResourceManager - Loaded /home/xnat/pipelines/catalog/notifications/Notifier.xml

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='user']/values/unique/text()^ [24e03ff2-49dd-42db-b2aa-872e60a29122]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='pwd']/values/unique/text()^ [1409828191947]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='useremail']/values/unique/text()^ [sven.p...@philips.com]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='adminemail']/values/unique/text()^ [ad...@dev.pf-xchg.de]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^concat('XNAT update: ', /Pipeline/parameters/parameter[name='sessionLabel']/values/unique/text(),'

snapshot generation complete')^ [XNAT update: ANONYMIZE snapshot generation complete]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='mailhost']/values/unique/text()^ [smtp.gmail.com]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^concat('Dear ',/Pipeline/parameters/parameter[name='userfullname']/values/unique/text(),',<br> <p> Snapshots have been generated for ', /Pipeline/parameters/parameter[name='sessionLabel']/values/unique/text(), 'XNAT Team.')^ [Dear x.xnatadmin,<br> <p> Snapshots have been generated for ANONYMIZEXNAT Team.]

INFO main org.nrg.pipeline.manager.ResourceManager - Loaded /home/xnat/pipelines/catalog/notifications/Notifier.xml

INFO main org.nrg.pipeline.manager.ExecutionManager - Launching step 0

FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated

Sven Prevrhal

unread,

Sep 4, 2014, 9:33:59 AM9/4/14

to xnat_di...@googlegroups.com

BTW, my nginx conf is

server {

ssl on;

ssl_certificate /etc/ssl/certs/www.domain.tld.crt;

ssl_certificate_key /etc/ssl/private/www.domain.tld.key;

listen 443 ssl;

server_name .dev.pf-xchg.de;

client_max_body_size 4g;

location /xnat {

index index.html;

root /var/www/;

proxy_pass http://dev.pf-xchg.de:8080/xnat;

# proxy_redirect http://dev.pf-xchg.de http://dev.pf-xchg.de:8080

/xnat;

}

Sven

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='useremail']/values/unique/text()^ [x...@xxx.com]

DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='adminemail']/values/unique/text()^ [ad...@dev.pf-xchg.de]
DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^/Pipeline/parameters/parameter[name='adminemail']/values/unique/text()^ [ad...@dev.pf-xchg.de]
DEBUG main org.nrg.pipeline.xpath.XPathResolverSaxon - Expre ^concat('XNAT update: ', /Pipeline/parameters/parameter[name='sessionLabel']/values/unique/text(),'

Herrick, Rick

unread,

Sep 4, 2014, 10:10:14 AM9/4/14

to xnat_di...@googlegroups.com

You do need to register the certificate, not with Tomcat as such, but with the JDK/JRE that the pipeline is using. Java has its own certificate store, usually stored in the file JAVA_HOME/jre/lib/security/cacerts. The procedure to extract the certificate directly from your server is described here:

https://groups.google.com/forum/#!searchin/xnat_discussion/pkix/xnat_discussion/BbNispEicdc/zPSYRXDFnyEJ

There’s also a link on that post to Oracle’s docs for importing the certificate itself into the JRE’s certificate store. You’ll want to specify the cacerts as the option for the -keystore option. The default cacerts password is changeit, so your command would look something like (this is using the path to cacerts on our CentOS servers):

keytool -import -alias dev -file cert.crt -keystore /usr/lib/jvm/java-1.7.0-oracle.x86_64/jre/lib/security/cacerts

You can use whatever value you want for the alias. It’s mainly there for easy reference later.

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Herrick, Rick

unread,

Sep 4, 2014, 10:12:00 AM9/4/14

to xnat_di...@googlegroups.com

See my previous reply for more details, but I think since you have the cert right on your local store, you could do this:

keytool -import -alias dev -file /etc/ssl/certs/www.domain.tld.crt -keystore /usr/lib/jvm/java-1.7.0-oracle.x86_64/jre/lib/security/cacerts

In other words, you don’t need to go through the open_ssl step to import the certificate, but should be able to import the cert directly off your primary certificate file.

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

From: xnat_di...@googlegroups.com [mailto:xnat_di...@googlegroups.com] On Behalf Of Sven Prevrhal

Sent: Thursday, September 04, 2014 8:34 AM
To: xnat_di...@googlegroups.com

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

Sven Prevrhal

unread,

Sep 4, 2014, 4:12:03 PM9/4/14

to xnat_di...@googlegroups.com

Thanks a lot Rick, I followed this, meaning, I copy-pasted your line and changed it to

keytool -import -alias dev -file /etc/ssl/certs/www.domain.tld.crt -keystore /usr/lib/jvm/java-7-oracle/jre/lib/security/cacerts

which is the keystore on my Ubuntu system. I asked me for password and then whether I trust the certificate, and then said it added it to the keystore. Great.

Except, The pipeline still doesn't work. I cleaned out all queued workflows directly with psql and then uploaded a dataset to trigger the AutoRun workflow. I see it stuck at Queued, Here's the XML, perhaps you can see what's wrong. I don't get an error anymore in the pipelines/logs/, in fact, I don't get a new log file at all, neither in the Tomcat logs, where I do see the new inserts but nothing else. This is also true when I run the WebBasedQCImageCreator pipeline - just stuck at Queued, no errors at the two places. I also added the tail of catalina.out. I also don't see the nice thumbnails in the session scans list, that's related to AutoRun not properly executing right?

Sven

===================================

XML for the AutoRun:

XML for the WebBasedQCImageCreator:

catalina.out

Initializing PipelineRepository...Finished loading the pipeline repository!