pipelines fail with self-signed cert

[Miklos Espak]

Hi there,

we have an XNAT site that uses self-signed certificate. I try to enable the pipeline engine for it, but the pipelines fail with an exception like this:

FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - Couldnt search for queued workflows
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 faultActor:
 faultNode:
 faultDetail:
        {http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1902)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1338)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:154)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1032)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1328)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
        at org.apache.axis.components.net.JSSESocketFactory.create(JSSESocketFactory.java:186)
        at org.apache.axis.transport.http.HTTPSender.getSocket(HTTPSender.java:191)
        at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:404)
        at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
        at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
        at org.apache.axis.client.Call.invoke(Call.java:2767)
        at org.apache.axis.client.Call.invoke(Call.java:2443)
        at org.apache.axis.client.Call.invoke(Call.java:2366)
        at org.apache.axis.client.Call.invoke(Call.java:1812)
        at org.nrg.xnattools.xml.AbsService.createServiceSession(AbsService.java:137)
        at org.nrg.xnattools.xml.XMLSearch.searchAll(XMLSearch.java:101)
        at org.nrg.pipeline.client.XNATPipelineLauncher.isPipelineQueuedOrAwaitingOrOnHold(XNATPipelineLauncher.java:281)
        at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:63)
        at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:231)
        at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:221)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1320)
        ... 26 more

Is there a way to get around this?

E.g. could I somehow tell the pipeline engine to use http instead https without changing to http globally in the build.properties?

This is XNAT 1.6.1.

Cheers,
Miklos

[Herrick, Rick]

I think you just need to install your cert into the keystore for the JVM that your pipeline execution is using. This is sort of described here:

 

https://groups.google.com/forum/#!searchin/xnat_discussion/pkix/xnat_discussion/BbNispEicdc/zPSYRXDFnyEJ

 

I say “sort of” because there’s a whole step about pulling the cert from the site itself using OpenSSL that you shouldn’t need to go through. You might need to do a little bit of conversion to get your key installed in the server in a format that the Java keystore is happy with, but then again maybe not. It’s been a while since I’ve done that, so I’m not really sure what the Java keytool application requires.

 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Miklos Espak]

It seems that I already imported the certificate in the past because I got this message when I wanted to do it now:

[tomcat@cmic-xnat ~]$ keytool -import -file cert.crt
Enter keystore password: 
Certificate already exists in keystore under alias <tomcat>
Do you still want to add it? [no]:  no
Certificate was not added to keystore

I modified the scripts in the pipeline engine's bin directory by adding these parameters:

-Djavax.net.ssl.keyStore=/opt/tomcat/.keystore -Djavax.net.ssl.keyStorePassword=********

but the error is the same.

I put an echo in the XnatPipelineLauncher script to see the java command that it executes and tried to run that command directly. It is the same, of course.

Is it possible that somewhere internally it runs a process in a new JVM and does not pass down the keystore options? It does not seem very likely, but I have no other idea at the moment.

It would also be a solution if the pipeline engine used the http protocol without encryption, since that is on localhost, anyway. But I do not see any way to set that.

Thanks,
Miklos

[Simon Doran]

Hi Miklos,

  I had exactly this problem at the last installation that I performed. This is a particular nuisance even if one doesn't explicitly use additional analysis pipelines, because the auto run pipeline, which generates the image thumbnails, fails. So an immediate symptom is that data uploads "succeed", but with no thumbnails and a message like this

AutoRun:QueuedStart Time: 2014-10-06 15:23:52.044[Mark as Failed]

on the session report page.

  I didn't dig very deeply, but it seemed that the underlying java class just didn't like self-signed certificates. I need to go back and revisit the topic, because I also remember writing other code of my own that failed with self-signed certificates. I found only two ways around it:

Method 1: Purchase a genuine SSL certificate signed by an appropriate authority. They are not overly expensive and this also has the benefit that your users won't get scared off by modern browsers telling them that your XNAT site is a potentially security risk, with at least one browser displaying "Help! Get me out of here!" when the user first logs on.

Method 2: As a short term fix and not recommended on production systems, compromise by setting security.channel=any in the file Tomcat/webapps/<your XNAT>/WEB-INF/conf/services.properties. Then restart Tomcat. This allows an alternate login via http (typically on port 8080) and allows the pipelines to run. Just don't tell your users and make sure you always give them the https URL.

  Hope that helps.

  Best wishes,

Simon

[Miklos Espak]

Hi Simon, Rick

thanks for the answer.

I've managed to sort out the issue, finally.

I did the followings. Not sure if each step is required.

I created a root CA and generated a leaf cert for the site with openssl.

Installed the Tomcat Native Connector.

Modified the tomcat server.xml to use the native connector:

<Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

Specified the certificate files in the connector instead of the keystore, as earlier.

           SSLCertificateFile="..."

           SSLCertificateKeyFile="..."

           SSLCertificateChainFile="..."

For the pipeline engine, I imported the site certificate file into the Java trust store, and specified the trust store options in the pipeline launcher scripts.

Then the ssl handshake succeeded.

I'm not sure if the separate root CA and leaf cert has solved the issue or the native connector, but it works now.

Thanks for all the help,

Miklos

On 7 October 2014 08:55, Simon Doran <simon...@icr.ac.uk> wrote:

Hi Miklos,

  I had exactly this problem at the last installation that I performed. This is a particular nuisance even if one doesn't explicitly use additional analysis pipelines, because the auto run pipeline, which generates the image thumbnails, fails. So an immediate symptom is that data uploads "succeed", but with no thumbnails and a message like this

AutoRun:QueuedStart Time: 2014-10-06 15:23:52.044[Mark as Failed]

on the session report page.

  I didn't dig very deeply, but it seemed that the underlying java class just didn't like self-signed certificates. I need to go back and revisit the topic, because I also remember writing other code of my own that failed with self-signed certificates. I found only two ways around it:

Method 1: Purchase a genuine SSL certificate signed by an appropriate authority. They are not overly expensive and this also has the benefit that your users won't get scared off by modern browsers telling them that your XNAT site is a potentially security risk, with at least one browser displaying "Help! Get me out of here!" when the user first logs on.

Method 2: As a short term fix and not recommended on production systems, compromise by setting security.channel=any in the file Tomcat/webapps/<your XNAT>/WEB-INF/conf/services.properties. Then restart Tomcat. This allows an alternate login via http (typically on port 8080) and allows the pipelines to run. Just don't tell your users and make sure you always give them the https URL.

  Hope that helps.

  Best wishes,

Simon

On Monday, October 6, 2014 2:45:08 PM UTC+1, Miklos Espak wrote:

[Sven Prevrhal]

Thanks for the posts on this topic - currently stuck there too. I don't actually know why since pipelines have worked for me in the past. I issued the offending XNATRestClient call on the command line, and get the same SSL related error. 

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I followed Rick's advice to grab the cert and add it via keytool, but the error remains the same. I wonder if the solution really has to be as involved as described by Miklos, Has anybody found anything simpler or has an idea?

Thanks

Sven

[Sven Prevrhal]

On a further note, can somebody shed light on the following:

The log shows the XNATRestClient wants to connect via https, but the web client's "Workflow Details" page shows aliasHost http://localhost:8080/xnat/. When I try XNATRestClient on the command line with localhost:8080. it works just fine! How do I force the XNATRestClient to actually use the aliasHost? Is this the right way?

Thanks,Sven

[Sven Prevrhal]

OK, this is a conversation between me and myself...:-)

So I see now that XnatRestClient uses the host that is specified in my pipeline's main XML file in the parameters block:

<argument id="host">

<value>^/Pipeline/parameters/parameter[name='host']/values/unique/text()^</value>

</argument>

I would have hoped that specifying localhost:8080 as aliasHost for the XnatPipelinelauncher (with -useAlias) would also change above host parameter, but the error log clearly shows it doesn't. Replacing the host parameter in all XnatRestClient steps works, so that's good! However, this makes the script less general, which is a small drawback.Is there a solution to this?

[Flavin, John]

I have a design pattern I try to use in pipelines where I expect “aliasHost” may be used. I define a parameter in the Parameters section called “resolved_host” (or resolvedHost; I am not very consistent between camelCase and underscores) as such

<parameter>
    <name>resolved_host</name>
    <values>
        <unique>^if(boolean(/Pipeline/parameters/parameter[name='aliasHost'])) then /Pipeline/parameters/parameter[name='aliasHost']/values/unique/text() else /Pipeline/parameters/parameter[name='host']/values/unique/text()^</unique>
    </values>
</parameter>

“resolved_host” takes the value of “aliasHost” if it exists, otherwise it takes the value of “host”, which we know will always be there because it is required. Then when I need to pass a host to any resource in a step, I pass “resolved_host”. 

Flavin

CNDA Pipelines Developer

fla...@mir.wustl.edu

@cndapipelines

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Sven Prevrhal]

Thanks John,

This looks good - I will try.

Sven

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/0dmtn56Ru18/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.