XnatPipelineLauncher SSLHandshakeException

[Sanket Gupte]

Heylo,

Wasn't using pipelines on this instance of my xnat, until now. And I just realized all the Autorun pipelines are queued. So, I killed them all. Mercilessly. "delete from wrk_workflowdata where status = ‘Queued’;"

Then, I was pretty sure that my pipeline path and everything is correct. So grepped for "pipeline.ProcessLauncher" in application.log* of tomcat webapp.
Got this "ERROR org.nrg.pipelineProcessLauncher - Couldn't launch " blah blah.
So, I went ahead and executed the "blah blah" .
And it gave me the following.

AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 faultActor:
 faultNode:
 faultDetail:
        {http://xml.apache.org/axis/}stackTrace:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:279)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1446)
*** etc


I am not quite sure how to fix this, and any help would be appreciated. I did look at these posts.
https://groups.google.com/d/msg/xnat_discussion/0dmtn56Ru18/oUqTon70oDgJ
and
https://groups.google.com/d/msg/xnat_discussion/FNVYWNYdddA/Pn0kYMIwtOIJ

But couldn't really find a fix for my problem.

N.B. : I have tomcat with apache and modjk . And run ssl on apache.

Thanks a lot
Sanket
 

[Herrick, Rick]

The JVM that is running your pipeline doesn’t have the certificate you’ve configured for your Apache installed in its keystore. You basically need the answer to this post:

http://stackoverflow.com/questions/9619030/resolving-javax-net-ssl-sslhandshakeexception-sun-security-validator-validatore

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Sanket Gupte]

Okay so I did that.
And now i get a different error.

AxisFault
 faultCode: {http://xml.apache.org/axis/}HTTP
 faultSubcode:
 faultString: (404)Not Found
 faultActor:
 faultNode:
 faultDetail:
        {}:return code:  404

        {http://xml.apache.org/axis/}HttpErrorCode:404

(404)Not Found
        at org.apache.axis.transport.http.HTTPSender.readFromSocket(HTTPSender.java:744)
        at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
* etc


Did I do something wrong ?

[Herrick, Rick]

I don’t think so. It looks like it can’t find the resource on the server, which indicates that it’s getting past the SSL handshake. Without knowing which URL is failing it’s hard to say why you might be getting the 404. If that was from, e.g., your AutoRun pipeline, it probably indicates that the resource it’s trying to access is missing, but again, hard to say without more context.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

From: Sanket Gupte <gupte...@umbc.edu>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Sanket Gupte]

Oh.. Amm.. yes it was autorun pipeline..

Here is more detail. I just cut short the error, thought that was enough.. but may be the following detail might help ?




Param Value Pair host=https://cerebra.nih.gov/
Param Value Pair u=c5f05923-bf68-476f-bd59-475ab90bd53c
Param Value Pair pwd=********
Param Value Pair xnat_project=457
Param Value Pair userfullname=A.Admin
Param Value Pair supressEmail=false
Param Value Pair useremail=sanket...@nih.gov
Param Value Pair session=Cerebra_E01601
Param Value Pair xnatserver=Cerebra
Param Value Pair mailhost=<>.nih.gov
Param Value Pair sessionType=xnat:mrSessionData
Param Value Pair adminemail=sanket...@nih.gov
Param Value Pair sessionLabel=182698-1
Param Value Pair pwd=********
Logging to File /opt/xnat/pipeline_1_6/logs/pipeline_2015_05_19_12_20_08.log

AxisFault
 faultCode: {http://xml.apache.org/axis/}HTTP
 faultSubcode:
 faultString: (404)Not Found
 faultActor:
 faultNode:
 faultDetail:
        {}:return code:  404

        {http://xml.apache.org/axis/}HttpErrorCode:404

(404)Not Found
        at org.apache.axis.transport.http.HTTPSender.readFromSocket(HTTPSender.java:744)
        at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

        at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
        at org.apache.axis.client.Call.invoke(Call.java:2767)
        at org.apache.axis.client.Call.invoke(Call.java:2443)
        at org.apache.axis.client.Call.invoke(Call.java:2366)
        at org.apache.axis.client.Call.invoke(Call.java:1812)
        at org.nrg.xnattools.xml.AbsService.createServiceSession(AbsService.java:137)
        at org.nrg.xnattools.xml.XMLSearch.searchAll(XMLSearch.java:101)
        at org.nrg.pipeline.client.XNATPipelineLauncher.isPipelineQueuedOrAwaitingOrOnHold(XNATPipelineLauncher.java:281)
        at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:63)
        at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:231)
        at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:221)
Couldnt connect to host https://cerebra.nih.gov/

[Herrick, Rick]

I’ll bet your alias token is expired. Check your database:

select * from xhbm_alias_token where alias = 'c5f05923-bf68-476f-bd59-475ab90bd53c’;

Or you can try running the script from the command line but substituting an actual username and password in place of the alias token and secret if your security permits that.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

From: Sanket Gupte <gupte...@umbc.edu>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Date: Tuesday, May 19, 2015 at 1:33 PM
To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] XnatPipelineLauncher SSLHandshakeException

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Sanket Gupte]

You lose your bet :D :D

So, I archived an actual scan. The autorun got queued. So, I went to the back end, and ran the XNATPipelineLanucher script/code again, to get the above mentioned error.

And even now, the alias token still exists in xhbm_alias_token :D

Any suggestions ?

[Flavin, John]

Look in the log file /opt/xnat/pipeline_1_6/logs/pipeline_2015_05_19_12_20_08.log. There may be more detail of what the pipeline was trying to do and why it failed.

Flavin

CNDA Pipelines Developer

fla...@mir.wustl.edu

@cndapipelines

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Sanket Gupte]

Would the pipeline stay queued if the mail isn't getting sent ??

May be that's queuing it ?

DEBUG main org.nrg.pipeline.utils.PipelineProperties - Setting REST mail host to: https://cerebra.nida.nih.gov/data/services/mail/send
FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - Couldnt search for queued workflows

AxisFault
 faultCode: {http://xml.apache.org/axis/}HTTP
 faultSubcode:
 faultString: (404)Not Found
 faultActor:
 faultNode:
 faultDetail:
        {}:return code:  404

        {http://xml.apache.org/axis/}HttpErrorCode:404

(404)Not Found
        at org.apache.axis.transport.http.HTTPSender.readFromSocket(HTTPSender.java:744)
        at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

WARN main org.springframework.web.client.RestTemplate - POST request for "https://cerebra.nida.nih.gov/data/services/mail/send" resulted in 404 (Not Found); invoking error handler


ERROR main org.nrg.pipeline.utils.MailUtils - Message failed to send through REST service, retrying with direct SMTP.
org.springframework.web.client.HttpClientErrorException: 404 Not Found
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:75)
        at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:486)

[Rick Herrick]

Check your server access logs, both on Tomcat and Apache. The fact that you're getting a 404 on both that Axis call and the mail send indicates that there aren't errors with the services, but with how your pipeline is addressing the server itself. In other words, I think the URL is bad. Either Apache can't find the resource on Tomcat or Tomcat can't find the URL within the application.

The reason the pipeline is staying queued is because the pipeline can't call back to XNAT at all to update the pipeline workflow status.

[Sanket Gupte]

Okay. So I checked Tomcat and APache access logs.

These are the things that I thought were relevant.

cat /<>/tomcat7/logs/localhost_access_log.2015-05-19.txt | grep POST| grep 404
127.0.0.1 - - [19/May/2015:12:15:34 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:15:35 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:15:35 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:15:35 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:15:36 -0400] "POST /data/services/mail/send HTTP/1.1" 404 -
127.0.0.1 - - [19/May/2015:12:20:10 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:20:11 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:20:11 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:20:11 -0400] "POST /axis/CreateServiceSession.jws HTTP/1.0" 404 -
127.0.0.1 - - [19/May/2015:12:20:11 -0400] "POST /data/services/mail/send HTTP/1.1" 404 -

 I couldn't find any such 404 things for apache.

Now, I tried doing this. So, my url is htts://cerebra.nida.nih.gov 
So, based on what i see above , I did "htts://cerebra.nida.nih.gov/axis/CreateServiceSession.jws"  in the browser ..
And it gave

There is a Web Service here

Which means, the url was valid, and there is no 404.
Then I did this "htts://cerebra.nida.nih.gov/dataservices/mail/send"
And got

The method specified in the request is not allowed for the resource identified blah blah

which means, this is also a valid url, and it's finding everything as it should.

Then I did "htts://cerebra.nida.nih.gov/blah" .. coz you know, what the hell.
And got

The server has not found anything matching the request URI

Which means a 404 . So, you know .. it's all good.

Now this was accessing from outside, the xnat machine.

Now I did the same from the xnat machine, as "localhost" instead of "htts://cerebra.nida.nih.gov"  .. and the results were exactly .. "EXACTLY" the same.
------------

So....  is there any other way of finding .. or figuring out, why or if, there is a problem with finding the URL.. ?

[Flavin, John]

I think the mail send failure is a red herring. What is the “-host” argument in your pipeline launch string?

[Sanket Gupte]

The host argument seems right. Its "https://cerebra.nida.nih.gov".
Not sure what else to do :(

[Flavin, John]

Can you send me your /opt/xnat/pipeline_1_6/logs/pipeline_2015_05_19_12_20_08.log file? It is possible that you should not if there is any sensitive information in there, but perhaps you can scrub some things. It just seems that it may be easier for me to figure out if I can poke through the log myself.

Flavin

[Sanket Gupte]

Attached. There aren't sensitive things. It was a fake data. :D And the server is internal access only.

Thanks a lot.

[Flavin, John]

Your pipeline engine can’t connect to your XNAT. I can’t be sure why that is. If you can log on to your pipeline execution server and just try to do some curl calls to your XNAT server, you may find that it isn’t connecting. That’s where I would focus my debugging efforts right now. But that also isn’t anything I can help you with if you do find a problem.

Flavin

> On May 21, 2015, at 12:43 PM, Sanket Gupte <gupte...@umbc.edu> wrote:
>
> Attached. There aren't sensitive things. It was a fake data. :D And the server is internal access only.
>
> Thanks a lot.

> <pipeline_2015_05_19_12_20_08.log>

[Herrick, Rick]

Yeah, Flavin and I were discussing this yesterday and are pretty certain
that there¹s something fishy going on with your server and/or
infrastructure. Those 404s indicate that the call is making it to some
server somewhere: it¹s an active acknowledgement of the HTTP request and
indicates that it doesn¹t have the thing you¹re asking for. If the server
was down or unreachable or in any way non-functional, you¹d get a
completely different error. But you¹re also getting a 404 for every
request you make, indicating that the mapping of URLs for a functional
XNAT isn¹t happening somehow.

I tried the same operations that you did to my local dev server and got
these results:

POST http://xnatdev.wurstworks.com/axis/CreateServiceSession.jws

Server nginx/1.4.6 (Ubuntu)
Date Wed, 20 May 2015 23:32:25 GMT
Content-Length 0
Connection keep-alive
Set-Cookie JSESSIONID=85E00A918B59436632B2502402FB6DD2; Path=/; HttpOnly
Location
http://xnatdev.wurstworks.com/app/template/Login.vm;jsessionid=85E00A918B59
436632B2502402FB6DD2


That seems sensible: I¹m posting that request but I didn¹t provide any
credentials, so it wants me to log in. If I add BASIC auth credentials, I
get a 500, which again makes sense because I haven¹t provided a SOAP
request for the Web service.

So if your XNAT is working when you go to that URL from the browser on
your desktop but then gives you these kinds of responses when the pipeline
runs, the only thing I can think of is that the IP routing is messed up
somehow and the pipeline is actually going to another machine somewhere
that has no idea what those URL mappings are. Regardless, this is 99%
likely not an XNAT issue and is instead an issue in Tomcat or network
configuration or something like that.

--
Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
(314) 740-5961

>--
>You received this message because you are subscribed to the Google Groups
>"xnat_discussion" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to xnat_discussi...@googlegroups.com.
>To post to this group, send email to xnat_di...@googlegroups.com.
>Visit this group at http://groups.google.com/group/xnat_discussion.
>For more options, visit https://groups.google.com/d/optout.