snapshot no longer work after upgrade from 1.6.2 to 1.6.5

[Qingyan Guan]

Hi Rick,

this is continue conversion regarding the upgrade.

we just noticed that the snapshot is not displaying after two weeks of upgrade.

I've searched through this group discussion regarding snapshot or pipeline, and tried all possible solution. But either worked for me.

I am assume if I installed fresh 1.6.5, it would work just fine. Since I upgraded from old version, and I left pipeline engine at its original place, it made the situation little complicated. 

here is setting in build.properties.

$XNAT_HOME: /var/xnat/xnat-1.6.5

$pipeline_HOME: /var/xnat/pipeline

after RUN setup.sh and update.sh, there is pipeline folder still generated under $XNAT_HOME/pipeline.

in Web interface, administrator -> configuration -> file system, the pipeline location stayed as original path /var/xnat/pipeline

with this setting, no snapshot were showing for newly archived MRsessions.

Then I changed administrator -> configuration -> pipeline location to /var/xnat/xnat-1.6.5/pipeline ($XNAT_HOME/pipeline), archived a new session, still no snapshot showing.

I also tried to add a pipeline to site and project to manually launch the WebQCImageCreator following the instruction:https://wiki.xnat.org/display/XNAT16/Installing+Pipelines+in+XNAT

and checked on automatic run pipeline on archive. then archive new session, no effect.

if I run manually from new added menu, it gives me error as "A pipeline is currently queued or running for this data. Please wait for the pipeline to complete"

any thoughts please?

[Herrick, Rick]

Look at the launch.log file. You should see the commands used to launch the pipelines logged in there. Try running those commands from the command line and see what happens. If there’s an error launching the pipeline (and there is: that’s how you get that “queued” message without the pipeline ever actually updating the status), you should be able to see it on the command line.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Qingyan Guan]

Which pipeline engine I should use?
One is outside of xnat home, and another one is inside of xnat home.

---

From: Herrick, Rick
Sent: ‎2/‎27/‎2016 3:01 PM
To: xnat_di...@googlegroups.com
Subject: Re: [XNAT Discussion] snapshot no longer work after upgrade from1.6.2 to 1.6.5

You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/PKNvttxqI5A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.

[Herrick, Rick]

I have no idea. Whichever one you recently installed, I’d guess.

One thing to note, there was a change in the pipeline engine with 1.6.5. We actually turned the downloaded engine into an installer. If you see a file named build.gradle in your pipeline folder, that’s the installer. It’s normally run during the setup process. The easy way to run it on its own is to copy the file named sample.gradle.properties in the pipeline installer folder to gradle.properties, including the destination property (which is where the pipeline engine will be installed). Set the properties in there appropriately then run this:

./gradlew

That will build the pipeline engine into the destination folder. At that point, all you should need to do is make sure your XNAT has the correct setting for the location of the pipeline engine in your configuration.

[Qingyan Guan]

I only see autoRun.xml get launched, which is transfer to archive action. No errors at all.
Is snapshot enabled by default, or it has to be configured somewhere?

---

From: Herrick, Rick
Sent: ‎2/‎27/‎2016 3:01 PM
To: xnat_di...@googlegroups.com

Subject: Re: [XNAT Discussion] snapshot no longer work after upgrade from1.6.2 to 1.6.5

You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/PKNvttxqI5A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.

[Herrick, Rick]

No, the autorun is what generates the snapshots and is run after a session is transferred to the archive. Autorun is enabled by default and the default autorun pipeline generates the snapshots.

What do you see in the launch.log file?

[Qingyan Guan]

this is what my launch.log looks like when I achieved a mr session from prearchieve. I also attached a pipeline log file under $XNAT_HOME/pipeline/logs/. there are errors.

====$TOMCAT/webapps/ROOT/logs/launch.log========

2016-02-27 11:46:02,798 Launching pipeline with command: /usr/local/xnat/xnat-1.6.5/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml -id xnatpro_E03301 -host https://XNAT.washington -u ac8807dc-1216-4244-86db-2b0e84801b2d -pwd ******** -dataType xnat:mrSessionData -label subject_01 -supressNotification -project "ANONPHILIPSVNQ" -notify kwoe...@u.washington.edu -notify qg...@uw.edu -parameter xnat_project='ANONPHILIPSVNQ' -parameter userfullname='Q.Guan' -parameter supressEmail='false' -parameter useremail='qg...@uw.edu' -parameter session='xnatpro_E03301' -parameter xnatserver='xnatpro' -parameter mailhost='localhost' -parameter sessionType='xnat:mrSessionData' -parameter adminemail='kwoe...@u.washington.edu' -parameter sessionLabel='subject_01' -workFlowPrimaryKey 42304 

2016-02-27 12:04:52,541 Launching pipeline with command: /usr/local/xnat/xnat-1.6.5/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml -id xnatpro_E03305 -host https://XNAT.washington.edu -u dbd0752b-a669-4cd8-909d-623f19521790 -pwd ******** -dataType xnat:mrSessionData -label subject_002 -supressNotification -project "ANONPHILIPSVNQ" -notify kwoe...@u.washington.edu -notify qg...@uw.edu -parameter xnat_project='ANONPHILIPSVNQ' -parameter userfullname='Q.Guan' -parameter supressEmail='false' -parameter useremail='qg...@uw.edu' -parameter session='xnatpro_E03305' -parameter xnatserver='xnatpro' -parameter mailhost='localhost' -parameter sessionType='xnat:mrSessionData' -parameter adminemail='kwoe...@u.washington.edu' -parameter sessionLabel='subject_002' -workFlowPrimaryKey 42310 

[Qingyan Guan]

when I run WebbasedQCImageCreator from command line, it gives me connection timed out error. 

============output starts here=========

xnat@XNATpro:~$ /usr/local/xnat/xnat-1.6.5/pipeline/image-tools/WebBasedQCImageCreator -

session subject_002 -project ANONPHILIPSVNQ -xnatId xnatpro_E03305 -host https://xnatpro

.ibic.washington.edu -u qguan -pwd PASSWORD

java.net.ConnectException: Connection timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)

        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)

        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)

        at java.net.SocksSocketImpl.connect(Unknown Source)

        at java.net.Socket.connect(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)

        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)

        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)

        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)

        at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

        at org.nrg.xnattools.SessionManager.createJSESSION(SessionManager.java:84)

        at org.nrg.xnattools.SessionManager.getJSESSION(SessionManager.java:136)

        at org.nrg.xnattools.service.WebServiceClient.connect(WebServiceClient.java:88)

        at org.nrg.plexiViewer.converter.WebBasedQCImageCreator.createQCImages(WebBasedQCImageCreator.java:141)

        at org.nrg.plexiViewer.converter.WebBasedQCImageCreator.main(WebBasedQCImageCreator.java:586)

====output end==================

=====output using the old pipeline engine===========

xnat@XNATpro:~$ /usr/local/xnat/pipeline/image-tools/WebBasedQCImageCreator -session subject_002 -project ANONPHILIPSVNQ -xnatId xnatpro_E03305 -host https://xnatpro.ibic.washington.edu -u qguan -pwd PASSWORD

/usr/local/xnat/pipeline/image-tools/WebBasedQCImageCreator: 3: /usr/local/xnat/pipeline/image-tools/WebBasedQCImageCreator: source: not found

0 GETJSESSION CALLED null

org.apache.http.conn.HttpHostConnectException: Connection to https://xnatpro.ibic.washington.edu refused

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)

        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)

        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)

        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)

        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)

        at org.nrg.xnattools.SessionManager.createJSESSION(SessionManager.java:64)

        at org.nrg.xnattools.SessionManager.getJSESSION(SessionManager.java:112)

        at org.nrg.xnattools.service.WebServiceClient.connect(WebServiceClient.java:88)

        at org.nrg.plexiViewer.converter.WebBasedQCImageCreator.createQCImages(WebBasedQCImageCreator.java:141)

        at org.nrg.plexiViewer.converter.WebBasedQCImageCreator.main(WebBasedQCImageCreator.java:587)

Caused by: java.net.ConnectException: Connection timed out

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)

        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)

        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)

        at java.net.SocksSocketImpl.connect(Unknown Source)

        at java.net.Socket.connect(Unknown Source)

        at sun.security.ssl.SSLSocketImpl.connect(Unknown Source)

        at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:549)

        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)

        ... 11 more

================output end===========

nothing showed up in launch.log.

On Saturday, February 27, 2016 at 6:05:31 PM UTC-8, Qingyan Guan wrote:

this is what my launch.log looks like when I achieved a mr session from prearchieve. I also attached a pipeline log file under $XNAT_HOME/pipeline/logs/. there are errors.

====$TOMCAT/webapps/ROOT/logs/launch.log========

2016-02-27 11:46:02,798 Launching pipeline with command: /usr/local/xnat/xnat-1.6.5/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml -id xnatpro_E03301 -host https://XNAT.washington -u ac8807dc-1216-4244-86db-2b0e84801b2d -pwd ******** -dataType xnat:mrSessionData -label subject_01 -supressNotification -project "ANONPHILIPSVNQ" -notify kwoe...@u.washington.edu -notify qg...@uw.edu -parameter xnat_project='ANONPHILIPSVNQ' -parameter userfullname='Q.Guan' -parameter supressEmail='false' -parameter useremail='qg...@uw.edu' -parameter session='xnatpro_E03301' -parameter xnatserver='xnatpro' -parameter mailhost='localhost' -parameter sessionType='xnat:mrSessionData' -parameter adminemail='kwoelfer@u.washington.edu' -parameter sessionLabel='subject_01' -workFlowPrimaryKey 42304 

2016-02-27 12:04:52,541 Launching pipeline with command: /usr/local/xnat/xnat-1.6.5/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml -id xnatpro_E03305 -host https://XNAT.washington.edu -u dbd0752b-a669-4cd8-909d-623f19521790 -pwd ******** -dataType xnat:mrSessionData -label subject_002 -supressNotification -project "ANONPHILIPSVNQ" -notify kwoe...@u.washington.edu -notify qg...@uw.edu -parameter xnat_project='ANONPHILIPSVNQ' -parameter userfullname='Q.Guan' -parameter supressEmail='false' -parameter useremail='qg...@uw.edu' -parameter session='xnatpro_E03305' -parameter xnatserver='xnatpro' -parameter mailhost='localhost' -parameter sessionType='xnat:mrSessionData' -parameter adminemail='kwoelfer@u.washington.edu' -parameter sessionLabel='subject_002' -workFlowPrimaryKey 42310 

[Herrick, Rick]

The pipeline task can’t reach the address https://xnatpro.ibic.washington.edu from wherever it’s running. This is a network or firewall issue, most likely. From the server where the pipeline task is being launched, you can try something like this:

curl -u username:password https://xnatpro.ibic.washington.edu/data/projects

You should see a response something like this:

{"ResultSet":{"Result":[{"secondary_ID":"XNAT 01","pi_lastname":"","description":"","name":"XNAT 01","ID":"XNAT_01","pi_firstname":"","URI":"/data/projects/XNAT_01"}], "totalRecords": "1"}}

You won’t though. You’ll get a timeout or some connection error.

[Qingyan Guan]

Yes, the server is located inside the firewall. Does this pipeline run on different port that I need to open it up for it? Good thing is the transfer runs ok. Is it using the same hostname?
What do you recommend to fix this issue?

---

From: Herrick, Rick
Sent: ‎2/‎28/‎2016 12:20 PM
To: xnat_di...@googlegroups.com

[The entire original message is not included.]

[Herrick, Rick]

The pipeline doesn’t run “on a port” as such. It’s just making an http request—or more correctly an https request—to that address. I have no idea about the hostnames or where the firewall is between wherever the pipeline is running and the XNAT server or any of that. That’s completely dependent on your deployment and where things are located.

If you do need a port opened in the firewall, then it would just be port 80 for http or port 443 for https. If you have a network admin, I’d just have that person have a look at it.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 362-1882

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Qingyan Guan]

What I try to say is the xnat server is located behind firewall. It's been configured accessible from public networks.
It was running fine before upgrade, so I don't think it's issue related to firewall configuration. I think I must've messed something out during update.
Do you recommend to reinstall the pipeline alone to default location?

---

From: Herrick, Rick
Sent: ‎2/‎29/‎2016 10:08 AM

[Herrick, Rick]

I don’t think the location of the pipeline engine is particularly relevant in this case. The pipeline execution seems to start up just fine, which is really the main thing that changing the location would affect. It’s just that, when it’s run from that location, it can’t reach the URL for some reason and the call is timing out.

Did you try running that curl command on the server where the pipeline engine is running?

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Qingyan Guan]

Yes, it gives me "couldn't connect to host".
If I run
Curl -u user:password http://localhost:8080/data/projects/

It outputs the full list of projects.

---

From: Herrick, Rick
Sent: ‎2/‎29/‎2016 10:55 AM

  &