1.6.3 - AutoRun pipeline stays "Queued"

[Torsten Rohlfing]

Hi -

After upgrading to 1.6.3 (including the pipeline fixes released later), the AutoRun jobs for my uploaded and archived sessions stay "Queued" permanently and never run.

Where would I start looking for the cause of this?

Thanks!

Torsten

[Mohana Ramaratnam]

Hello

Look in TOMCAT_HOME/webapps/XNAT_PROJECT_HOME/logs/application.log

Copy the command string from the log file and manually launch it and see where the issue is

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Torsten Rohlfing]

That file is empty - zero file length, and timestamp from last server restart (yesterday). I archived two sessions from the pre-archive since, and they both show AutoRun as Queued, but no activity in this log file.

The only log files with any activity during the archiving are prearchive.log, access.log, and velocity.log, but none show anything related to the pipeline.

TR

[Jordi]

Hi there,

Any update on that topic? I think I am experiencing the same issue in a freshly updated v1.6.3...Torsten did you managed to solve it?

cheers,

Jordi

[Torsten Rohlfing]

Hi Jordi -

No, there has been no development, further insight, or solution here.

Sorry.

Torsten

[Mohana Ramaratnam]

Jordi, Torsten:

Things to check:

a) Using admin account, login to XNAT instance, under confuguration check if the path to the installation of pipeline engine is correct. Unless you have made changes, pipeline engine is installed in PATH_TO_XNAT_HOME/pipeline

b) Check is tomcat user has permissions into execute in PIPELINE_HOME/bin

c) As mentioned in the earlier post, Jordi - check TOMCAT_HOME/webapp/XNAT_RPOJECT/logs/application.log (Torsten does not have any logging info in this file). 

d) Also check catalina logs

Mohana

[Jordi Huguet]

Thanks Mohana, 

Everything looks OK but logs appear in catalina.out not in application.log.

In my case I think is an SSL-related issue, as stated at the pipeline engine specific logs: 

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

...

I used a self-signed SSL certificate for my Apache server and added it as a trustStore option to the pipeline call, but it is not trusting it I believe. 

Anyway at least I could copy paste the pipeline call from Tomcat logs and execute it manually using tomcat user and localhost:8080 instead of my HTTPS-based site URL.

Jordi

[Herrick, Rick]

There appears to be an issue with logging to certain logs in 1.6.3, so that's why you're seeing the logging going to the wrong place. We'll have someone look into it.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[David Gutman]

I forgot to post this ... but here's some code that will trigger the
pipelines for anything that is missing snapshots (using pyxnat)
I'll post this as a GIST or to my github repo as well....


### create a credentials file so you don't have to include your
password info if you use GITHUB or whatever

### xnat_credentials.py file
xnat_server = 'SERVER IP'
xnat_username = XNAT_USER
xnat_password = XNAT_PASSWORD
xnat_cachedir = '/path/to/cache/for/pyxnat'


### trigger_snapshot_creation.py
import pyxnat as xnat
import xnat_credentials as xnc
import os.path as op
import sys, time
import time
xnat = xnat.Interface(server=xnc.xnat_server,user=xnc.xnat_username,password=xnc.xnat_pass,cachedir=xnat_cachedir)

def check_for_snapshots( subj, expt, project):
"""This will scan a session to see if snapshots are available, and
return True/False if it finds them..
In some cases a scan may exist that has no snapshot, so I will
have to set a threshold/ % that would suggest
the Autorun pipeline properly generated webQC Images--- some
images by default do not generate snapshots such as phoenix
reports and other type of non-image scans"""
expt_obj = xnatvault.select.project(Default_Project).subject(subj).experiment(expt)
scans_found = 0
scans_w_snapshot = 0
for scan in expt_obj.scans():
a = scan.resource('SNAPSHOTS')
if a.id():
scans_w_snapshot +=1
scans_found += 1
print "Found a total of %d snapshots for %d scans" % (
scans_w_snapshot,scans_found)
return scans_w_snapshot



Default_Project = 'PROJECT_TO_CHECK_FOR_SNAPSHOTS'
subj_list = xnatvault.select.project(Default_Project).subjects().get('label')

for subj in subj_list:
expt_list =
xnatvault.select.project(Default_Project).subject(subj).experiments().get('label')
for expt in expt_list:
snapshots_found = check_for_snapshots( subj, expt, Default_Project)
if not snapshots_found:
print "NO snapshots found for",subj,expt
expt_obj =
xnatvault.select.project(Default_Project).subject(subj).experiment(expt)
expt_obj.trigger_pipelines()
time.sleep(2) ### inserting a small delay so I don't queue
up 100's of processes
### may want to throttle this/add ap ause as it doens't
seem to queue things but tries to run everything at once

David A Gutman, M.D. Ph.D.
Assistant Professor of Biomedical Informatics
Senior Research Scientist, Center for Comprehensive Informatics
Emory University School of Medicine

[Herrick, Rick]

That's wicked awesome, David! Thanks a lot for that. Unfortunately we don't have a gist and bitbucket doesn't have a gist-like capability. I'll figure out a way to pull these all together though...

Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
(314) 827-4250

[David Gutman]

I can post it to my XnatView Repository---- I'll send a link tonight
or tomorrow for those who are interested..

[Herrick, Rick]

OK, I figured out the logging problem. A character got truncated from the logging properties configuration file, which breaks the default logger. Luckily, this is very easily fixed:

* Open the file WEB-INF/conf/log4j.properties in your deployed XNAT web application
* Go to line 13, which will look like this: og4j.rootLogger = ERROR, app
* Add the letter 'l' to the beginning of that line, so that it reads: log4j.rootLogger = ERROR, app
* Save the file
* Restart XNAT

You can also change the source for this configuration file by editing the file plugin-resources/originals/log4j.properties. Subsequent updates will get the fixed version of the properties file.

Once you have fixed that issue, you should be able to run back through your pipeline launch process and this time get the actual logging output in the application.log as expected. Once you have that, you can get the command used to launch the pipeline. Try running that from the command line and you should get some informative output from the pipeline execution.

Rick Herrick
Sr. Programmer/Analyst
Neuroinformatics Research Group
Washington University School of Medicine
(314) 827-4250

-----Original Message-----
From: xnat_di...@googlegroups.com [mailto:xnat_di...@googlegroups.com] On Behalf Of Torsten Rohlfing
Sent: Tuesday, April 08, 2014 12:13 AM
To: xnat_di...@googlegroups.com
Subject: Re: [XNAT Discussion] 1.6.3 - AutoRun pipeline stays "Queued"

Torsten

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Herrick, Rick]

BTW, the download versions of XNAT on the FTP site have this issue fixed. The virtual machines do not, since it's non-trivial to update those images.

[Torsten Rohlfing]

Okay, so now that I have the logging problem fixed, I can see the failing pipeline launch in application.log:

2014-04-08 15:49:42,127 [Thread-17] ERROR org.nrg.pipeline.ProcessLauncher -  Couldnt launch /home/xnat/xnat/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml -id NeuroXNAT_E00054 -host https://###/xnat -u ### -pwd ### -dataType xnat:mrSessionData -label ### -supressNotification -project "test" -notify ### -parameter xnat_project='test' -parameter userfullname='###' -parameter supressEmail='false' -parameter useremail='###' -parameter session='NeuroXNAT_E00054' -parameter xnatserver='NeuroXNAT' -parameter mailhost='localhost' -parameter sessionType='xnat:mrSessionData' -parameter adminemail='###' -parameter sessionLabel='###' -workFlowPrimaryKey 422

Running the script with the given parameters by hand from a shell, I get this error:

Param Value Pair aliasHost=http://localhost:8080/xnat
....
Param Value Pair workflowid=422
Exception in thread "main" java.lang.NoClassDefFoundError: org/springframework/util/MultiValueMap
        at org.nrg.pipeline.utils.MailUtils.setMailService(MailUtils.java:150)
        at org.nrg.pipeline.client.CommandLineArguments.<init>(CommandLineArguments.java:241)
        at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:252)
Caused by: java.lang.ClassNotFoundException: org.springframework.util.MultiValueMap
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 3 more

Any ideas?

Thanks!

Torsten

[Herrick, Rick]

There's either a messed up class path in your XnatPipelineLauncher script or you're missing a file. In the standard 1.6.3 release, the class org.springframework.util.MultiValueMap is in the jar file spring-core-3.0.7.RELEASE.jar located in the pipeline engine's lib folder. That file is in turn referenced in the classpath of all of the scripts that use it. In this specific case, it should be in the file bin/XnatPipelineLauncher.

So:

1. Verify that spring-core-3.0.7.RELEASE.jar is in the folder lib.
2. Verify that bin/XnatPipelineLauncher contains some path like PIPELINE_HOME/lib/spring-core-3.0.7.RELEASE.jar, where PIPELINE_HOME is not the literal value but instead wherever your pipeline is located.

One of those two things is not the case.

Well, there's an outside chance that the jar is corrupted. I just ran an MD5 checksum on my copy of the library and got this:

feeca5dd71af07bda262b0ed14dc1951  spring-core-3.0.7.RELEASE.jar

Check on those things and let me know what you find.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

From: Torsten Rohlfing <torsten...@gmail.com>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>

Date: Tuesday, April 8, 2014 5:53 PM
To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] 1.6.3 - AutoRun pipeline stays "Queued"

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[David Gutman]

Torsten--- did you install 1.6.3 form scratch or attempt to do an
update from bitbucker repository? When I tried the latter I ran into
tons of issues with the classpath not being sent properly in some of
the modules....

[Torsten Rohlfing]

I have the jar; but here's the closest match in XnatPipelineLauncher:

/home/xnat/xnat-1.6.3/pipeline/lib/org.springframework.core-3.0.5.RELEASE.jar

I just re-created this using

pipeline/setup.sh

But got the same result.

My build.properties has only this to say about the pipeline location:

#xdat.pipeline.location=/Library/XNAT/PIPELINE

So I am not sure where the apparently outdated paths might be from?

Thanks Rick!

Torsten

[Torsten Rohlfing]

I got the .tar.gz from the website and installed from that.

[Herrick, Rick]

Have a look at the file templates/etc/bin/XnatPipelineLauncher in your pipeline engine. The templates folder is the source for scripts that get built out into the bin, catalog, and other resource folders in the pipeline engine. If the template version of the script isn't properly updated, then you'll get the messed up class path in your bin folder.

I've checked the download version of the pipeline engine and it's got the appropriate class path set in the script template and the right version of the jar in the lib folder, so is it possible you extracted your 1.6.3 builder over a previous version?

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 827-4250

From: Torsten Rohlfing <torsten...@gmail.com>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Date: Tuesday, April 8, 2014 7:28 PM
To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] 1.6.3 - AutoRun pipeline stays "Queued"

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Tim Olsen]

Generally, when I upgrade a pipeline, I can’t just pull the changes into the existing (pre-built) pipeline engine and run setup.  I need to install the pipeline from scratch.  I often do that by moving the existing pipeline folder to a backup location.  Pull a fresh copy of the pipeline code from bitbucket and put it in the same directory/name that the old one was.  Then run the setup script.

 

I think once you’ve run the setup script on a pipeline engine, it can’t be run again on that instance.

 

I’ve missed some of the dialogue on this thread, so I apologize if you already tried a fresh install of the pipeline.

 

Tim

[Torsten Rohlfing]

Well, I am 100% sure that I did not copy anything from our previous install into the tree that I extracted from the downloaded 1.6.3 release...

That said, I just re-extracted and took the fresh pipeline/ tree, then ran the pipeline setup, and what do you know ... now the pipeline works again :)

So for us at least, that means problem solved.

Thanks everyone for helping us track this down!

Torsten

[Zaivets2A]

Hi Mohana:

"installed in PATH_TO_XNAT_HOME/pipeline

b) Check is tomcat user has permissions into execute in PIPELINE_HOME/bin"

My pipeline folder has only one folder with xdat-beans-1.6.4.jar file

/Libs/XNAT/pipeline/lib/xdat-beans-1.6.4.jar

Here is the fragment of my build.properties file:

# Location of your pipeline installation.

# If you have pipeline installed outside of your xnat_builder folder, un-comment this property to set\

 it

# the appropriate location. As with all folder locations, these paths should be absolute, not relativ\

e to

# your xnat_builder location.

# avz - uncommented

xdat.pipeline.location=/Libs/XNAT/pipeline

Shall I re-build it with commented pipeline notation?

Thank you,

Alex Z

[Mohana Ramaratnam]

Hi Alex,

You would need to download the pipeline engine in the location pointed to in the build.properties file and rerun the xnat setup. 

(XNAT setup copies the xdat-beans jar into the PIPELINE_HOME/lib folder and invokes the pipeline engine setup). 

(You could, alternatively, run only the PIPELINE_HOME/setup.sh script with appropriate input arguments, after you download the pipeline engine. If you go down this path, you would have to back-up the xdat-beans jar and copy it into PIPELINE_HOME/lib after the PIPELINE_HOME/setup.sh is run).

--