Pipeline Questions
Sanket Gupte
So I am trying to create a freesurfer pipeline for my xnat deployment. I found a recon-all.xml file in the catalog directory of the pipeline. I have read everything on pipelines given in the documentation and also heard the talk by Mohana.But I am still not clear about it. I am sorry, to be bugging you guys for this.
Starting with the important point first-
1. I am not able to start a new pipeline on any of my projects because some AutoRun.xml pipeline is Queued for each of the data. How can i stop/remove this pipeline? Why did the autorun pipeline start by itself? Or did I do something to enable it. Its on all the data that came into xnat (through C-store recv)
2. The recon-all.xml in the config/freesurfer_tools seems like a resource descriptor doc. Is there an example of pipeline descriptor for freesurfer in the pipeline directory of the deployment? If not, can someone pls send me a freesurfer pipeline descriptor and if it has a template file already made, then it would be an icing to the cake.
3. I think there is a minor documentation error on <http://docs.xnat.org/Integrating+Pipeline+Engine+and+XNAT>, it begins with adding WebbasedQCImageCreator.xml and then ends up adding DiconToNifti.xml. Or am I perceiving it in a wrong way.
Thank you very much
Sanket
Herrick, Rick
The easiest way to remove the Queued pipelines from the project is by removing the rows from the database table, e.g. in psql:
delete from wrk_workflowdata where status = ‘Queued’;
Be careful with this because of course if you have a legitimate workflow queued this could cause problems.
The autorun pipeline runs because that’s just what it does: runs automatically. I *believe* that you can’t stop the autorun pipeline from running automatically, but you can modify what autorun does by editing the catalog/xnat_tools/AutoRun.xml file under your pipeline installation.
The recon-all.xml descriptor (and most everything else under the catalog folder) is NOT an actual pipeline configured to use the Freesurfer pipeline integration. Each file under catalog is actually a resource descriptor that configures a tool you can use within a pipeline definition.
Unfortunately, I don’t have any examples of a Freesurfer pipeline configuration. I was actually bemoaning my lack of knowledge about how to make that work this very morning. I know some people do have it configured and working though. If anyone has a sample configuration that they can post (stripping out any sensitive data, e.g. passwords or secure account data, etc.), that’d be great. If not, I’ll try to see if I can dig some stuff out and post up on the XNAT site.
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/xnat_discussion/-/1pa8opl8OrAJ.
To post to this group, send email to
xnat_di...@googlegroups.com.
To unsubscribe from this group, send email to
xnat_discussi...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/xnat_discussion?hl=en.
The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
Sanket Gupte
The pipeline concept is a bit tricky to grasp in the beginning. As a matter of fact, my boss allotted me this project, just because he couldn't figure out the pipeline things. :)
And yes, I would really appreciate if someone attaches an entire Freesurfer pipeline here.
Mohana Ramaratnam
This is going to be a long reply so relax, sit back and enjoy the flight (!)
a) Issues with AutoRun - being Queued
Looks like AutoRun is being triggered by is not launched. Checklist:
i) Is Pipeline engine installed correctly? Does
PIPELINE_HOME/bin/XnatPipelineLauncher exist with execute permissions
for the user starting tomcat process? Does PIPELINE_HOME/logs folder
exist?
ii) Is the PIPELINE_HOME location correct in WEBAPP -> Administer ->
Default Settings - check Pipeline Installation Location. Typically
this would be XNAT_HOME/pipeline
iii) Check TOMCAT_HOME/webapps/YOUR_XNAT_APP_NAME/logs/application.log
- is there any error entry here of the type
org.nrg.pipeline.ProcessLauncher - Couldnt launch ......
iv) If there is, copy the string starting at
PIPELINE_HOME/bin/XnatPipelineLauncher and run it manually on the
command-prompt. What error message do you get?
v) What is in the folder PIPELINE_HOME/logs?
b) Use of AutoRun pipeline
As you have figured out, its supposed to run automatically on
archival. This pipeline generates the snapshots for each of the scans
in your imaging session. It also makes a REST call to launch all
pipelines associated with the project which are marked as "run
automatically on archival". It then sends email notification.
c) Pipeline 101:
When you write a shell script, you have:
i) Input parameters supplied as constants within the script, supplied
as command line arguments or you can source a file.
ii) There are steps in a particular order and logic (typically if
statements, loops) to invoke executables. For each executable that the
script invokes, the executable is located in a particular location,
there is an order in which the arguments are supplied and also a
possible prefix to the argument option (eg. -a MY_A_VALUE or --a
MY_A_VALUE or /a MY_A_VALUE etc).
iii) Its the responsibility of the shell to process a shell script.
XNAT Pipelines consist of a pipeline engine and a collection of XML
documents viz. pipeline XML, resource XML and parameter XML. The
pipeline XML and the resource XML typically are in
PIPELINE_HOME/catalog. XNAT pipeline engine is the interpreter which
processes a pipeline XML (ala shell script) and knows (using the
resource XML) how to construct a string which would result in
invocation of the executable. The input arguments to the Pipeline (ie
the pipeline XML) can be supplied in-line (ie constants within the
pipeline document) or through the parameter XML or name value pairs
as command line arguments to the pipeline engine.
So pipeline engine would need to know path to pipeline XML, input
parameters to the pipeline, path to parameter XML document (optional)
and path to the catalog folder. XNAT pipeline engine comes in two
flavors - one which updates XNAT workflow as each step is executed and
the other which processes the steps unaware of the presence of XNAT.
PIPELINE_HOME/bin/XnatPipelineLauncher is the pipeline engine with
XNAT workflow update capability
and
PIPELINE_HOME/bin/PipelineRunner is the engine unaware of XNAT.
iv) What does it take to create a XNAT pipeline:
Pipelines in XNAT would be triggered either by the user clicking
Action -> Build link or automatically on archival.
*) You need to identify which executable(s) (resource) you need to
invoke and in which order. You describe the executable in a resource
XML. You describe the order of invocation in the pipeline XML.
*) You need to provide the input parameters to the pipeline. Pipeline
engine can resolve XNAT XPATH statements and extract the input
parameters.
*) A pipeline would generate some data (a collection of images and
measures). You need to know how you would want to represent the
measures in XNAT. Do you need to create a new schema to model the
result of a pipeline? If yes, you create the schema.
v) Developer tasks for enabling XNAT pipeline
*) If input arguments to pipeline are to be selected using some logic,
you would need to create a velocity page (ui), a screen class and an
action class to launch the pipeline. The action class would write a
parameter document and invoke the XnatPipelineLauncher. The Site
administrator while adding the pipeline to the site repository, would
specify the velocity page which is to be rendered when the user clicks
Action -> Build -> Particular Pipeline
*) If the input arguments to pipeline can be extracted from the
session XML using XNAT XPATH statements, there wont be a need for a
velocity page, a screen class and an action class. A default page is
rendered to the user and an inbuilt action class would generate the
parameter XML and invoke XnatPipelineLauncher.
*) As a step in the pipeline, generate the XML instance of the new
schema representation and upload the XML and the associated data to
XNAT. We are the process of creating importer service which would be
capable of generating the XML and storing the data.
*) Create a page which would display the results of the pipeline and
create the display document which would show the results (aka listing)
in the project page.
e) Freesurfer pipeline:
At WashU, we have chosen to represent the data generated by Freesurfer
as an image assessor. The freesurfer pipeline used here at WashU,
contains input parameters which are specified (see input-parameter
section) using XNAT XPATH. So there is no velocity page, screen class
or action class required. I am attaching the required documents. You
will not be able to use the documents since you may not have all the
executables required. We could make these available if needed. Extract
the attached Freesurfer.zip in PIPELINE_HOME/templates/misc/catalog
and launch PIPELINE_HOME/setup.sh <ADMIN_EMAIL> <MAIL_SMTP_SERVER>.
I would advise you take a look at
PIPELINE_HOME/catalog/images/WebBasedQCImageCreator.xml and add that
to a project and launch the pipeline. Its easier to deal with this
pipeline, as it doesnt have an additional schema, additional
executables etc.
f) Finally:
There are lots of details that I have intentionally skipped to keep
things simple. Feel free to email if you have any doubts.
Regards
Mohana
> --
> You received this message because you are subscribed to the Google Groups
> "xnat_discussion" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/xnat_discussion/-/i5BAUx89y7kJ.
Sanket Gupte
And thanks for the heads up. I am not sure about 'relaxing' but I did enjoy the flight.
I am still having troubles with Autorun.
a) Auto Run Issues
i) Yes Pipeline Engine Installed Correctly
XnatPipelineLauncher exists
ii) PIPELINE_HOME location is correct
iii) Application log has the error of type ProcessLauncher- Couldnt launch...
All the log files (the entire application) and the tomcat is all owned by the same user.
After Copying the string in that error and putting it up in the command prompt. It gives the attached (pipeline_Error.txt)error messages.
The words that stand out in these errors are 'No compiler found in your classpath! (you may need to add 'tools.jar')'
I already have tools.jar in my path. (If it wasn't there I wouldnt have been able to run update.sh in the first place)
I copied the tools.jar (from the bin folder of java) to tomcat_home/bin. But it didn't change anything. Still same error.
The folder PIPELINE_HOME/logs have logs per date-time. The following are the things in the last/latest log.
----------------------------------
INFO... something something.
...
.....
FATAL main org.nrg.pipeline.client.XNATPipelineLauncher - Couldnt search for queued workflows
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.lang.RuntimeException: No compiler found in your classpath! (you may need to add 'tools.jar')
faultActor:
faultNode:
faultDetail:
{http://xml.apache.org/axis/}hostname:127.0.0.1
java.lang.RuntimeException: No compiler found in your classpath! (you may need to add 'tools.jar')
at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
...and so on...
-------------------------------------
Any more suggestions? Where do I copy the tools.jar for it to be able to find it...
---
I havnt got a chance to work on the freesurfer pipeline that you attached, but I will look into it soon.
As soon as the autorun pipeline starts working I will first start by trying out the WebBasedQCImageCreator.xml pipeline. and then move on to freesurfer.
Thanks a lot for all your help and thanks again for this detailed explanation.
Mohana Ramaratnam
Try this solution:
http://www.digizol.org/2008/03/toolsjar-runtimeexception-no-compiler.html
Regards
Mohana
> --
> You received this message because you are subscribed to the Google Groups
> "xnat_discussion" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/xnat_discussion/-/iyI3zjB1DGQJ.
Sanket Gupte
Thanks for the reply.
Sorry for the delay in getting back.
That solution doesnt work.
I actually copied tools.jar in %CATALINA_HOME%/common/lib (there was no 'common' dir, I had to make it) and in %CATALINA_HOME%/lib
(and of course restarted tomcat)
But it doesnt work. The AutoRun pipeline remains queued forever. The 'transfer' gets completed successfully.
Any suggestions for where to look for the problem.
DIGRESSION:
I don't know if someone has done this but I would suggest if it is possible, then in the near future, someone should do this. Not necessarily one of you guys, but any general person who is using xnat.
Make an 'image' for Virtual Machine, that has all the xnat components and related applications installed and running properly. The most basic ones only.. So, that anyone wanting to use it can just mount that image and start running it, and hence avoid any problems. I know, everyone have their own choice and preference of the operating system, but it could be of help to someone for sure. I am myself planning to do it as soon as all my components start working properly. I run it on OpenSUSE, could be of help to someone for sure..
Sanket Gupte
The WebBasedQCImageCreator also works now. Will try out Freesurfer. Thanks for all the help.
crn...@umich.edu
Mohana Ramaratnam
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/groups/opt_out.
