xnat pipelines

[Daniel C]

Hi all,

Installing pipelines in XNAT works but whenever I attempt to run them I get a nonspecific error (screenshot attached). I've been trying the SimpleEchoPipeline included in the XNAT installation (with small edits made for local pathnames) and am attaching that xml file too.

What am I doing wrong or missing? Appreciate any help or input.

[John Flavin]

Daniel,

I can't tell exactly what is happening just from this. I'll need a little more information.

How did you set up your pipeline engine? Did you run the pipeline build script (either setup.sh or gradlew)? I ask because the paths that you've put into the pipeline have "templates" as part of the path. After the pipeline engine has been built by running the setup script, everything under "templates" gets copied out to some destination directory. (The reason for this is that the pipelines can be written with "template strings" that get substituted with specific values when you run setup. For instance, the template string @PIPELINE_DIR_PATH@ will get resolved to wherever you build your pipeline engine.)

Can you tell me step-by-step how you set up the pipeline in XNAT?

* Where did you get the pipeline engine? (If you’re using the vagrant VM, it almost certainly came with that. But you may have gotten it somewhere else.)

* How did you set it up? (What was the destination, what values did you use for the parameters, etc.)

* How did you add the pipeline to the site? (What was the path you gave, and was it correct?)

* How did you try to launch it?

That's all I have for now. We will have to do a little digging before we figure out if you've found a bug or if you've made a mistake somewhere along the (long, confusing) path to configuring the pipelines.

Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.
<xnatpipescreen.tiff><SimpleEchoPipeline.xml>

[Adam Wechter]

Hello Flavin,

We have run the setup.sh to build the pipeline.  To install I followed the instructions on the website:  Here .  We added the executable XnatPipelineLauncher to the path global variable.  We are running xnat 1.7.

[John Flavin]

Can you be more specific about the paths you used? 

* When you ran the pipeline setup script, what was the destination path you used?

* When you added the pipeline to the site, what was the path to the pipeline XML that you entered into the form? 

* Did you enter anything into the other field, the "custom webpage" field?

[Adam Wechter]

Destination path given in the gradlew.properties was /sulcusdata/xnat/pipeline  (this made a mess of our pipeline folder I think it should have been a level down and I'm planning on changing and rebuilding).  Module paths were given as /tmp/modules.  This directory remains empty even after building.  To my knowledge the pipeline was pointed at the xml in /sulcusdata/xnat/pipeline/sample_pipelines/SimpleEchoPipeline.xml.  I am not sure there was custom webpage field that was filled in.  I only see sitename and xnatURL

[Daniel C]

To answer your last two questions: the path to the pipeline xml is currently /sulcusdata/xnat/pipeline/xnat-pipeline/templates/resources/sample_pipelines/SimpleEchoPipeline.xml .
Nothing was entered for the custom webpage field.

[John Flavin]

That's the problem. (Or at least it is a problem.) You’re using the "template" pipeline and not the deployed pipeline. Once you build the pipeline engine with the setup script, it will copy everything out of "templates/resources" etc. and put it all into the location you specified as the destination path. You’re still using the template version. You should be using the deployed version (which I think would be at /sulcusdata/xnat/pipeline/sample_pipelines/SimpleEchoPipeline.xml). 

And I think that sample pipeline you were working from was... let's say it was not written following best practices. I've attached a better sample that you can try.

1. Drop it in as a replacement of the existing one at /sulcusdata/xnat/pipeline/xnat-pipeline/templates/resources/sample_pipelines/SimpleEchoPipeline.xml

2. Run the pipeline setup script.

3. Remove the pipeline that you installed into your XNAT.

4. Re-add the pipeline, this time using the deployed path (which, again, I *think* will be /sulcusdata/xnat/pipeline/sample_pipelines/SimpleEchoPipeline.xml; you should check on your system and make sure.)

Let me know if that works.

Flavin

[Daniel C]

Hi Flavin,

I just removed everything in our pipeline folder, reinstalled, put in your updated SimpleEchoPipeline to the templates/resources folder and ran gradlew with gradle.properties as listed:

xnatUrl=http://10.48.86.212:8080
sitename=XNAT
adminemail=<adam's email>
smtpServer=<server>
destination=/sulcusdata/xnat/pipeline
modulePaths=/tmp/modules

gradlew ran successfully and everything seems to have made it to the destination folder. After adding /sulcusdata/xnat/pipeline/sample_pipelines/SimpleEchoPipeline.xml to XNAT and running it, I'm getting the same error as before. What else are we missing?

[John Flavin]

I am not sure what is going wrong. I tried going through all the steps that you did, but for me everything worked fine. I was able to launch the pipeline. (There was one thing I had to fix; see below.)

Here is what I did.

1. Clone a fresh pipeline engine repository from the source (https://github.com/NrgXnat/xnat-pipeline-engine) then cd into that directory.

2. Add the properties to gradle.properties. The file you posted looks fine to me; my values are different.

3. Run the setup script.

4. In my XNAT, I add the pipeline to my site. The path you used looks correct; it should be ${pipeline_home}/sample_pipelines/SimpleEchoPipeline.xml.

5. Add the pipeline to my project. The pipeline doesn't have any parameters, so nothing to customize here.

6. Add the "Build" action to the Actions box on MR sessions. Make sure that you used the correct name: PipelineScreen_launch_pipeline.

7. Navigate to an MR session report page in XNAT and click the Build action. For me, that brings up the page to select which pipeline to launch. 

8. Select "SimpleEchoPipeline" and hit "Submit".

9. Again, there aren't any parameters, so I hit "Submit" again.

That launches the pipeline successfully.

I did have to modify the pipeline a little. The one I sent you yesterday had a small bug. I've attached a fixed version. But that bug wouldn't have affected you, because you can't even bring up the page to launch the pipeline.

Can you check if any errors are logged when you load the pipeline launch page? Perhaps something in xdat.log, velocity.log, or application.log?

Flavin

[Daniel C]

Sure, I'm attaching velocity.log and xdat.log, not sure how to interpret these but if they mean anything to you, I'd definitely appreciate you taking a look.

I did have the build action added; we have been successfully selecting the pipeline. The error we get happens after hitting submit and submit.

[John Flavin]

Those logs look unrelated to me. It's possible that something is being logged in another file. Try launching the pipeline again, and see if anything is written to any of the log files when you do that.

If you’re saying the whole process goes fine until you actually submit, that makes me think that XNAT is having a problem invoking the schedule script and the pipeline engine. 

* Did you add ${pipeline_home}/bin to the PATH in your ~/.bash_profile? 

* Does the xnat user on your system have proper read/write/execute permissions on...

** the ${pipeline_home}/bin directory, 

** the executables inside that directory, and 

** the build directory? (The build directory is a directory where XNAT will create a new unique subdirectory for each pipeline execution to write its files. For you, I think the location will be /sulcusdata/xnat/build, but I am not sure.)

Flavin

-- 
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

<xdat.log><velocity.log>

[Daniel C]

After attempting to run the pipeline again and sorting the logs folder by time modified, it looks like turbine.log has some new stuff written. The log file is attached. Also pasting the line that seems to be relevant to the pipeline problem?

2017-04-21 10:27:58,945 [http-bio-8080-exec-3] ERROR org.nrg.xnat.turbine.modules.actions.ManagePipeline - Error occurred trying to redirect to the pipeline launch page.

For the other questions, yes, we modified the ~/.bash_profile and chmod 777'd the /bin and /build directories.

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsub...@googlegroups.com.

[John Flavin]

That error does look relevant, yes. I looked through the log file you sent, and I also think the other errors are relevant.  Take a look at this one:

2017-04-21 09:04:43,975 [http-bio-8080-exec-5] ERROR org.apache.turbine.Turbine - Turbine.handleException: 
java.io.FileNotFoundException: /data/xnat/build/COBRA/20170421_090443/201_MR1/201_MR1_params_20170421.xml (No such file or directory)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:566)
at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2526)
at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:220)
at org.nrg.xnat.turbine.modules.actions.ManagePipeline.saveParameters(ManagePipeline.java:461)
at org.nrg.xnat.turbine.modules.actions.ManagePipeline.doLaunchpipeline(ManagePipeline.java:409)
<snip>

There are several like that in your log. What this error is saying is that one part of the pipeline launching process is trying to read a params file from a build directory, but the file doesn't exist. That file should be written by an earlier part of the same launching process, but I suspect that write operation failed silently, and we only see the error when a later part of the process tries to read the file.

My first hypothesis was that your XNAT does not have the proper permissions on the build directory. I was almost able to reproduce your error by changing ownership of my /data/xnat/build directory to root and giving it 700 permissions. When I tried to launch the SimpleEchoPipeline, I got this in my turbine.log:

2017-04-25 12:18:57,758 [http-bio-8080-exec-20] ERROR org.apache.turbine.Turbine - Turbine.handleException:
java.io.FileNotFoundException: /data/xnat/build/foo/20170425_121857/5Yp0E/5Yp0E_params_20170425.xml (Permission denied)
at java.io.FileOutputStream.open0(Native Method)
at java.io.FileOutputStream.open(FileOutputStream.java:270)
at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at java.io.FileOutputStream.<init>(FileOutputStream.java:162)
at org.apache.xmlbeans.impl.store.Cursor._save(Cursor.java:566)
at org.apache.xmlbeans.impl.store.Cursor.save(Cursor.java:2526)
at org.apache.xmlbeans.impl.values.XmlObjectBase.save(XmlObjectBase.java:220)
at org.nrg.xnat.turbine.modules.actions.ManagePipeline.saveParameters(ManagePipeline.java:461)
at org.nrg.xnat.turbine.modules.actions.ManagePipeline.doLaunchpipeline(ManagePipeline.java:409)
<snip>

That looks to me like something very very close to the error you’re seeing. But there is one key difference: my FileNotFoundException gives the reason "Permission denied", and yours gives "No such file or directory". The errors are really similar, though, so I do think we are on to something, but I don't think my hypothesis is correct. 

You said you gave the "/build" directory 777 permissions, but can you double check? I can see from the error message in your log that XNAT is trying to create the pipeline execution's build directory under /data/xnat/build. Is that the directory that you changed permissions of? Make sure that XNAT has read/write permissions on that directory, and all subdirectories. (Even though I don't think this is a permissions problem, I just want to make sure.)

Also, check that the build path is where you expect it to be. Go to Administer > Site Administration, and click on the File System tab. You can verify and customize the paths that XNAT uses for various things, including the build path.

Lastly, check the subdirectories under the build path. The error I pasted above is for a file /data/xnat/build/COBRA/20170421_090443/201_MR1/201_MR1_params_20170421.xml. I assume that file doesn't exist, because that's what the error says, but do any of the intermediate subdirectories between /data/xnat/build and the (nonexistent) file exist? Do they have proper permissions?

Flavin

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

<turbine.log>

[Daniel C]

Yes we are absolutely on to something. The build directory has full 777 permissions but you're right, none of those intermediate subdirectories exist (starting with COBRA). So the question is at which stage of the process are those directories created, and why don't they exist for us? The build path in Site Administration points to the correct place. Should I try creating the project again? It's possible that the build path may not have been correct when I created the COBRA project or the MR session and that's why the folder doesn't exist.

[John Flavin]

I think you could solve your problem by creating that project directory. All the other subdirectories under that one are created anew for each pipeline launch.

I don't think I can answer the question of why the project directory wasn't created when it should have been. 

Flavin

[Daniel C]

I tried creating the directory /sulcusdata/xnat/build/COBRA as suggested, set it to 777 permissions, and still got the same error when trying to run the pipeline. The turbine error log gives the same file not found error message. Any idea at what stage that project directory is created?

Flavin

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.
<xdat.log><velocity.log>

-- 
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at https://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.
<turbine.log>

[John Flavin]

Remember that your build directory is not /sulcusdata/xnat/build, it is /data/xnat/build. So try either, A. Creating /data/xnat/build/COBRA, or B. Changing your site settings so the build path is /sulcusdata/xnat/build (and then you've already created the correct project directory). 

I'm guessing that XNAT tried making the /data/xnat/build/COBRA directory when you created the project, but that failed. If you can remember what day that was and look back in the logs, I suspect you'll find some relevant error message about it.

Flavin

[Daniel C]

Those settings are in the Site Administration --> File System, right? Yes, we've had those changed to /sulcusdata/xnat/build for quite some time now.

I tried creating a new project and running the pipeline but there is no /sulcusdata/xnat/build/<newproject> directory created either.

-Daniel

[John Flavin]

Let me go over everything we know so I can make sure I have everything correct. Can you confirm that I have these things for me?

1. Your Build path in Site Administration > File System is set to "/sulcusdata/xnat/build". 

2. When you launch a pipeline, XNAT is attempting to create a build directory and run the pipeline from a subdirectory of "/data/xnat/build/COBRA". 

Here is my guess of what happened: When you created this project, the site-wide build directory was set to /data/xnat/build. The project stored its project-specific build directory as /data/xnat/build/${project}. Then you changed the site-wide build directory to /sulcusdata/xnat/build, but the project-specific build directory was not changed. Now when you try to launch pipelines, XNAT is attempting to create a subdirectory under the project-specific build directory, which doesn't exist.

One thing that doesn't fit with my hypothesis: you said that you created a new project, and no build directory was created at /sulcusdata/xnat/build/${new_project}. I'm not sure if that disproves the hypothesis, though. We can check.

Can you confirm these things for me?

1. Check your site-wide build directory. I 100% expect that you are correct and it is set to /sulcusdata/xnat/build, but it is still good to check.

2. Check the COBRA project's build directory. (I'll tell you how in just a moment.) I suspect the value will be /data/xnat/build/COBRA.

3. Check the new project's build directory. I am less certain of this one, but I hope the value is /sulcusdata/xnat/build/${new_project}.

To check a project's build directory, you have to load a URL that will return some XML. Sorry, there isn't a pretty user interface for this. You can view a project's archive spec at ${your_xnat}/data/projects/${project}/archive_spec. Near the top, there should be a section <arc:paths>...</arc:paths>, and inside that is <arc:buildPath>.

Flavin

[Daniel C]

Just ran the pipeline in our <newproject> and I'm not getting that error anymore. Where should I look to find the actual echoed statement?

But to confirm your points:
1. yes, I checked the Build path in Site Administration > File System, it is /sulcusdata/xnat/build
2. yes, you are correct about COBRA project's build directory being /data/xnat/build/COBRA
3. yes, the new project's build directory is /sulcusdata/xnat/build/$new_project

So I guess the simplest solution (for COBRA) is to just create a new project?

[John Flavin]

The way the pipeline is written, it echoes the statement into a file in the build directory that gets created for that pipeline execution. To find the build directory, you can look in ~/logs/launch.log for a list of all the pipeline launch strings. You may have to figure out which is the one you want by a little trial-and-error. Once you find the right launch string, the build directory is one of the arguments that is passed in that string.

To solve your sitewide and project build space mismatch, I can think of two solutions. (Well, technically, I thought of a third solution but it didn't work. So let's say I can think of two solutions that will work correctly.)

1. Make a new project. This is the easiest solution, if you don't have a lot of data and settings and stuff already built up in your existing project.

2. Edit the value in the database. I am not expert in sql, so I can't write this for you. I can tell you that the paths are in the table arc_pathinfo, and you can tell which project is which by joining to the project's entry in the arc_project table. This select query will get you the correct row:

select * from arc_pathinfo path inner join (select paths_arc_pathinfo_id, id from arc_project) proj on path.arc_pathinfo_id = proj.paths_arc_pathinfo_id where proj.id = 'COBRA';

Flavin