1.6.5 Pipeline build

[Simon Doran]

Hi All,

  I built and started running 1.6.5 with very few problems some weeks back. For various reasons our machine has, until the last couple of days, not had the connectivity to start uploading new data. Now it has and I noticed immediately that the Autorun pipeline was not working properly.

  Closer inspection revealed that, although no errors appeared to have been flagged - or maybe I just missed them! - during installation, gradle was not present on the machine at the time I originally ran /bin/setup.sh.

  So, I installed gradle, edited $XNAT_HOME/pipeline/gradle.properties and typed sh gradlew. Everything seemed to go to plan

Unzipping /home/xnat/.gradle/wrapper/dists/gradle-2.2.1-bin/88n1whbyjvxg3s40jzz5ur27/gradle-2.2.1-bin.zip to /home/xnat/.gradle/wrapper/dists/gradle-2.2.1-bin/88n1whbyjvxg3s40jzz5ur27

Set executable permissions for: /home/xnat/.gradle/wrapper/dists/gradle-2.2.1-bin/88n1whbyjvxg3s40jzz5ur27/gradle-2.2.1/bin/gradle

Building pipeline engine

 * From:        /data/XNAT_anonymised/code/xnat_1.6.5/pipeline

 * To:          /home/xnat/XNAT/Home/code/Current/pipeline

 * Email:       xxx

 * Site name:   XNAT_anonymised

 * Site URL:    https://xxx:8443/XNAT_anonymised

 * SMTP server: 192.168.213.96

Resource folders:

 * /data/XNAT_anonymised/code/xnat_1.6.5/pipeline/templates/resources

Script folders:

 * /data/XNAT_anonymised/code/xnat_1.6.5/pipeline/templates/scripts

Binary folders:

:copyBinaries UP-TO-DATE

:copyLibs

:copyResources

:copyScripts

:copyAll

BUILD SUCCESSFUL

Total time: 7.353 secs

but then when I tried to run the failed Autorun command via 

/home/xnat/XNAT/Home/code/Current/pipeline/bin/XnatPipelineLauncher -pipeline xnat_tools/AutoRun.xml  ...

I got

Error: Could not find or load main class org.nrg.pipeline.client.XNATPipelineLauncher

As far as I know I typed all the entries in the gradle.properties file correctly and pipeline/lib appears to have lots of jars in it.

Any suggestions?

Best wishes,

Simon

[Herrick, Rick]

You actually shouldn’t need gradle installed at all. The XNAT setup script invokes setup.sh (or bat) in the pipeline installation folder, which in turn calls the gradlew[.bat] script in the same folder. That checks to see whether you have gradle installed, uses that if so, and downloads a local copy of gradle if not (https://docs.gradle.org/current/userguide/gradle_wrapper.html).

My guess is that your build.properties file is missing a couple properties. The new pipeline build scripts actually work in a manner similar to the XNAT builder: there’s an installer folder and a destination folder (and an optional pipeline modules folder). If the XNAT setup can’t find the installer or deployment folders, it will try default locations (pipeline-installer and pipeline in the builder folder for the installer and deployment folders respectively).

The class that you’re getting the message about is in one of the basic pipeline libraries. Check the lib folder and verify that pipelineClient-1.6.5.jar is there. If it’s not, then something went wrong with the installation. I doubt this is the case, since you said there are plenty of libraries there.

Also, make sure the pipeline location specified in your XNAT configuration is where the application was deployed TO, not FROM.

Now make sure the paths in the bin/XnatPipelineLauncher script are correct. Edit that file and make sure that the paths used in the -classpath parameter point to libraries in the correct location.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 362-1882

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Simon Doran]

Hi Rick,

  Thanks for getting back to me. I've just been following through some of your suggestions:

You actually shouldn’t need gradle installed at all. The XNAT setup script invokes setup.sh (or bat) in the pipeline installation folder, which in turn calls the gradlew[.bat] script in the same folder. That checks to see whether you have gradle installed, uses that if so, and downloads a local copy of gradle if not (https://docs.gradle.org/current/userguide/gradle_wrapper.html).

 

Hmmm ... well my empirical observation was that the most obvious reason that running the pipeline failed was that there was no bin/ directory. So I'm assuming that the installer just didn't run successfully.
 

My guess is that your build.properties file is missing a couple properties.

Well, I saw these lines and left them commented out because I have left all the pipeline stuff sitting in the default $XNAT_HOME/pipeline

# If you have pipeline installed outside of your xnat_builder folder, un-comment this property to set it
# the appropriate location. As with all folder locations, these paths should be absolute, not relative to
# your xnat_builder location.
# xdat.pipeline.installer.location=/Library/XNAT/pipeline-installer
# xdat.pipeline.deployment.location=/Library/XNAT/pipeline
# xdat.pipeline.modules.locations=/Library/XNAT/pipeline_modules

However, I noticed that when I cloned the source code from the repository, there was no pipeline-installer directory or pipeline-modules directory. Am I supposed to create these manually?

The new pipeline build scripts actually work in a manner similar to the XNAT builder: there’s an installer folder and a destination folder (and an optional pipeline modules folder). If the XNAT setup can’t find the installer or deployment folders, it will try default locations (pipeline-installer and pipeline in the builder folder for the installer and deployment folders respectively).

The class that you’re getting the message about is in one of the basic pipeline libraries. Check the lib folder and verify that pipelineClient-1.6.5.jar is there.

It is in $XNAT_HOME/pipeline/lib
 

If it’s not, then something went wrong with the installation. I doubt this is the case, since you said there are plenty of libraries there.

Also, make sure the pipeline location specified in your XNAT configuration is where the application was deployed TO, not FROM.

My XNAT configuration in the webapp has this:

Pipeline Installation Location: /home/xnat/XNAT/Home/code/Current/pipeline
 (where $XNAT_HOME above = /home/xnat/XNAT/Home/code/Current

Now make sure the paths in the bin/XnatPipelineLauncher script are correct. Edit that file and make sure that the paths used in the -classpath parameter point to libraries in the correct location.

The script starts off like this:
java $XNAT_PIPELINE_LAUNCHER_OPTS -classpath "/home/xnat/XNAT/Home/code/Current/pipeline"/lib/mail-1.4.5.jar ...
 
and there is a list of loads of jars, including:

:"/home/xnat/XNAT/Home/code/Current/pipeline"/lib/pipelineClient-1.6.5.jar

So far, so good. But then, guess what?

ls "/home/xnat/XNAT/Home/code/Current/pipeline"/lib/
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 activation-1.1.1.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 ant-1.6.1.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 aopalliance-1.0.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 axis-1.4.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 axis-jaxrpc-1.4.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 axis-saaj-1.4.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 axis-wsdl4j-1.5.1.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 commons-beanutils-1.8.3.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 commons-codec-1.5.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 commons-collections-3.2.1.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 commons-configuration-1.0.jar
-rw-rw-r-- 1 xnat xnat 0 Dec  7 16:51 commons-dbcp-1.2.1.jar

That's bizarre. All the file sizes are zero.
Bottom line: I recloned the repository, copied the lib directory and hey presto, the pipeline runs!

What on earth is going on?

Best wishes,

Simon


 

[Simon Doran]

Hi Rick,

  Sorry, that was a false dawn. It looked as if it was working, but then the pipeline failed with this error:

ERROR main org.nrg.pipeline.client.XNATPipelineLauncher - Unable to launch pipeline /home/xnat/XNAT/Home/code/Current/pipeline/catalog/xnat_tools/AutoRun.xml

java.lang.NoClassDefFoundError: org/nrg/xdat/bean/XnatAbstractresourceBean

        at java.lang.Class.getDeclaredMethods0(Native Method)

        at java.lang.Class.privateGetDeclaredMethods(Class.java:2625)

        at java.lang.Class.privateGetPublicMethods(Class.java:2743)

        at java.lang.Class.getMethods(Class.java:1480)

  It definitely looks as if the installation just didn't happen as it should have. I don't know why.

  What is the best way to completely rebuild the pipeline module from scratch, including any compilation necessary? Let's say I delete the $XNAT_HOME/pipeline directory and re-clone it. What then?

  Can I do this without affecting the rest of the running instance of XNAT? Do there need to be changes to the webapp itself?

  Thanks for your help[.

  Best wishes,

Simon

[michael....@gmail.com]

Installation of XNAT 1.6.5 fails

Hi

Sorry for posting this here but I can not create a new topic (or at least it is not shown when I refresh the page).

I tried to install XNAT 1.6.5 on a Debian 8 according to https://wiki.xnat.org/display/XNAT16/XNAT+1.6+Installation+Guide

Everything is fine until I try to login with the default account (admin/admin):

XNAT has encountered an error with your request:

Status: Unknown status

URI: Unknown URI

Message: Unknown error occurred

If this error continues to occur, please contact your system administrator with information about how to recreate the problem.

The strange thing is the URL: http://x.x.x.x:8080/xnat/app/template/xnat/app/template/Configuration.vm
XNAT seems to insert parts of the URL twice ('/xnat/app/template')

Please advice

Thank you

Michael

[Herrick, Rick]

The problem now is that the pipeline engine wasn’t there during setup. One of the steps during setup is to copy the file xdat-beans-xxx.jar into the pipeline lib folder. That jar is created during the XNAT build process from the various bean and model classes generated from the data-type schema. It also gets copied into the folder plugin-resources/repositories/xdat/jars and eventually into the WEB-INF/lib folder of the deployed web application. So copy it from one of those locations into the lib folder of your pipeline installation and you should be good to go.

If you still want to rebuild the pipeline engine from scratch, the best thing to do is to clone the pipeline repo (https://bitbucket.org/nrg/pipeline_1_6dev). Go into that folder and copy sample.gradle.properties to the file gradle.properties. Modify gradle.properties as appropriate. Here’s what I have for one of my dev repos:

xnatUrl=http://xnatdev.xnat.org

siteName=XNAT

adminEmail=ad...@xnat.org

smtpServer=mail

destination=/data/xnat/pipeline

modulePaths=/data/xnat/modules/pipeline

This means that the pipeline will use http://xnatdev.xnat.org to call back into XNAT, send notification emails to ad...@xnat.org, and—most importantly—install the pipeline to the folder /data/xnat/pipeline. modulePaths is optional (we don’t have any documentation on pipeline modules at this point, but will try to get something up on the XKB knowledge base).

Now in that folder just run:

./gradlew

Upon completion, your pipeline engine will be installed into the folder specified by the destination property. Copy xdat-beans-xxx.jar from your XNAT installation or deployed web app into the lib folder of the destination.

To answer your questions about XNAT, you don’t need to stop XNAT and this doesn’t affect XNAT, outside of obviously causing issues if the pipeline engine isn’t at the location specified in the file system configuration in XNAT when a pipeline like AutoRun gets launched. So you probably don’t want to be archiving sessions while you’re doing this, but the good thing is that the pipeline “build" is quite fast, since it’s really just a series of copy operations with some string replacement.

Hope that helps…

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

From: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com> on behalf of Simon Doran <simon...@icr.ac.uk>
Reply-To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>
Date: Tuesday, December 8, 2015 at 3:09 AM
To: "xnat_di...@googlegroups.com" <xnat_di...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[Simon Doran]

Hi Rick,

  Thanks. That fixed it.

Simon