Start a Pipeline from REST API?

[Ben Wagner]

Is it possible to kick off a pipeline trough REST?

Thanks,
Ben

[Mohana Ramaratnam]

Ben,

Pipelines are associated to a project and identified by a step id (see the schema <XNAT_HOME>/plugin-resources/project-skeletons/xnat/src/schemas/project/project.xsd).

To invoke a pipeline, POST to:

/REST/projects/PROJECT_ID/experiments/EXPERIMENT_ID/pipelines/STEP_ID

If you set the STEP_ID to triggerPipelines then <PIPELINE_HOME>/catalog/xnat_tools/AutoRun.xml is executed.

A single pipeline can be set with stepid AUTO_ARCHIVE if you check the box while setting up the pipeline for a project.

If you want to invoke the pipeline with stepid AUTO_ARCHIVE this is the REST call:

/REST/projects/PROJECT_ID/experiments/EXPERIMENT_ID/pipelines/AUTO_ARCHIVE

The catch is that all the parameters that the pipeline needs, must be derivable from the xmlpath of the Experiment (values set using schemalink) or must be constants (csv). See <PIPELINE_HOME>/sample_pipelines/SampleAutoRunPipeline,xml

So the answer is yes but not all pipelines.

Regards
Mohana

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To post to this group, send email to xnat_di...@googlegroups.com.
To unsubscribe from this group, send email to xnat_discussi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xnat_discussion?hl=en.

[Jan]

Hey Mohana 

I run xnat-1.6.3 from the example VM xnat offers and have the same question like Ben. 

Now I tried as you descriped to call the following statement:

curl -u admin:admin -X POST http://xnatdev/REST/projects/test/experiments/XNAT_E00002/pipelines/mitk

mitk is the name of my pipeline, wasn't sure if its meant by STEP_ID

Also I didn't know which parameters I have to pass for the scanids or any other parameters the pipeline needs. (e.g. --data "scandis:[4,5,6]")

Could you tell me perhaps which REST Calls I need, to get the information of available pipelines for a project and there input parameters to trigger them.

I would be delighted to hearing from you.

Regards

Jan

[Mohana Ramaratnam]

Hi Jan,

As of now, you can not pass parameters while you launch pipelines via REST call. Its a feature addition that is needed! 

Only those pipelines whose input arguments can be figured out  from the session XML using XPATH and/or constants that are set while adding the pipeline to you project, can be launched via REST call. 

Regards

Mohana 

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.

Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[James Dickson]

Hi Mohana,

Right now this would be my number 1 feature request for xnat. In the past I could hijack the ManagePipeline page and post parameters to kick off a pipeline, but currently this doesn't work because the xnat_csrf addition. Any idea when this feature could be available?

Thanks

James

[Flavin, John]

Jan,

You can get the information about which pipelines are available on a project at

/REST/projects/PROJECT_ID/pipelines

Given a pipeline name, you can see its project-level default parameters at

/REST/projects/PROJECT_ID/pipelines/PIPELINE_NAME

That will tell you the input parameters the pipeline expects to see, their default values, and whatever description text has been included in the pipeline. It will also tell you the authors, version, and what steps the pipeline will execute.

…but if memory serves the latter REST query was not implemented until after XNAT 1.6.3 shipped. You would need XNAT 1.6.4 for that one to work. The former, querying the project about its pipelines, should work in either version. 

Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.

Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[Flavin, John]

This feature is actually available in the current version of XNAT. You can POST parameters essentially just as you describe and they will be passed to the pipeline.

There are some details of the syntax of what exactly gets accepted that I am not sure about. I know that you can do “name=value”; I don’t know if you can do “name:value”.  But I am certain you can do “name=[val1,val2,…]” and they’ll get turned into a list (because I did that part myself).

Flavin

(For details of the code that parses the request, if you’re into that sort of thing, see this commit: https://bitbucket.org/nrg/xnat_builder_1_6dev/commits/d5b5b38e4e31fd5571286119fc3430154958422e)

[James Dickson]

Hi John,

Thanks for that. I've been looking for this functionality for a while. This code is only a few weeks old! Its currently part of 1.6.4 right?

James

[Flavin, John]

Correct. This is in XNAT 1.6.4.

If you start using it and something either doesn’t work or doesn’t work like you expect, let me know and I’ll try to get it fixed. Post to the group, email me, or I'm @cndapipelines on twitter.

Flavin

[Herrick, Rick]

It shouldn’t be too difficult to implement, especially if we just looked for the pipeline parameter as form-encoded key-value pairs.  I’ll have a look and see if I can slip it into 1.6.4 (end of week release date!).

 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

314-740-5961

[Herrick, Rick]

It’s so not difficult that it’s already done. I totally forgot Flavin had done that! Man, I need to get something to keep track of all these things, like an issue tracker or wiki or something like that…

 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

314-740-5961

 

[Jan]

Hey guys,

thanks a lot, especially Flavin, for your fast answers and the solution for my problem. 

I'll try it out and give feedback if something doesn't work.

Regards 

Jan

 

[dagmar....@googlemail.com]

Dear all,
I'm using 1.6.4., but for some reason the above discussed features seem not to work.

The first REST-Call works perfectly:

curl -X GET -u 8741f7d3-873b-4870-a9cd-e56f0a0240e0:1416869787530 http://192.168.50.5:8080/xnat/data/archive/projects/somnonetz_test/pipelines/
{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Demo pipeline created NIFTI files from the DICOM files and addes these files at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/mriconvert/DicomToNifti.xml"},{"Name":"edfMetadataExtractor","Description":"Extracts EDF Metadata and inserts into the xnat datatype (psgScanData)","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/somnonetz_tools/edfMetadataExtractor.xml"}]}}

but trying to get the specification of a given pipeline using the above syntax gives the same result:

curl -X GET -u 8741f7d3-873b-4870-a9cd-e56f0a0240e0:1416869787530 http://192.168.50.5:8080/xnat/data/archive/projects/somnonetz_test/pipelines/DicomToNifti
{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Demo pipeline created NIFTI files from the DICOM files and addes these files at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/mriconvert/DicomToNifti.xml"},{"Name":"edfMetadataExtractor","Description":"Extracts EDF Metadata and inserts into the xnat datatype (psgScanData)","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/somnonetz_tools/edfMetadataExtractor.xml"}]}}

Any idea what went wrong?

Thanks in advance and best wishes

Dagmar

[Flavin, John]

That is strange. Can you try using the same REST path but with “/details” added to the end? I.e.

/data/archive/projects/somnonetz_test/pipelines/DicomToNifti/details

Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

[dagmar....@googlemail.com]

Dear Flavin,

again, the same result:

curl -X GET -u admin:admin http://192.168.50.5:8080/xnat/data/archive/projects/somnonetz_test/pipelines/DicomToNifti/details

{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Demo pipeline created NIFTI files from the DICOM files and addes these files at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/mriconvert/DicomToNifti.xml"},{"Name":"edfMetadataExtractor","Description":"Extracts EDF Metadata and inserts into the xnat datatype (psgScanData)","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/opt/xnat/pipeline/catalog/somnonetz_tools/edfMetadataExtractor.xml"}]}}

The only log that is updated ist the the access.log,

2014-11-26 17:38:35,813 - admin 192.168.50.1 GET http://192.168.50.5:8080/xnat/data/archive/projects/somnonetz_test/pipelines/DicomToNifti/details

oh, probably it's an upgraded version. The xnat-gui says 1.6.4, VERSION says 1.6.4 but the release notes say The latest release for XNAT is version 1.6.3.
I'll ask my collegue who set up the instance.
Might that be the problem?

[Flavin, John]

Could be. The code to return the pipeline details was only added in 1.6.4. I find it plausible that if that code isn’t in place, the REST interpreter would find the closest match it knows about and just dump everything after "pipelines/“.

If you have access to the build directory, can you check if the following file exists?

{xnat_builder}/plugin-resources/webapp/xnat/java/org/nrg/xnat/restlet/extensions/PipelineDetailsRestlet.java

Flavin

[dagmar....@googlemail.com]

yes, it exists, but says:

 * org.nrg.xnat.restlet.extensions.PipelineDetailsRestlet
 * XNAT http://www.xnat.org
 * Copyright (c) 2014, Washington University School of Medicine
 * All Rights Reserved
 *
 * Released under the Simplified BSD.
 *
 * Last modified 20140513_1147
 */

should it be newer?

My colleague said he used the new 1.6.4-release to build the instance.

[Herrick, Rick]

The updated release notes didn't make it into the release. If you have PipelineDetailsRestlet, you should be good.

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

[dagmar....@googlemail.com]

I reproduced the result with the provided xnat 1.6.4-server-vm:

(added the pipeline under Administer > Pipelines with: /home/xnat/pipeline_1_6dev/catalog/mriconvert/DicomToNifti.xml
added the pipeine to the project prj001)

using the pipeline-resource:

curl -X GET -u admin:admin http://192.168.56.101/data/archive/projects/prj001/pipelines/
{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Demo pipeline created NIFTI files from the DICOM files and addes these files at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/home/xnat/pipeline_1_6dev/catalog/mriconvert/DicomToNifti.xml"}]}}dagi@castels:~$

using the details-resource:

curl -X GET -u admin:admin http://192.168.56.101/data/archive/projects/prj001/pipelines/DicomToNifti/details
{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Demo pipeline created NIFTI files from the DICOM files and addes these files at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/home/xnat/pipeline_1_6dev/catalog/mriconvert/DicomToNifti.xml"}]}}

Would it be possible that you might check if you get the same result within the vm?

Best wishes and thanks in advance

Dagmar

[Flavin, John]

I installed the 1.6.4 desktop VM and everything worked fine for me.

I logged in with admin/admin and typed the addresses into the browser.

https://xnatdev/data/projects/prj001/pipelines
{"ResultSet":{"Result":[{"Name":"DicomToNifti","Description":"Pipeline creates NIFTI files from DICOM files. The NIFTI files are added at SCAN level","Datatype":"All Datatypes","Applies To":"All Datatypes","Generates":"","Path":"/home/xnat/Development/pipeline-1.6.4/templates/misc/catalog/mricron/DicomToNifti.xml"}]}}

https://xnatdev/data/projects/prj001/pipelines/DicomToNifti
{"authors":[{"lastname":"Mohana","firstname":"Ramaratnam"}],"inputParameters":[{"values":{"schemaLink":"xnat:imageSessionData/scans/scan/ID"},"description":"The scan ids of all the scans of the session","name":"scanids"},{"values":{"schemaLink":"xnat:imageSessionData/ID"},"description":"The XNAT ID (Accession Number) of the session","name":"xnat_id"},{"values":{"schemaLink":"xnat:imageSessionData/label"},"description":"The session id  of the session","name":"sessionId"},{"values":{"schemaLink":"xnat:imageSessionData/project"},"description":"Project ID","name":"project"},{"values":{"schemaLink":"xnat:imageSessionData/subject_ID"},"description":"Subject ID","name":"subject"},{"values":{"csv":"Y"},"description":"Set this value to Y if .nii file should be created. Set it to N if .img/.hdr files should be created","name":"create_nii"}],"generates":"","description":"Pipeline creates NIFTI files from DICOM files. The NIFTI files are added at SCAN level","appliesTo":"All Datatypes","path":"/home/xnat/Development/pipeline-1.6.4/templates/misc/catalog/mricron/DicomToNifti.xml","steps":[{"id":"0","description":"Create NIFTI folder"},{"id":"1","description":"Create folder for each series in NIFTI subfolder"},{"id":"0a","description":"Create RAW folder"},{"id":"1a","description":"Copy Scan DICOM DATA into RAW folder"},{"id":"1b","description":"Unzip the scan files"},{"id":"2","description":"Convert each series file into a 4d NIFTI file"},{"id":"3","description":"Upload files to XNAT"},{"id":"4","description":"Upload files to XNAT"},{"id":"END-Notify","description":"Notify"}],"version":"1”}

(Same result with and without /details.)

Did you set the pipeline up to run automatically on archive? If so, then you’ll have to use the name “AUTO_ARCHIVE_DicomToNifti” to get the details. You can check this in the archive spec: /data/projects/prj001/archive_spec. There will be an entry for the DicomToNifti pipeline, and the “stepId” attribute will tell you the name you need to use.

Flavin

[dagmar....@googlemail.com]

Dear Flavin,

many thanks for testing, now I could track down the error in my request - I used  xnat/data/archive/projects rather than xnat/data/projects...
Now it works!

Best wishes

Dagmar

[Herrick, Rick]

That's really strange. Basically the /archive paths are exactly the same as the paths without /archive in them. The paths with /archive are preserved for backwards compatibility with existing scripts and functions, but if you map an archive URI, e.g. /data/archive/foo, that is actually done through a method that maps it to both /data/archive/foo and /data/foo. I'll have a look later and see if there's any way the archive version of the API function might work differently from the non-archive function.

[lazar.f...@mountsinai.org]

Based on this discussion, I seem to conclude that it is possible to start execution

of a pipeline via rest api and pass arguments to it 

however, as a first step I did this

curl -X GET https://xxxxx/data/projects/yyy/pipelines

and go this:

<html>

<head>

   <title>Status page</title>

</head>

<body>

<h3>The server understood the request, but is refusing to fulfill it</h3><p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.4">here</a>.<br>

Please continue your visit at our <a href="/">home page</a>.

</p>

</body>

</html>

XNAT claims to be: version 1.6.4-SNAPSHOT-7949a2921503

any suggestions?

thank you

[John Flavin]

I think you are seeing a known bug (or perhaps “questionable design decision”). If a user has “member” permissions on a project, the user can launch pipelines, but can’t see what pipelines are installed on a project. I fixed that fairly recently, but I don’t think that fix got into 1.6.5. So all currently released versions of XNAT will behave in this way.

But seeing this error message should not stop you from launching a pipeline. If you can launch a pipeline in the webapp, clicking around with your mouse, you will also be able to do so via the proper REST requests.

Flavin

[lazar.f...@mountsinai.org]

Dear John

Could you describe how would I get the name of the pipeline and its arguments if I cannot see it?

how do I get all of this from the web browser?

Thanks

Lazar

[John Flavin]

That’s a good question. I’m not entirely sure, because I don’t know what information you do have to work from. Let me go look...

You should be able to look at the archive_spec:

http://{xnat}/data/{project}/archive_spec

You can open that in your browser, or use curl. That will return an XML document that gives some general information about the project, including all the installed pipelines and all their parameters. If you want to launch one, the pipeline name you should use in the rest path on your POST is the value of the “stepId” attribute on the <arc:pipeline> element, not the value of <arc:name> or <arc:displayText>. 

Visit this group at https://groups.google.com/group/xnat_discussion.

[lazar.f...@mountsinai.org]

Dear John

http://{xnat}/data/projects/{project}/pipelines/{step_id}/experiments/{experiment_id}?paramName1=paramValue1&paramName2=paramValue2&…

Yes I can start pipeline the way you describe.

however, the parameters are not passed.. :-(

So, if pipeline parameters can be deduced from the xnatid of the session, then everything appears to work.

However, passing additional parameters (like T1w_ScanId for the structural pipeline) does nothing.

the pipeline is executed with "default" parameters which for T1w_ScanId happens to be the list of all scans in the session.

any ideas why?

maybe instead of {experiment_id}? I should write something like {experiment_id}/parameters?pName1=pValue1&pName2=pValue2

??

Thanks

Lazar

[John Flavin]

Sorry, I missed a key fact. Your version of XNAT is 1.6.4-SNAPSHOT; launching a pipeline via rest with parameters was added in 1.6.4, which was released later than 1.6.4-SNAPSHOT.

With your current version of XNAT, you can launch a pipeline via rest, but it will not accept any parameters. As you’ve seen.

[lazar.f...@mountsinai.org]

Dear John

I was worried about that "snapshot" thing...

Do you have an opinion how difficult it is to upgrade to 1.6.4 ? (or maybe 1.6.5?) and what might be problems during the upgrade?

Thanks

Lazar

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@

[lazar.f...@mountsinai.org]

Dear John

would the mechanism using storeXML work on the 1.6.4.snapshot? I saw a mention here

https://groups.google.com/forum/#!msg/xnat_discussion/ixT795KklJs/f2L2y8-ACAAJ

 curl -u $USERNAME:$PASSWORD "$host/data/projects/$projID/pipelines/$StepID/experiments/{$NotSureWhatIsHERE}?inbody=true" -X POST -d @paramsFile.xml

is there a better description how to use this facility? what should be in the $NotSureWhatIsHERE ??

I assume that pipelinename and stepID are synonyms

Also I tried this on our system

GET /data/services/workflows/{pipeline name}?display=LATEST&experiment={session id}

and it returned an error. Is this something I did incorrectly or the version of xnat we have does not support functionality?

Thank you very much for your help

Lazar

To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussion+unsubscribe@

[John Flavin]

You asked a lot of questions; I’ll try to answer them all.

would the mechanism using storeXML work on the 1.6.4.snapshot? I saw a mention here

https://groups.google.com/forum/#!msg/xnat_discussion/ixT795KklJs/f2L2y8-ACAAJ

I’m not sure exactly what Kate was doing with StoreXML, so I can’t help with that.

 curl -u $USERNAME:$PASSWORD "$host/data/projects/$projID/pipelines/$StepID/experiments/{$NotSureWhatIsHERE}?inbody=true" -X POST -d @paramsFile.xml

is there a better description how to use this facility? what should be in the $NotSureWhatIsHERE ??

1. $NotSureWhatIsHERE should be the ID of an experiment (usually an image session).

2. 1.6.4-SNAPSHOT can’t accept parameters when launching a pipeline, either in the query params or an xml file.

I assume that pipelinename and stepID are synonyms

Generally, yes, but with some exceptions. When you added the pipeline to your project, you had an opportunity to set the pipeline up to run automatically when a session is archived. If you did that, then the stepId is “AUTO_ARCHIVE_” + pipeline name. If you didn’t do that, then stepId  is the pipeline name.

Also I tried this on our system

GET /data/services/workflows/{pipeline name}?display=LATEST&experiment={session id}

and it returned an error. Is this something I did incorrectly or the version of xnat we have does not support functionality?

As far as I know, that should work on your XNAT version. What error was returned? What did you put in for the pipeline name and session id?

[lazar.f...@mountsinai.org]

Dear John

You asked a lot of questions; I’ll try to answer them all.

Thank you very much  for your kind responses

 

 curl -u $USERNAME:$PASSWORD "$host/data/projects/$projID/pipelines/$StepID/experiments/{$NotSureWhatIsHERE}?inbody=true" -X POST -d @paramsFile.xml

is there a better description how to use this facility? what should be in the $NotSureWhatIsHERE ??

1. $NotSureWhatIsHERE should be the ID of an experiment (usually an image session).

2. 1.6.4-SNAPSHOT can’t accept parameters when launching a pipeline, either in the query params or an xml file.

OK I see thanks

Also I tried this on our system

GET /data/services/workflows/{pipeline name}?display=LATEST&experiment={session id}

and it returned an error. Is this something I did incorrectly or the version of xnat we have does not support functionality?

As far as I know, that should work on your XNAT version. What error was returned? What did you put in for the pipeline name and session id?

I ran a pipeline via web interface and everything appeared to work correctly.

Then in the session history, I see the listing of the pipelines which were executed in the past.

As far as I understood,

GET /data/services/workflows/{pipeline name}?display=LATEST&experiment={session id}

should reproduce the same list of executed pipelines (the one I see in web browser)

So I use the session ID which is listed as an argument for the pipeline. (In our case this is accession number)

the error message I get is the error I get most of the time:

<html>

<head>

   <title>Status page</title>

</head>

<body>

<h3>The method specified in the request is not allowed for the resource identified by the request URI</h3><p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.6">here</a>.<br>

Please continue your visit at our <a href="/">home page</a>.

</p>

</body>

</html>

this is not very descriptive...

any suggestions?

Thank you very much again for your help

Lazar