How to automate pipeline execution when scan/session is already archived?

[Ruba]

Hi,

I have seen that there is 'Command Automation' option under 'Plugin Settings' to execute pipeline automatically based on XNAT event. In my case, I have already some sessions/scans archived in XNAT. I need to run the pipeline on those scan resources. Is there any way to automate this process? I have more than 500 scans in XNAT. It would be difficult to run the pipeline on each of  these scans manually.

I would appreciate it if anyone could give me some advice in this regards. Thanks.

Regards,

Ruba

[kel...@wustl.edu]

Hi Ruba,  

The upcoming version of XNAT 1.8 and associated Container Service will allow you to "batch launch" containers across multiple scan resources.  

On the current version of XNAT I've instead used a session-level container that derives the appropriate scan. This can be launched with the selectable table bulk-launch functionality.

From a Session Tab, choose "Run Container"

 
Which will allow you to choose sessions to run your command on:

A few important notes: 

• You will have to re-configure your command to work at the session level.
  
• The bulk-launcher will attempt to resolve inputs for all selected sessions before providing a launch dialog. For 500 sessions, this could be quite awhile.  You may want to run smaller batches to start with.
  
• The bulk-launcher will attempt to launch containers for all selected sessions as quickly as it can.  This can easily overwhelm a development environment or single Docker server.  We recommend that this type of processing be done on a Docker Swarm cluster and that appropriate "reserve-memory" constraints be added to Docker commands to insure resource availability.

Best,

Matt

[Martin Boswell]

Ruba,

When I do this sort of thing, I automate it by writing a shell script to construct and launch cURL commands to launch a pipeline using the XNAT API.  You can launch a pipeline by running a curl command like this template:

curl -u USER:PASSWORD -X POST "https://SERVER.XNAT.COM/data/projects/PROJECT-NAME/pipelines/PIPELINE-NAME/experiments/EXPERIMENT-NAME?overwrite=Y"

There's some older discussion of this at "Start a Pipeline from REST API?"  If you want to run a pipeline that is set to automatically run when an experiment is archived, then be sure that you understand the issue of this "Step ID" vs. the Pipeline Name (see John's message in that thread from Apr 20, 2016).  

 - Martin

[Rubaida Easmin]

Hi,

Many thanks to both of you for your kind reply. I really appreciate it.

I will have a look at these solutions. Hope this will solve my problem.  

Regards,

Ruba

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/5AFQJjl8Xes/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/ce8ebc05-1b0a-45cd-80d6-89e12fb400ean%40googlegroups.com.

[Ruba]

Hi Matt,

I have tried to use bulk launch functionality to automate the pipeline launch for already achieved scans. The problem is, I have a single docker server and it is not possible to create a docker swarm cluster now.

Instead of bulk launching, I want to create a shell script and use REST API to run the pipeline automatically (as @Martin suggested). When I run this command ( curl  -u admin:admin -X GET http://10.1.1.17/data/projects/PROJECT_NAME/pipelines) following command, I get this result ({"ResultSet":{"Result":[]}}). I installed my pipeline using container service and there is nothing under the 'Pipeline' section for a particular project. All the docker images of pipelines are listed in 'Plugin setting'. 

So, is it possible to run the plugin/container using REST API in XNAT? 

Thanks in advance.

Regards,

Ruba

[Ruba]

Hi, 

I have found this link to launch the container via REST API. So, I tried the following command:

curl  -u admin:admin -X GET http://10.1.1.17/data/projects/PROJECT_NAME/commands/summary-pipeline

Here, 'summary-pipeline' is the name of my container image. I got this message:

<html>

<head>

   <title>Status page</title>

</head>

<body>

<h3>The request could not be understood by the server due to malformed syntax</h3><p>You can get technical details <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.1">here</a>.<br>

Please continue your visit at our <a href="/">home page</a>.

</p>

</body>

</html>

Can you please help me with this?

Thanks,

Ruba

[Moore, Charlie]

Hi Ruba,

All of the Container Service APIs use the XAPI framework. They all start with $SERVER_URL/xapi, not $SERVER_URL/data.

Thanks,

Charlie

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Ruba <rubaida...@gmail.com>
Sent: Sunday, May 23, 2021 1:21 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] Re: How to automate pipeline execution when scan/session is already archived?

 

* External Email - Caution *

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/7d90eb02-28a2-458c-a3eb-e0f681358274n%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[kel...@wustl.edu]

Hi Ruba,

Another note, if you would prefer to utilize the queuing functionality that comes along with Docker Swarm, you can enable swarm mode on the command line of your Docker server:

$ docker swarm init

This command will configure your Docker server as a single-node swarm server.

Best,

Matt

[Rubaida Easmin]

Thanks to you both.

I tried the script with xapi and it can launch the container now.

@Kelsey, Matt, I have enabled swarm mode and run the command you mentioned. I run the pipeline for two scans and it immediately created two containers. Although the containers are still running (tested using docker ps), I got a 502 Bad Gateway error. I am testing all these commands in xnat-vagrant VM for now. 

Regards,

Ruba

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/6d9e4cb9-d53b-4270-b6a2-380b4899bf99n%40googlegroups.com.

[Rubaida Easmin]

Hi Britta,

I have followed this link to set up plugins in xnat ( version 1.7.6 ). Not sure if this will help you, but according to this tutorial, when you pull the image in xnat, you have to write commands for the image to instruct how to get the inputs, where to initiate the container, and where to write the results. According to these commands, you will get the 'Run Container' option in session/scan tab. I have also attached the link here to write commands to execute the image.

Best wishes,

Ruba

On Tue, Jun 15, 2021 at 9:53 AM Britta M <britta....@gmail.com> wrote:

Hello everybody,

I followed your previous posts regarding the bulk launch functionality, but I am not able to launch a container on multiple archived sessions/scans. I don't have the "Run Container" option in the session tab, although I've installed the batch launch plugin and the container service plugin in XNAT version 1.8.0 and configured the docker server as a single-node swarm server. Do I have to configure anything else? Or how can I "batch launch" a container in XNAT 1.8?

Many thanks!

Britta

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/004b5d16-45f0-48fe-92c5-c1ab66d8915fn%40googlegroups.com.

[Britta M]

Hi Ruba,

Thank you very much for your reply. The problem was that I had no command that was reconfigured to work on session level, so that I could only see the Run Container option for single scans. 

However, I am not really sure how to configure a command, for example, dcm2niix to loop over multiple session of a project/subject - I just ran the standard "debug" example on session level. If you have any simple example for that, I would be very happy! 

Best, 

Britta

[Ruba]

Hi Britta,

I am not sure how you organise your data in scan level. However, an example to configure the command inside xnat is given below. In this case, 'Run Container' option will appear under scan level. I have attached a screenshot  to show how I organise the scans in xnat. Based on this structure I wrote the commands for docker images.

Let me know if anything is not clear.

Best,

Ruba

Data in xnat:

 

Command:

{

  "name": "conversion-pipeline",

  "label": " conversion-pipeline ",

  "description": "convert scans",

  "version": "1.0",

  "schema-version": "1.0",

  "info-url": " https://docker.com/repositories",

  "image": "name of image",

  "type": "docker",

  "command-line": "niftiConversion /input /output",

  "mounts": [

    {

      "name": "pet-in",

      "writable": false,

      "path": "/input"

    },

    {

      "name": "result-out",

      "writable": true,

      "path": "/output"

    }

  ],

  "environment-variables": {},

  "ports": {},

  "inputs": [],

  "outputs": [

    {

      "name": "result",

      "description": "Converted NIFTI files",

      "required": true,

      "mount": "result-out",

      "path": null,

      "glob": null

    }

  ],

  "xnat": [

    {

      "name": "fdopa-conversion-pipeline",

      "label": "data-conversion-pipeline",

      "description": "Convert fdopa scans",

      "contexts": [

        "xnat:imageScanData"

      ],

      "external-inputs": [

        {

          "name": "scan",

          "description": "Input scan",

          "type": "Scan",

          "matcher": "'PET' in @.resources[*].label",

          "default-value": null,

          "required": true,

          "replacement-key": null,

          "sensitive": null,

          "provides-value-for-command-input": null,

          "provides-files-for-command-mount": null,

          "via-setup-command": null,

          "user-settable": null,

          "load-children": true

        }

      ],

      "derived-inputs": [

        {

          "name": "scan-pet",

          "description": "The pet resource on the scan",

          "type": "Resource",

          "matcher": "@.label == 'PET'",

          "default-value": null,

          "required": true,

          "replacement-key": null,

          "sensitive": null,

          "provides-value-for-command-input": null,

          "provides-files-for-command-mount": "pet-in",

          "user-settable": null,

          "load-children": true,

          "derived-from-wrapper-input": "scan",

          "derived-from-xnat-object-property": null,

          "via-setup-command": null

        }

      ],

      "output-handlers": [

        {

          "name": "result-resource",

          "accepts-command-output": "result",

          "via-wrapup-command": null,

          "as-a-child-of": "scan",

          "type": "Resource",

          "label": "CONVERTED_NIFTI",

          "format": null

        }

      ]

    }

  ]

}

[Britta M]

Thanks a lot for sharing your example! My problem, however, is how to re-configure this command to work on session level, so that the conversion is done for multiple scans within a session....

[Rubaida Easmin]

This is an example to start a container from session level. You can just ignore the marked comments if you have only input and output folder in docker command.

Hope this helps.

Best,

Ruba

{
  "name": "fdopa-analysis",
  "label": " fdopa-analysis ",
  "description": "Runs fdopa pipeline on a scan",

  "version": "1.0",
  "schema-version": "1.0",
  "info-url": " https://docker.com/repositories",

  "image": "",
  "type": "docker",
  "command-line": "script /ancFile /input /template #SESSION_LABEL# /output",
  "mounts": [
    {
      "name": "anc-in",
      "writable": false,
      "path": "/ancFile"
    },

    {
      "name": "pet-in",
      "writable": false,
      "path": "/input"
    },
   

{
      "name": "template-in",
      "writable": false,
      "path": "/template"
    },

    {
      "name": "result-out",
      "writable": true,
      "path": "/output"
    }
  ],
  "environment-variables": {},
  "ports": {},
  "inputs": [

    {
      "name": "session-label",
      "description": "",
      "type": "string",
      "matcher": null,
      "default-value": null,
      "required": true,
      "replacement-key": "#SESSION_LABEL#",
      "sensitive": null,
      "command-line-flag": null,
      "command-line-separator": null,
      "true-value": null,
      "false-value": null
    }

  ],
  "outputs": [
    {
      "name": "result",

      "description": "The CSV and PDF files",

      "required": true,
      "mount": "result-out",
      "path": null,
      "glob": null
    }
  ],
  "xnat": [
    {

      "name": "fdopa-pipeline",
      "label": "fdopa-pipeline",
      "description": "Run fdopa pipeline on a Scan",
      "contexts": [
        "xnat:imageSessionData"
      ],
      "external-inputs": [
        {
          "name": "session",
          "description": "Input session",
          "type": "Session",
          "matcher": null,

          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,
          "provides-files-for-command-mount": null,
          "via-setup-command": null,
          "user-settable": null,
          "load-children": true
        }
      ],
      "derived-inputs": [
        {

          "name": "scan",
          "description": "Get PET resources",

          "type": "Scan",
          "matcher": "'PET' in @.resources[*].label",
          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,
          "provides-files-for-command-mount": null,

          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "session",

          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        },

        {
          "name": "anc-file",
          "description": "anc file from PET resource",

          "type": "Resource",
          "matcher": "@.label == 'PET'",
          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,

          "provides-files-for-command-mount": "anc-in",

          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "scan",
          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        },

        {
          "name": "project",
          "description": null,
          "type": "Project",
          "matcher": null,

          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,
          "provides-files-for-command-mount": null,

          "user-settable": false,
          "load-children": true,
          "derived-from-wrapper-input": "session",

          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        },

        {
          "name": "project-resource",
          "description": "The template resource on the project",
          "type": "Resource",
          "matcher": "@.label == 'TEMPLATES'",

          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,

          "provides-files-for-command-mount": "template-in",
          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "project",

          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        },

        {
          "name": "scan-pet",
          "description": "The pet resource on the scan",
          "type": "Resource",

          "matcher": "@.label == 'CONVERTED_NIFTI'",

          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,
          "provides-value-for-command-input": null,
          "provides-files-for-command-mount": "pet-in",
          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "scan",
          "derived-from-xnat-object-property": null,
          "via-setup-command": null
        }

,
        {
          "name": "session-label",
          "description": null,
          "type": "string",
          "matcher": null,

          "default-value": null,
          "required": true,
          "replacement-key": null,
          "sensitive": null,

          "provides-value-for-command-input": "session-label",
          "provides-files-for-command-mount": null,
          "user-settable": null,
          "load-children": true,
          "derived-from-wrapper-input": "session",
          "derived-from-xnat-object-property": "label",
          "via-setup-command": null
        }

      ],
      "output-handlers": [
        {
          "name": "result-resource",
          "accepts-command-output": "result",
          "via-wrapup-command": null,

          "as-a-child-of": "session",
          "type": "Resource",
          "label": "RESULT",
          "format": null
        }
      ]
    }
  ]
}

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/d5386867-b9ad-4dc1-936e-d33d356c4d7bn%40googlegroups.com.

[Britta M]

Thanks a lot!