Run a container at project level after scan resource modification

[Victor Ferat]

Hello community and developers,

It appears that the Event Subscription Service currently only allows for selecting an action that is defined with the same datatype as the selected event. For instance, if I want to subscribe to an action defined for a resource (context: "xnat:resourceCatalog"), I can only do so for events like "Resource created" or "Resource edited".

My first question is whether it would be possible to implement subscriptions to events with child types. For example, when the "Scan created" event is activated, this scan has a unique session, subject, and project. It seems feasible to run an action defined within the context of one of these three contexts ("xnat:projectData", "xnat:imageSessionData", or "xnat:subjectData"). Does this feature already exist, or do you plan to add it in a future release of XNAT?

If it is not currently possible, a workaround could be to work with resources as they are defined for project, subject, session, and scan. I want to run a container (with the container service plugin) defined at the project level (Project Data or Project resource) when a scan resource is added and/or modified.

The general idea is to create a subscription to "Resource Updated" that will run a command when a scan resource is modified. These container commands need to be defined in the "xnat:resourceCatalog" context. My idea is to derive an input project from this resource and then run my command at the project level using the "provides-files-for-command-mount": "input" parameter.

Here is the code I am using:

"xnat": [
      {
        "name": "test"
        "label": "test",
        "description": "test",
        "contexts": [
          "xnat:resourceCatalog"
        ],
        "external-inputs": [
            {
                "name": "resource",
                "description": "Scan resource",
                "type": "Resource"
            }
        ],
        "derived-inputs": [
          {
              "name": "scan",
              "description": "scan",
              "type": "Scan",
              "derived-from-wrapper-input": "resource"
          },
          {
            "name": "session",
            "description": "session",
            "type": "Session",
            "derived-from-wrapper-input": "scan"
          },
          {
              "name": "project",
              "description": "Project",
              "type": "Project",
              "derived-from-wrapper-input": "session",
              "provides-files-for-command-mount": "input",
          }],
        "output-handlers": [
          {
            "name": "output-resource",
            "accepts-command-output": "output",
            "as-a-child-of": "project",
            "type": "Resource",
            "label": "TEST",
          }]

However, after several trials, I'm still unable to derive the project for the scan's resource:

Derived input "CommandWrapperDerivedInput{id=51, name=project, label=null, description=Project, type=Project, matcher=null, defaultValue=null, required=true, rawReplacementKey=null, sensitive=null, providesValueForCommandInput=null, providesFilesForCommandMount=input, userSettable=null, loadChildren=true, derivedFromWrapperInput=session, derivedFromXnatObjectProperty=null, viaSetupCommand=null, multiple=false, parser=null}" claims parent "session", but I couldn't find "session". Are the inputs out of order?

Thanks for your help,
Victor

[John Flavin]

Hi Victor,

Are you able to provide more of the logs from one of your trials? I can see the error log you've provided, but without any of the previous context it is hard to know why that error occurred. 

The containers.log and/or containers-services-commandresolution.log could be helpful.

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/4e566e83-a3e5-4a44-9045-a5956a1fdbd8n%40googlegroups.com.

[Victor Ferat]

Hey John ! 

Thanks for your feedback, here are the full command definition and the logs:

The idea behind all of these is to run heudiconv whenever a scan resource is created/updated and save the output of heudiconv as a resource of the project ( to which the scan belongs). Maybe there is a easier way to achieve this.

Here is the events subscription:

a well as the full command definition:

{

    "name": "heuditest3-project",

    "label": "heuditest3-project",

    "description": "Runs heudiconv on a project's scans",

    "version": "0.11.3+d20220512",

    "schema-version": "1.0",

    "info-url": "https://github.com/nipy/heudiconv",

    "image": "heuditest3:latest",

    "type": "docker",

    "command-line": "heudiconv -f reproin -l bids -b -o /output --files /input",

    "mounts": [

      {

        "name": "dicom-in",

        "writable": false,

        "path": "/input"

      },

      {

        "name": "nifti-out",

        "writable": true,

        "path": "/output"

      }

    ],

    "environment-variables": {},

    "ports": {},

    "inputs": [

      {

        "name": "reproin",

        "label": null,

        "description": "reproin file",

        "type": "boolean",

        "matcher": null,

        "default-value": "false",

        "required": false,

        "replacement-key": "000",

        "sensitive": null,

        "command-line-flag": "-f",

        "command-line-separator": null,

        "true-value": null,

        "false-value": null,

        "select-values": [],

        "multiple-delimiter": null

      },

      {

        "name": "bids",

        "label": null,

        "description": "Create BIDS metadata file",

        "type": "boolean",

        "matcher": null,

        "default-value": "false",

        "required": false,

        "replacement-key": "111",

        "sensitive": null,

        "command-line-flag": "-b",

        "command-line-separator": null,

        "true-value": "y",

        "false-value": "n",

        "select-values": [],

        "multiple-delimiter": null

      }

    ],

    "outputs": [

      {

        "name": "output",

        "description": "The nifti files",

        "required": true,

        "mount": "nifti-out",

        "path": null,

        "glob": null

      }

    ],

    "xnat": [

      {

        "name": "heuditest3-project",

        "label": "heuditest3-project",

        "description": "Run heudiconv on a Project resource",

        "contexts": [

          "xnat:resourceCatalog"

        ],

        "external-inputs": [

            {

                "name": "resource",

                "description": "Scan resource",

                "type": "Resource"

            }

        ],

        "derived-inputs": [

          {

              "name": "scan",

              "description": "scan",

              "type": "Scan",

              "derived-from-wrapper-input": "resource"

          },

          {

            "name": "session",

            "description": "session",

            "type": "Session",

            "derived-from-wrapper-input": "scan"

          },

          {

              "name": "project",

              "description": "Project",

              "type": "Project",

              "derived-from-wrapper-input": "session",

              "provides-value-for-command-input": null,

              "provides-files-for-command-mount": "input",

              "via-setup-command": null,

              "user-settable": null,

              "load-children": true

          }],

        "output-handlers": [

          {

            "name": "output-resource",

            "accepts-command-output": "output",

            "via-wrapup-command": null,

            "as-a-child-of": "project",

            "type": "Resource",

            "xsi-type": null,

            "label": "BIDS",

            "format": null,

            "description": null,

            "content": null,

            "tags": []

          }

        ]

      }

    ],

    "container-labels": {},

    "generic-resources": {},

    "ulimits": {}

  }

 and the logs:

containers-services-commandresolution.log

2023-03-14 13:24:25,144 [DefaultMessageListenerContainer-5] ERROR org.nrg.containers.services.impl.CommandResolutionServiceImpl - Could not resolve command
org.nrg.containers.exceptions.CommandResolutionException: Derived input "CommandWrapperDerivedInput{id=51, name=project, label=null, description=Project, type=Project, matcher=null, defaultValue=null, required=true, rawReplacementKey=null, sensitive=null, providesValueF>        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.initializePreresolvedInputTree(CommandResolutionServiceImpl.java:1489)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolveInputTrees(CommandResolutionServiceImpl.java:341)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolve(CommandResolutionServiceImpl.java:395)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.access$200(CommandResolutionServiceImpl.java:274)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl.resolve(CommandResolutionServiceImpl.java:267)
        at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:567)
        at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:49)
        at sun.reflect.GeneratedMethodAccessor1756.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)
        at java.lang.Thread.run(Thread.java:750)

containers.log

2023-03-14 13:26:13,360 [http-nio-8080-exec-2] ERROR org.nrg.containers.services.impl.CommandServiceImpl - Cannot create command. Validation failed. Errors:        Command "heuditest3-project" - wrapper "heuditest3-project" - derived input "project" is derived from an unknown XNAT input "session". Known inputs: resource, project        Command "heuditest3-project" - wrapper "heuditest3-project" - derived input "session" is derived from an unknown XNAT input "scan". Known inputs: resource, project, session

Thanks again for your help,

Victor 

[Victor Ferat]

In this command definition, it seems possible to get a parent object ( in this case Session -> Project) instead of the classical Parent -> child definitions described here. Is this feature still exist ?

[John Flavin]

In this command definition, it seems possible to get a parent object ( in this case Session -> Project) instead of the classical Parent -> child definitions described here. Is this feature still exist ?

Yes, what you have described should be possible. You should be able to define a derived input in the Command Wrapper that navigates "up" the XNAT model hierarchy. For example, if you have an input that takes a Session you should be able to derive from that a Project input.

I can't give an answer in your specific case with 100% confidence because I haven't used the event service to launch containers very much. But what you have sent looks like it will be helpful—thanks for the error stack trace and the command and event subscription definitions. I will have to reproduce your setup and either find a workaround for you or see the same error you’re seeing and locate a bug to fix. That could take some time.

In the meantime, could I ask why you want to save the outputs to a project resource? If the command/container are running at a scan level, my intuition would tell me to place the resulting outputs back into XNAT as a new scan resource, or perhaps as a session assessor. Something "close" to the input data. Is there some reason that a project resource makes more sense in your workflow?

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/cf7602ba-4f2b-47d4-b521-eac1a2f4bf57n%40googlegroups.com.

[Victor Ferat]

Hey John,

Thanks for your feedback. I simplified my use case since my last message, but the issue is the same.

 The idea is to be able to run a container on a project resource (R1) which create a new project resource (R2) each time R1 is updated. For that I need to create a command with  "contexts": ["xnat:ResourceCatalog"] to be able to listen to resource update events. 

I need my output-handler (R2) to be of type Resource and as-a-child-of project (i.e. the same project as R1). How could I extract the project of R1 to give it as input to the output-handler of R2 ?

I tried to derivate the project from R1:

  "xnat": [

    {

      "name": "deface-on-resource",

      "label": "deface-on-resource",

      "description": "Run pydeface on a Resource",

      "contexts": [

        "xnat:ResourceCatalog"

      ],

      "external-inputs": [

        {

          "name": "resource",

          "description": "Input resource",

          "type": "Resource",

          "load-children": true,

          "provides-files-for-command-mount": "nifti-in"

        }

      ],

      "derived-inputs": [

        {

          "name": "project",

          "description": "The resource project",

          "type": "Project",

          "required": true,

          "derived-from-wrapper-input": "resource"

        }

      ],

      "output-handlers": [

        {

          "name": "deface-resource",

          "accepts-command-output": "deface",

          "as-a-child-of": "project",

          "type": "Resource",

          "label": "DEFACE",

          "format": null,

          "description": "defaced using pydeface.",

          "content": null,

          "tags": []

        }

      ]

    }

  ],

2023-05-15 18:27:01,112 [DefaultMessageListenerContainer-9] ERROR org.nrg.containers.services.impl.CommandResolutionServiceImpl - An input of type "Project" cannot be derived from an input of type "Resource".
2023-05-15 18:27:01,112 [DefaultMessageListenerContainer-9] ERROR org.nrg.containers.services.impl.CommandResolutionServiceImpl - Could not resolve command
org.nrg.containers.exceptions.CommandResolutionException: Input "project" could not be resolved for this item. You might check the JSONPath matchers in command.json and confirm that this item has Project data.
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.findResolvedValues(CommandResolutionServiceImpl.java:1664)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.findResolvedValues(CommandResolutionServiceImpl.java:1631)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolveInputTrees(CommandResolutionServiceImpl.java:352)

        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolve(CommandResolutionServiceImpl.java:395)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.access$200(CommandResolutionServiceImpl.java:274)
        at org.nrg.containers.services.impl.CommandResolutionServiceImpl.resolve(CommandResolutionServiceImpl.java:267)
        at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:567)
        at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:49)

        at sun.reflect.GeneratedMethodAccessor1604.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
        at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
        at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerCo

ntainer.java:1159)

        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)

But still struggling with the same error as my previous post. It seems a pretty common use case, but I can't find any working example.

[Kate Alpert]

Hi, Victor,

Unfortunately, the only XNAT object that can be derived from a resource within a CS command is a Scan, so while your previous approach should have worked, this one will not (there is no reason that you shouldn't be able to derive any other datatype from a resource, but support for this hasn't been added). 

If you go back to your original command at the start of this thread (where the resource is a child of a scan and not a project) - can you delete the command completely and then re-add it to XNAT? It seems like your derived inputs are being read out of order, and I'm wondering if you added the session/scan inputs as an "edit" on an existing command. Possibly, deleting the command and re-adding it in one shot would address this ordering issue.

Best,

Kate

--

You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/cf0162be-648a-4fc1-a1df-26d4525a6a75n%40googlegroups.com.

[Victor Ferat]

Hey Kate,

Thanks a lot for your feedback.

Unfortunately, the only XNAT object that can be derived from a resource within a CS command is a Scan. So, while your previous approach should have worked, this one will not. There is no reason why you shouldn't be able to derive any other datatype from a resource, but support for this hasn't been added.

That's really unfortunate. I see a lot of use cases where someone would like to create or update a new resource while another one is being updated. As you said, it seems pretty straightforward to derive a Project from one of its resources. Do you know if there is a workaround or if support for this feature is planned to be added in the next release? I am not familiar with Java, so I can't contribute myself to this one.

If you go back to your original command at the start of this thread (where the resource is a child of a scan and not a project), can you delete the command completely and then re-add it to XNAT? It seems like your derived inputs are being read out of order, and I'm wondering if you added the session/scan inputs as an "edit" on an existing command. Possibly, deleting the command and re-adding it in one shot would address this ordering issue.

I went back to this previous example and was able to make it work thanks to your inputs. Thanks a lot! As you explained, it seems to be an issue of the "edit" function of an existing command.

Best,
Victor