Command resolution fails after upgrade to XNAT 1.8.1 and Container Service 3.0.0

319 views
Skip to first unread message

Alexander Bartnik

unread,
May 20, 2021, 10:01:53 AM5/20/21
to xnat_discussion
Hello,

We've been using XNAT 1.7.6 with Container Service 2.1.0 for a little over a year and had it working pretty well for our use case, mainly controlling the container service with commands sent via REST API. We decided to test out XNAT 1.8.1 using the docker-compose repo, which we have left as mostly the default config. The only changes made to the default config have been to add path translation (/data/xnat in docker -> /xnat-data on localhost). I have verified that the paths line up correctly.

I've been trying to run the xnat/dcm2niix:latest docker using the default command.json (this works on our old instance running XNAT 1.7.6 and CS 2.1.0). I've tried running the command manually from the XNAT UI, as an automatic command set via the plugin interface, as an event subscription when a scan is created, and via the REST API. All of the above result in the following error in /data/xnat/home/logs/containers-services-commandresolution.log:
2021-05-20 13:15:06,608 [DefaultMessageListenerContainer-10] ERROR org.nrg.containers.services.impl.CommandResolutionServiceImpl - Cannot derive input "scan-dicom". Parent input's XNAT object is null.

Likewise, this is the traceback in /data/xnat/home/logs/containers/log:
2021-05-20 13:15:06,630 [DefaultMessageListenerContainer-10] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - Container command resolution failed for wfid 52.
org.nrg.containers.exceptions.CommandResolutionException: Input "scan-dicom" could not be resolved for this item. You might check the JSONPath matchers in command.json and confirm that this item has Scan data.
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.findResolvedValues(CommandResolutionServiceImpl.java:1699)
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.findResolvedValues(CommandResolutionServiceImpl.java:1666)
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolveInputTrees(CommandResolutionServiceImpl.java:335)
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.resolve(CommandResolutionServiceImpl.java:377)
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl$CommandResolutionHelper.access$200(CommandResolutionServiceImpl.java:248)
    at org.nrg.containers.services.impl.CommandResolutionServiceImpl.resolve(CommandResolutionServiceImpl.java:245)
    at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:496)
    at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:49)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
    at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
    at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
    at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
    at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
    at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
    at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)
    at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)
    at java.lang.Thread.run(Thread.java:748)
2021-05-20 13:20:35,583 [DefaultMessageListenerContainer-2] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - Container command resolution failed for wfid 54.

None of these attempts show up in the command history in the plugin settings UI in XNAT.

This also happens while trying to run any of our other commands that work on the older instance, but I figured the dcm2niix command is the easiest to troubleshoot and ask for help with.

Did something change in either XNAT 1.8.1 or CS 3.0.0 regarding path translation or command resolution? I can't see anything we've done differently setting this instance up, but it looks like the container service isn't parsing the DICOM labels like it did in 2.1.0. When running the commands manually via the XNAT UI, the first pop up menu says something about running the command in bulk (screenshot attached), even though I am running the command on just one image. I'm also attaching a screenshot of the folder structure of the session I am working with.

Any help would be greatly appreciated!
dcmdir_structure.png
command_resolution.png

Alexander Bartnik

unread,
May 20, 2021, 3:57:39 PM5/20/21
to xnat_discussion
I just tested Container Service 2.1.0 on XNAT 1.8.1 and it appears to be broken. The history tab in the plugin settings UI is giving a 404 and any attempts to launch a container just hand indefinitely.

Moore, Charlie

unread,
May 20, 2021, 4:05:17 PM5/20/21
to xnat_di...@googlegroups.com
That's a much easier question to answer. There's a compatibility table available here: https://wiki.xnat.org/container-service/container-service-compatibility-matrix-126156827.html . CS 2.1.0 and XNAT 1.8.0 (or 1.8.1) are not compatible. As for the issue you're seeing with XNAT 1.8.1 and CS 3.0.0, that's quite strange. I don't know what would cause that. The main developer for the CS plugin is out today, but hopefully he'll have some ideas tomorrow.

Thanks,
Charlie
From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Alexander Bartnik <abar...@bnac.net>
Sent: Thursday, May 20, 2021 2:57 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Re: Command resolution fails after upgrade to XNAT 1.8.1 and Container Service 3.0.0
 

* External Email - Caution *
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/fe58fc75-9471-4d42-848e-460972315607n%40googlegroups.com.

 

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Alexander Bartnik

unread,
May 20, 2021, 8:16:23 PM5/20/21
to xnat_discussion
Ah, that makes sense, thanks for the link! I'll poke around in my logs in the meantime and update here if anything else useful turns up.

kel...@wustl.edu

unread,
May 21, 2021, 11:23:29 AM5/21/21
to xnat_discussion
Hi there,
 I'm having trouble recreating this error on my end (with XNAT 1.8.1 and Container Service 3.0). Can you verify that the Container Service 2.1.x plugin has been removed from the plugins directory? Note that the plugin jar naming is a bit different (containers-2.1.0-fat.jar -> container-service-3.0.0-fat.jar).

Do you see anything interesting in any of the other XNAT logs?

Best,
Matt


Alexander Bartnik

unread,
May 21, 2021, 1:05:03 PM5/21/21
to xnat_discussion
Hey there,

Thanks for getting back to me. I'm glad to see that command resolution works on XNAT 1.8.1 with container service 3.0.0, I was afraid I wouldn't be able to upgrade yet. Unfortunately the only entries in my containers.log and container-services-commandresolution.log all say the same thing, which is the error I copy/pasted in my first post.

I'm attaching a screenshot of my xnat directory, it's just the container-service-3.0.0-fat.jar and the old selectable table plugin (I've also tried removing the selectable table plugin just in case). I thought maybe I had a corrupted download of the container service jar so I tried with a fresh download, still getting the same error.

This is on an AWS EC2 instance FWIW. I've also tried blowing away the instance and starting fresh, but I'm still getting the same error. I tried a couple additional things to troubleshoot.

First, I tried launching commands via the swagger UI. For the sake of saving space I'm uploading the results to pastebin.
The most notable part of this to me is that the wrapper-id is set to the 0 and the workflow-id is never created.

Second, I tried launching the generic debug-comands (on project, session, and scan) to troubleshoot if the container service works at all for me. Those work, since they're not trying to resolve command inputs (screenshot of out.txt successfully written to subject files). I tried running the debug-container on a scan with the dcm2niix setup-command and that failed with the same error. So the container service is definitely working on my instance, there's just something weird with command resolution.

Thanks for taking the time to look into this!
xnat_tree.png
debug-command_output.png

Moore, Charlie

unread,
May 21, 2021, 1:39:45 PM5/21/21
to xnat_di...@googlegroups.com
Ah, regardless of whether or not it has an effect here, you should definitely get rid of the selectable table plugin. The functionality of that plugin has been in base XNAT since around 1.7.4.1, so that plugin may be overriding things on the session page. You'd want to clear your browser cache after restarting tomcat without that plugin too.

Thanks,
Charlie
Sent: Friday, May 21, 2021 12:05 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] Re: Command resolution fails after upgrade to XNAT 1.8.1 and Container Service 3.0.0
 

Alexander Bartnik

unread,
May 21, 2021, 5:34:26 PM5/21/21
to xnat_discussion
Makes sense, I've completely removed it.

Some good news though! I noticed the resolution failed when trying to find specific scan resources using derived-inputs, but was successful using the external inputs. I've found that if I remove the derived-inputs and have the external-input provide files for the command mount the container launches and runs perfectly (also needed to change the type from 'string' to 'Scan'). I'm still unsure if this just due to something with my specific setup and if I should just update all of my command.jsons to work this way.

Though I suppose I'll run into this again whenever I'll need to use derived-inputs in a command. It looks like that's the specific part that's causing this issue.

Alexander Bartnik

unread,
May 21, 2021, 6:15:26 PM5/21/21
to xnat_discussion
Did some more troubleshooting with the debug-container and found that command resolution works for on the project level.

So this seems to work:
      "external-inputs": [
    {
      "name": "project",
      "description": "Input project",
      "type": "Project",
      "required": true,
      "provides-value-for-command-input": null,
      "provides-files-for-command-mount": "in",
      "load-children": false
    }
  ],
  "derived-inputs": [
    {
      "name": "project-id",
      "type": "string",
      "matcher": null,
      "required": true,
      "provides-value-for-command-input": "project",
      "load-children": true,
      "derived-from-wrapper-input": "project",
      "derived-from-xnat-object-property": "id",
    }
  ]

While this derived-input from the dcm2niix command doesn't:
  "external-inputs": [
    {
      "name": "scan",
      "description": "Input scan",
      "type": "Scan",
      "required": true,
      "matcher": "'DICOM' in @.resources[*].label"
    }
  ],
  "derived-inputs": [
    {
      "name": "scan-dicoms",
      "description": "The dicom resource on the scan",
      "type": "Resource",
      "derived-from-wrapper-input": "scan",
      "provides-files-for-command-mount": "dicom-in",
      "matcher": "@.label == 'DICOM'"
    }
  ]
Reply all
Reply to author
Forward
0 new messages