Container-service not working after upgrade from 3.0.0 to 3.1.1

[Mark]

We currently have an xnat 1.8.4.1 instance using container-service plugin 3.0.0 that needs to be upgrade to 3.1.1 to match our standard deployment pattern, and the compatibility matrix at https://wiki.xnat.org/container-service/container-service-compatibility-matrix-126156827.html

When I restore the 1.8.4.1 database on a test 1.8.4.1 instance with container plugin 3.1.1, runs of dcm2niix are failing with the following exception:

2022-09-08 23:38:39,428 [http-nio-8080-exec-1] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - Issue creating workflow for  dcm2niix-scan
org.postgresql.util.PSQLException: ERROR: null value in column "id" violates not-null constraint
  Detail: Failing row contains (null, , null, Container launch, null, null, PROCESS, DATA, xnat:imageScanData, null, P1, null, null, Queued, null, dcm2niix-scan, null, null, 2022-09-08 23:38:39.398, null, null, 32806, 33173, null).

The "Set Container Launch Values" dialog is missing the scan folder, so it does not appear to be correctly loading scan meta data, and then attempts to run with null data.

If I restart test instance with containers plugin 3.0.0 dcm2niix works fine. I also found xnat 1.8.4.1 works fine with 3.1.1 starting with an empty database (ie. no db restore from the existing system). I am testing with a new scan that was added after restoring the db.

Any suggestions on how to get our existing 1.8.4.1 system working with containers plugin 3.1.1?

Regards, Mark

[John Flavin]

Hi Mark,

It's a bit tough for me to understand what is causing this. I can see from the information you've provided a few steps down the causal chain but I can't quite get back to the root cause.

The log message you sent says in the error message that the workflow CS is trying to save has a null "id" column. That field in the workflow is supposed to contain the ID of whatever object you're launching the container on. In the case of launching on a scan that ID would be the session ID; the scan ID is treated differently and held elsewhere.

Somehow your launch requests are being sent in with a null ID. That seems consistent with what you said about the launch UI not having all the correct data filled in. But that's where my understanding ends. I don't know why that ID wouldn't be found.

Do you see any other messages in the container service logs prior to the error that could be related? Is there anything else that seems off about your sessions/scans in the 1.8.4.1 XNAT? If you look at the XML representation are they missing any important fields like IDs?

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/81ac08ac-8902-42eb-8b42-9a6c1ff0a8e1n%40googlegroups.com.

[Mark]

Hi John,

Thanks for your reply. That is the only error log I am seeing and it seems to be a symptom rather than the cause.

I have started trying to debug with plugin 3.1.1, and find the problem xapi/rest call is at commandUiLauncher.js:1137. Attaching a debugger to the xnat jvm, the call to preResolve() at LaunchRestApi.java:159 eventually calls resolveInputTrees() at CommandResolutionServiceImpl.java:310 which is returning resolvedInputTrees with

resolvedValue=ResolvedInputValue{type=SubjectAssessor, value=null, valueLabel=null, xnatModelObject=null, jsonValue=null}

for plugin 3.0.0 it is

resolvedValue=ResolvedInputValue{type=SubjectAssessor, value=/archive/experiments/XNAT2_E00041/scans/1, valueLabel=/archive/experiments/XNAT2_E00041/scans/1, xnatModelObject=null, jsonValue=/archive/experiments/XNAT2_E00041/scans/1

There are a lot of library calls I don't have source for so I am not seeing why 3.1.1 is not resolving input values.. Please let me know if this is any help, or if there is something else I can try.

Thanks again, Mark

[Charlie Moore]

Hi Mark,

Just to double check: are you clearing your browser cache when you change the plugin? I could see some javascript getting cached on the front-end that's interfering with what gets passed to the backend.

Thanks,

Charlie

[John Flavin]

Mark,

Yes, that did help diagnose the issue. There was a change in CS 3.1 or 3.1.1 that caused issues with how the command wrapper input's "type" fields were represented in the database and translated back into objects. 

There are two potential fixes. The first and probably easiest is to remove the dcm2niix command from your site and re-add it. You would also need to re-enable it on your site and projects. Let me know if you need any help doing that.

If that way is not possible, you can run database commands to manually migrate the incorrectly mapped "type" values. But my recommendation is to do the first fix if at all possible.

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/ac789367-ba5f-460d-bd3d-076cf0d6c14an%40googlegroups.com.

[Mark]

Thanks John,

It is working after I removed and re-added dcm2niix, as suggested!

One thing that would help -- is there a db query or similar I can use to get a list of the projects that have containers enabled?

Charlie, thanks also for the cache suggestion -- I regularly get caught out. I ended up doing each debug run in a new private browser window..

Thanks again, Mark

[John Flavin]

One thing that would help -- is there a db query or similar I can use to get a list of the projects that have containers enabled?

This should work for that:

select distinct
	project
from xhbm_configuration
where tool = 'container-service'
    and path like 'wrapper-%'
    and enabled

If you want to know about a particular command / wrapper, you would need to join to the command and wrapper tables. And you'll have to extract the wrapper ID from the path in the config table. Like so:

select distinct
  config.project,
  command.id as command_id,
  command.name as command_name,
  wrapper.id as wrapper_id,
  wrapper.name as wrapper_name
from xhbm_command_entity command
join xhbm_command_wrapper_entity wrapper on command.id = wrapper.command_entity
join (
  select
    project,
    regexp_replace(path, 'wrapper-', '')::bigint as wrapper_id
  from xhbm_configuration
  where tool = 'container-service' and path like 'wrapper-%' and enabled
) config on config.wrapper_id = wrapper.id
order by config.project, command.id, wrapper.id;

John Flavin

Backend Team Lead

johnf...@flywheel.io

He/Him

flywheel.io

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/3bf7cb97-0175-439d-92ca-7690a06730bdn%40googlegroups.com.

[Mark]

Hi John,

That's perfect thanks.

Cheers, Mark