Container Service 3.5.0 has been released for compatibility with XNAT 1.8.10+

[Will Horton]

Hello all, we're happy to announce an update to the Container Service plugin. In addition to a few feature and bugfix updates, the primary deliverable of this release is support for Docker Engine v25 and higher. Here's a note from the developer: 

--

A Note About Our Docker Library

Originally we used the spotify/docker-client library to wrap the docker remote API in java method calls. They stopped updating that and put out their final release v6.1.1 in 2016.

We switched the Container Service to use a fork of that client, dmandalidis/docker-client in CS version 3.0.0. Given that this was a fork of the client we already used, it was a simple drop-in replacement with no changes needed.

But that library maintainer did continue to make changes. In 2023 they released a major version upgrade, v7.0.0, which dropped support for Java 8. That is the version of Java we use in XNAT (at time of writing) so this change meant we weren't able to update our version of this library. That was fine for a while...
...Until version 25 of the docker engine, in which they made an API change which caused an error in the version we used of docker-client. The library (presumably) fixed their issue but we weren't able to use that fix because our version of the library was frozen by their decision to drop Java 8 support.

This forced us to switch our library from docker-client to docker-java. This was not a drop-in replacement, and did require a migration. All the same docker API endpoints were supported in a 1:1 replacement—which took a little effort but was straightforward—except for one. The docker-java library did not support requesting GenericResources on a swarm service, which is the mechanism by which we allow commands to specify that they need a GPU. We opened a ticket reporting that lack of support (https://github.com/docker-java/docker-java/issues/2320), but at time of writing there has been no response. I created a fork (https://github.com/johnflavin/docker-java) and fixed the issue myself (https://github.com/docker-java/docker-java/pull/2327), but at time of writing that also has no response. I built a custom version of docker-java 3.4.0.1 and pushed that to the XNAT artifactory (ext-release-local/com/github/docker-java).

Long story short, as of CS version 3.5.0 we depend on docker-java version 3.4.0.1 for our docker (and swarm) API support.

--

Full documentation updates are available here: https://wiki.xnat.org/container-service/

Full changelog is available here: https://bitbucket.org/xnatdev/container-service/src/master/CHANGELOG.md

[Simon Doran]

Excellent news!

Thanks for all the hard work. This must have been a very frustrating problem and it's great that you have found a way forward.

Simon

[akluiber]

Have updated and getting this error launching a container. Only changes I've made are updating the container service plugin and docker packages to Version: 5:27.1.2-1~ubuntu.22.04~jammy

Are there additional post-upgrade steps required?

2024-08-26 13:53:11,758 [DefaultMessageListenerContainer-2] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - consumeResolveCommandAndLaunchContainer failed for wfid 25667.
java.lang.NullPointerException: authConfig was not specified
at java.util.Objects.requireNonNull(Objects.java:228)
at com.github.dockerjava.core.command.CreateServiceCmdImpl.withAuthConfig(CreateServiceCmdImpl.java:45)
at org.nrg.containers.api.DockerControlApi.createDockerSwarmService(DockerControlApi.java:803)
at org.nrg.containers.api.DockerControlApi.create(DockerControlApi.java:461)
at org.nrg.containers.api.DockerControlApi.create(DockerControlApi.java:434)
at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedDockerCommand(ContainerServiceImpl.java:670)
at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:636)
at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:621)
at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:596)
at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)
at java.lang.Thread.run(Thread.java:750)

[akluiber]

I should probably add this is running within an xnat-docker-compose environment.

[akluiber]

Downgrading back to docker v24 and container service v3.4.3 does resolve my issue.

[Will Horton]

Hmm, that's unfortunate. I just ran a very simple xnat-docker-compose instance through a series of tests and it seems to work fine. Granted, this was far from a stress test, but I wanted to see if I could replicate the issue through basic configuration.

What I tested:

Using Docker Desktop, running Docker v20 on my M2 Mac

(√) XNAT 1.8.10.1 with CS 3.4.3 -- ran debug-command and dcm2niix successfully

(√) XNAT 1.8.10.1 with CS 3.5.0 -- ran debug-command and dcm2niix successfully

After updating Docker Desktop, running Docker v27.1.1

(x) XNAT 1.8.10.1 with CS 3.4.3 -- fails to display installed images, can't run containers

(√) XNAT 1.8.10.1 with CS 3.5.0 -- ran debug-command and dcm2niix successfully

What version of XNAT are you running? And I see that you're running Ubuntu 22.04, it's my understanding that you have to manually install Tomcat9 with Java 8 in that environment. I'm wondering if that might be an issue, although I'm not sure how downgrading to CS 3.4.3 would fix the problem if that was the root cause. Just musing on possibilities... 

[akluiber]

xnat-docker-compose with v1.8.10.1

I tried again under a few different scenarios. My results so far are as follows:

Ubuntu 22.04, docker swarm, xnat-docker-compose v1.8.10.1, docker-ce v24, CS 3.4.3 - pass

Ubuntu 22.04, docker swarm, xnat-docker-compose v1.8.10.1, docker-ce v27, CS 3.5.0 - fail - authConfig was not specified

Ubuntu 22.04, docker swarm, xnat-docker-compose v1.8.10.1, docker-ce v24, CS 3.5.0 - fail - authConfig was not specified

Ubuntu 22.04, docker,  xnat-docker-compose v1.8.10.1, docker-ce v24, CS 3.5.0 - pass

Ubuntu 22.04, docker,  xnat-docker-compose v1.8.10.1, docker-ce v27, CS 3.5.0 - pass

Different server:

Ubuntu 20.04, docker swarm,  XNAT v1.8.10.1, docker-ce v27, CS 3.5.0 - pass

As above, I also upgraded a different non-docker XNAT server (ubuntu 20.04, docker-ce v27), which is a swarm manager, which isn't experiencing any problems. So I'm not sure what to make of these results. Seems to indicate an issue with the plugin and a docker swarm setup, within xnat-docker-compose...I guess?

[Will Horton]

Okay, that's helpful. I'll set up a swarm node and see if that helps me replicate the issue on my end

[akluiber]

Hi All,

Just wanted to report that I still get the same errors using XNAT 1.9 and updated plugins with docker swarm within xnat-docker-compose.

[Will Horton]

Just a quick note that I saw this. I got stuck trying to replicate an environment in docker-compose that reproduces this behavior, and then got distracted by release activities. I'll jump back on this and see what I can find out.  

[akluiber]

Let me know if there's any other troubleshooting information you can think of for me to provide.

[akluiber]

Hi all,

Bringing this back up. I seem to have resolved it.

I've since updated to 1.9.1.2 with updated plugins and was still seeing this issue with my xnat-docker-compose that runs on a swarm manager with the container service in swarm mode. I narrowed it down to having to remove and re-add my private docker registry, even though it seemed to be just fine according to the UI. After doing that, I was able to run my containers. Example error below.

2025-04-22 16:24:12,310 [org.springframework.jms.JmsListenerEndpointContainer#6-9] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - consumeResolveCommandAndLaunchContainer failed for wfid 38243.

java.lang.NullPointerException: authConfig was not specified

at java.util.Objects.requireNonNull(Objects.java:228)

at com.github.dockerjava.core.command.CreateServiceCmdImpl.withAuthConfig(CreateServiceCmdImpl.java:45)

at org.nrg.containers.api.DockerControlApi.createDockerSwarmService(DockerControlApi.java:803)

at org.nrg.containers.api.DockerControlApi.create(DockerControlApi.java:461)

at org.nrg.containers.api.DockerControlApi.create(DockerControlApi.java:434)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedDockerCommand(ContainerServiceImpl.java:670)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:636)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:621)

at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:596)

at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:50)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)

at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)

at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:110)

at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:84)

at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736)

at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696)

at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674)

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:331)

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:270)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1237)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1227)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1120)

at java.lang.Thread.run(Thread.java:750)