Docker container failed to run.

[Shaji Markose]

Hi,

I recently changed docker swarm to a new VM. Loaded images in the new server and joined the as worker node (docker swarmp join). Everything went well and the container run as expected. However, couple of days later the docker images failing to run. . Also noticed that the Images & Commands (Administer->Plugin Settings->Images & Commands) section got wiped out. The error is attached.

 Appreciate any help.

Shaji

[Shaji Markose]

The conatiner.log gives the following:

2024-02-01 22:39:03,280 [ajp-nio-0:0:0:0:0:0:0:1-8009-exec-25] ERROR org.nrg.containers.api.DockerControlApi - Failed to list images. Request error: GET unix://localhost:80/images/json: 200
org.mandas.docker.client.exceptions.DockerRequestException: Request error: GET unix://localhost:80/images/json: 200
        at org.mandas.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:2590)
        at org.mandas.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:2458)
        at org.mandas.docker.client.DefaultDockerClient.listImages(DefaultDockerClient.java:505)
        at org.nrg.containers.api.DockerControlApi._getImages(DockerControlApi.java:256)
        at org.nrg.containers.api.DockerControlApi.getImages(DockerControlApi.java:229)
        at org.nrg.containers.api.DockerControlApi.getAllImages(DockerControlApi.java:216)
        at org.nrg.containers.services.impl.DockerServiceImpl.getImages(DockerServiceImpl.java:241)
        at org.nrg.containers.rest.DockerRestApi.getImages(DockerRestApi.java:275)
        at org.nrg.containers.rest.DockerRestApi$$FastClassBySpringCGLIB$$8ca563f3.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:84)
        at org.nrg.xapi.rest.aspects.XapiRequestMappingAspect.processXapiRequest(XapiRequestMappingAspect.java:86)
        at sun.reflect.GeneratedMethodAccessor427.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)

There is more. I can attach if needed.

Thanks

[Evi Vanoost]

This seems to be an issue when your Docker version goes to v25. We just ran into that today. Downgrade to Docker 24.0.9 for now.
(on RHEL) - yum downgrade docker-ce-3:24.0.9-1.el9.x86_64
I've got similar errors in the logs, seems to be an issue with the container service API needing to be updated?

[Evi Vanoost]

Caused by: jakarta.ws.rs.client.ResponseProcessingException: com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of `org.mandas.docker.client.messages.ImmutableImage$Builder`, problem: Cannot build Image, some of required attributes are not set [virtualSize]
 at [Source: (org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream); line: 1, column: 3138] (through reference chain: java.util.ArrayList[0])
        at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:786)
        at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687)

From https://docs.docker.com/engine/api/version-history/

• The VirtualSize field in the GET /images/{name}/json, GET /images/json, and GET /system/df responses is now omitted. Use the Size field instead, which contains the same information.

[Shaji Markose]

I am using version 1.44. We haven't updated the version recently. 

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/dac600f7-897d-465f-bb4e-0ac501093e5en%40googlegroups.com.

[Richard Cole]

Have you mounted the "/data/xnat" to your new worker?  Confirm your NFS mounts..

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/CAO5Ybe3R55M8xRMQUVTSTNZhfau_FYq8NULFq2nsBsQ1bwdsPA%40mail.gmail.com.

[Evi Vanoost]

Any updates on this bug?
It seems to me the Docker container service needs to be updated, unsure where the code lives to do this.

[John Flavin]

Sorry about the lack of response.

 

Yes, solving this will require an update to the Container Service. However, it isn’t exactly a straightforward fix. The issue is occurring in an upstream dependency, and there are difficulties which prevent us from upgrading that dependency.

 

For the time being, if possible, please avoid upgrading docker while we work on a fix. Sorry that I don’t have a better answer at the moment.

 

John Flavin

 

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Evi Vanoost <evi.v...@gmail.com>
Date: Tuesday, February 20, 2024 at 1:30 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: External: Re: [XNAT Discussion] Re: Docker container failed to run.

You don't often get email from evi.v...@gmail.com. Learn why this is important

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/a1a0f97c-5f60-4fae-9c8a-c58d38df91c9n%40googlegroups.com.

[akluiber]

Hi all, is the solution to this still to downgrade to docker v24? I believe xnat 1.8.10 and container-service 3.4.3 were both released since the last post in this thread.

I recently updated system packages and xnat build, and am running into the same problem with my xnat-docker-compose instance.

[Ford McDonald]

This solution worked for me, as of yesterday.  Just downgrade and restart your docker service.

Here is the working list of installed packages and their versions:

containerd.io.x86_64                  1.6.9-3.1.el8                            @docker-ce-stable

docker-buildx-plugin.x86_64           0.13.1-1.el8                             @docker-ce-stable

docker-ce.x86_64                      3:24.0.9-1.el8                           @docker-ce-stable

docker-ce-cli.x86_64                  1:24.0.9-1.el8                           @docker-ce-stable

docker-ce-rootless-extras.x86_64      26.0.1-1.el8                             @docker-ce-stable

docker-compose-plugin.x86_64          2.26.1-1.el8                             @docker-ce-stable