Container Service "error while creating mount source path '/mount_path/project_name/arc001/session_name/SCANS/3/DICOM': mkdir /mount_path/archive: permission denied"

[Steve]

Hello,

I'd like to start off with some details of our set up:

OS=CentOS Stream 8

XNAT version=1.8.10.1

Container Service=container-service-3.4.3-fat.jar

Docker version=Docker version 26.1.3, build b72abbb

Plugin settings:

Our XNAT host machine is a VM with a nfs mount `/mount_path/`. XNAT is run with a user called `svc_user` that belongs to group `user_group`. See below for details:

```bash

ls -rtla /mount_path`

drwxr-xr-x 8 svc_user user_group 0 Nov 27 15:40

```

We've been getting an error at container launch (see below)

full error message:

```bash

2025-01-14 15:06:53,637 [DefaultMessageListenerContainer-7] ERROR org.nrg.containers.services.impl.ContainerServiceImpl - consumeResolveCommandAndLaunchContainer failed for wfid 45331.

com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"error while creating mount source path '/mount_path/project_name/arc001/session_name/SCANS/3/DICOM': mkdir /mount_path/archive: permission denied"}

at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)

at com.github.dockerjava.core.DefaultInvocationBuilder.post(DefaultInvocationBuilder.java:102)

at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:31)

at com.github.dockerjava.core.exec.StartContainerCmdExec.execute(StartContainerCmdExec.java:13)

at com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)

at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:33)

at com.github.dockerjava.core.command.StartContainerCmdImpl.exec(StartContainerCmdImpl.java:42)

at org.nrg.containers.api.DockerControlApi.startDockerContainer(DockerControlApi.java:875)

at org.nrg.containers.api.DockerControlApi.start(DockerControlApi.java:828)

at org.nrg.containers.services.impl.ContainerServiceImpl.start(ContainerServiceImpl.java:716)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedDockerCommand(ContainerServiceImpl.java:703)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:636)

at org.nrg.containers.services.impl.ContainerServiceImpl.launchResolvedCommand(ContainerServiceImpl.java:621)

at org.nrg.containers.services.impl.ContainerServiceImpl.consumeResolveCommandAndLaunchContainer(ContainerServiceImpl.java:596)

at org.nrg.containers.jms.listeners.ContainerStagingRequestListener.onRequest(ContainerStagingRequestListener.java:49)

at sun.reflect.GeneratedMethodAccessor1575.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)

at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)

at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)

at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)

at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)

at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)

at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)

at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)

at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)

```

Here we can see the XNAT_HOME `mount_path`

```bash

ls -rtla /mount_path/

total 0

drwxr-xr-x 6 root root 71 Jul 31 10:35 ..

drwxr-xr-x 2 svc_user user_group 0 Jul 31 17:16 inbox

drwxr-xr-x 2 svc_user user_group 0 Jul 31 17:16 quarantine

drwxr-xr-x 2 svc_user user_group 0 Sep 26 12:00 export

drwxr-xr-x 17 svc_user user_group 0 Oct 21 17:38 cache

drwxr-xr-x 8 svc_user user_group 0 Nov 27 15:40 backup

drwx------ 12 root root 0 Nov 27 15:40 .

drwxr-xr-x 17 svc_user user_group 0 Nov 27 16:30 pipeline

drwxr-xr-x 10 svc_user user_group 0 Jan 14 12:31 archive

drwxr-xr-x 2 svc_user user_group 0 Jan 14 12:31 prearchive

drwxr-xr-x 53 svc_user user_group 0 Jan 14 15:06 build

drwxr-xr-x 1 root root 0 Jan 14 15:17 .snapshot

```

Yet it seems like the container service is able to write to the mount. See below for an example of a directory written to the build directory:

```bash

ls -rtla /mount_path/build/

drwxr-xr-x 2 svc_user user_group 0 Jan 10 12:02 6f8f999d-3813-4fe5-84a8-cbf9288ab33b

```

Here you can see the above container stuck in the running status.

Can anyone provide some debugging tips? Any help would be greatly appreciated.

[John Flavin]

Hi Steve,

It appears to me that the Container Service isn't the entity attempting to create the path that ultimately fails, docker is. 

Some background: When CS gathers all the information it needs to launch a container, it uses the input values to load the relevant XNAT data objects, and from those objects find the paths to the files/directories on disk that docker will need to mount into the container. If your container needs a writable path, CS creates an empty directory in the build space and instruct docker to mount that path into the container. If you want data out of the archive, CS merely finds the path to that data and hands that path over to docker for mounting; CS shouldn't create anything in the archive.

When CS instructs docker to start a container which mounts some paths, then docker has its own behaviors that come into play. For instance, if you instruct docker to mount a path on your host that docker thinks does not exist, docker will create that path (with root ownership) and mount it.

My guess is that's what is happening here. Somehow or other when you tell CS to launch a container, it determines that it needs to mount a path like /mount_path/project_name/arc001/session_name/SCANS/3/DICOM. It tells docker to launch a container with that path mounted. Docker creates the container but sees that the given path to be mounted doesn't exist, tries to create it, and fails for some reason. But the fact that docker fails to create this path is, to me, not the root of the issue. We don't want docker to be attempting to create these archive paths in the first place. We want CS to determine the paths to the data your container needs, and for docker to mount the data from those paths. If there isn't any data at a path because the path doesn't exist then there is no reason for CS to tell docker to mount it.

This is what I would investigate:

• Does the path you get from the error message (equivalently you should be able to get the same path from the Container Mounts section of the Container information) exist?
• If it does exist, why is docker trying to create it? Somehow docker thinks that path doesn't exist. Does docker see the mounts at a different location? Do you need path translation?
• If it doesn't exist, why does CS think that is a path the container needs to mount to find input data? Is there stale data in your catalog files? Is the Command defined correctly; specifically, how does the input mount get its values?

John Flavin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/f6bb96fb-aae2-4b55-a646-3b010852772cn%40googlegroups.com.

[Steve]

Hi John,

Thank you for your detailed response and the debugging approach you suggested. I appreciate the clarity in your explanation. Below are my answers to your questions:


1. Does the path you get from the error message exist?

Yes, the path /mount_path/project_name/arc001/session_name/SCANS/3/DICOM does exist and is owned by svc_user:user_group. However, Docker seems to believe it does not exist and attempts to create it, leading to the permission error.


2. If the path exists, why is Docker trying to create it?

This is what I'm currently trying to determine. Some observations:

* svc_user has the correct permissions to read and write in /mount_path/archive, as verified with ls -ld /mount_path/archive (output provided in my original message).
* The Container Service can successfully write to /mount_path/build, which suggests the NFS mount is writable by the svc_user.
* However, when Docker attempts to mount /mount_path/project_name/arc001/session_name/SCANS/3/DICOM, it reports that the directory does not exist and attempts to create it, which fails due to permissions.

I noticed this from the container-service source code:
`at org.nrg.containers.api.DockerControlApi.createDirectoriesForMounts`
```java
private void createDirectoriesForMounts(final Container toCreate) throws IOException {
        final List<Container.ContainerMount> containerMounts = toCreate.mounts() == null ? Collections.emptyList() : toCreate.mounts();
        for (final Container.ContainerMount mount : containerMounts) {
            final Path mountFile = Paths.get(mount.xnatHostPath());
            if (!Files.isRegularFile(mountFile)) {  // <------------------------------------ returns False
                Files.createDirectories(mountFile);
            }
        }
    }

```
QUESTION: Based on my understanding, this snippet is where the directory creation is attempted if Docker does not recognize the path. Could you confirm whether this interpretation is correct?


3. Does Docker see the mounts at a different location? Do you need path translation?

I am investigating whether Docker perceives the mount path differently from the host, possibly requiring a path translation.

======================================================

I have tried to run containers based of images built from a DockerFile similar to following:

```Dockerfile
FROM neurodebian:nd20.04
RUN apt-get -y update && apt-get -y install dcm2niix

RUN mkdir -p /input /output

# Maybe use env here?
ARG UID=37536
ARG USERNAME=svc_user
ARG GID=30002
ARG GROUPNAME=user_group
ARG VERSION


RUN groupadd -g $GID $GROUPNAME && useradd -u $UID -g $GID $USERNAME

RUN chown -R $UID:$GID /input /output
RUN chmod -R 775 /input /output

USER $UID

LABEL org.nrg.commands="[{\"name\": \"dcm2niix\", \"label\": \"dcm2niix\", \"description\": \"Runs dcm2niix\", \"info-url\": \"https://github.com/rordenlab/dcm2niix\", \"image\": \"registry.gitlab.com/my_company/dcm2niix:1.0.2\", \"version\": \"1.6\", \"schema-version\": \"1.0\", \"type\": \"docker\", \"command-line\": \"dcm2niix -ba n -f '#SCAN_LABEL#' -o /output /input\", \"mounts\": [{\"name\": \"dicom-in\", \"writable\": \"false\", \"path\": \"/input\"}, {\"name\": \"nifti-out\", \"writable\": \"true\", \"path\": \"/output\"}], \"inputs\": [{\"name\": \"bids\", \"description\": \"Create BIDS metadata file\", \"type\": \"boolean\", \"required\": \"false\", \"default-value\": \"false\", \"replacement-key\": \"[BIDS]\", \"command-line-flag\": \"-b\", \"true-value\": \"y\", \"false-value\": \"n\"}, {\"name\": \"scan_label\", \"description\": \"XNAT scan label\", \"type\": \"string\", \"required\": \"true\", \"replacement-key\": \"#SCAN_LABEL#\", \"user-settable\": \"true\"}], \"outputs\": [{\"name\": \"nifti\", \"description\": \"The nifti files\", \"mount\": \"nifti-out\", \"required\": \"true\"}], \"xnat\": [{\"name\": \"dcm2niix-scan\", \"description\": \"Run dcm2niix on a Scan\", \"label\": \"dcm2niix\", \"contexts\": [\"xnat:imageScanData\"], \"external-inputs\": [{\"name\": \"scan\", \"description\": \"Input scan\", \"type\": \"Scan\", \"required\": \"true\", \"matcher\": \"'DICOM' in @.resources[*].label\"}], \"derived-inputs\": [{\"name\": \"scan-dicoms\", \"description\": \"The dicom resource on the scan\", \"type\": \"Resource\", \"derived-from-wrapper-input\": \"scan\", \"provides-files-for-command-mount\": \"dicom-in\", \"matcher\": \"@.label == 'DICOM'\"}, {\"name\": \"scan_label\", \"description\": \"The label of the scan\", \"type\": \"string\", \"derived-from-wrapper-input\": \"scan\", \"derived-from-xnat-object-property\": \"label\", \"provides-value-for-command-input\": \"scan_label\"}], \"output-handlers\": [{\"name\": \"nifti-resource\", \"accepts-command-output\": \"nifti\", \"as-a-child-of\": \"scan\", \"type\": \"Resource\", \"label\": \"NIFTI\"}]}]}]"
```

The following is the terminal command I ran:

```bash
docker run -it 73ff24d0616e --mount type=bind,src=/mount_path,dst=/input
```

I was able to "see" the files in the `/DICOM`. By running the container with a bind mount to /mount_path, I confirmed that the files in /DICOM are visible, suggesting that the issue might be specific to how the Container Service is handling the mount.

======================================================


4. If the path doesn’t exist, why does CS think it needs to be mounted?

The path exists.

My findings suggest that the path exists and is accessible by the container under normal circumstances, yet the Container Service still attempts to create it. This leads me to suspect a path resolution or permission issue specific to how CS interacts with Docker.

Does this approach align with your debugging recommendations? Let me know if you see any gaps in my investigation.

Thanks again for your help!

Best,
Steve

[Steve]

Hello all,

Just following up. 

Thanks !

Steven 

[Kelsey, Matt]

Hey Steve,

I just wanted to highlight the mention of path-translation below, as that is normally the cause of this type of error for me. If you are running XNAT/Tomcat in a container, this is a setting that would normally need to be modified from the default (along with Processing URL).

 

If you have command line access on the system running Docker, you can inspect any completed containers and list the associated bind mounts, verifying access to them from the Docker server. Note that you might need to disable “Automatically cleanup containers” for Docker to retain the completed containers. (Under Plugin Settings -> Compute Backend -> Compute Backend Configuration -> Edit)

 

 

 

Once you have started your test container in XNAT, issue the command:
docker ps -a

on the command line. Note the ID of your container, then issue the command:

docker inspect container_id



In addition to, sometimes, helpful error messages, you will see any bind mounts associated with the container:

 

The source portion of the bind mount (/Users/Kelsey/Projects/XNAT/xnat-docker-compose/xnat-data/etc/etc above) needs to be accessible to the machine hosting your Docker engine.
e.g.

 

If this doesn’t get you unstuck, my next debugging step is to start the container with a ‘sleep 1000’ command and use the ‘docker exec' command to poke around internally for clues.  Happy to walk you through that process if needed.

 

Best,

Matt

 

 

Can anyone provide some debugging tips? Any help would be greatly appreciated.

 

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/f6bb96fb-aae2-4b55-a646-3b010852772cn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/e9793a0c-21ef-40f2-b2ae-43f5317e22fen%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[John Flavin]

	private void createDirectoriesForMounts(final Container toCreate) throws IOException {
        final List<Container.ContainerMount> containerMounts = toCreate.mounts() == null ? Collections.emptyList() : toCreate.mounts();
        for (final Container.ContainerMount mount : containerMounts) {
            final Path mountFile = Paths.get(mount.xnatHostPath());
            if (!Files.isRegularFile(mountFile)) {  // <------------------------------------ returns False
                Files.createDirectories(mountFile);
            }
        }
    }

QUESTION: Based on my understanding, this snippet is where the directory creation is attempted if Docker does not recognize the path. Could you confirm whether this interpretation is correct?

That's not quite correct. That bit of code you've identified is a fix to a previous bug we were having around a similar but slightly different docker mount problem. I think that bug should be fixed, but I'll describe it anyway just in case.

Let's say CS determines the container needs to mount a path in the archive but that path doesn't exist yet. If this code weren't there, CS would tell docker to mount the path, docker would not see anything there, so docker would create the path and mount it. But docker runs as root, so now the file path is owned by root, and that causes permission issues when XNAT tries to read or write that path later. This code was introduced as a way to work around that problem by having XNAT (via CS) create an empty path which could then be owned by whatever user is running the XNAT process.

It seems to me that isn't what is happening in your case. In that bug, docker had no problem seeing that path and creating a directory there. In your case docker does have a problem seeing the path.

I do think your investigation does make sense. You can copy out the path that CS is trying to mount into the container and manually create your own container that mounts that same path. But it is hard for me to tell if you've fully gotten all the details correct, partially because of the way you've had to obfuscate your mount paths. (Which is not say you shouldn't do that, please do.)

In what you ran here:

The following is the terminal command I ran:

docker run -it 73ff24d0616e --mount type=bind,src=/mount_path,dst=/input

You only mounted the root mount path. But that's not what CS was trying to mount, it had a very long specific path. What was that path? It's hard for me to tell from what you've shared. The error message from docker says

error while creating mount source path '/mount_path/project_name/arc001/session_name/SCANS/3/DICOM': mkdir /mount_path/archive: permission denied

That has one path that starts /mount_path/project_name but the specific error message is about /mount_path/archive. I wonder if that inconsistency is important to figuring out the cause of the issue. Maybe you need a path translation that adds or removes the /archive level of the path?

John Flavin

To view this discussion visit https://groups.google.com/d/msgid/xnat_discussion/SA1PR02MB8656F90637869FB0ACBF9183A9C32%40SA1PR02MB8656.namprd02.prod.outlook.com.