Bazel test and sandbox question: How to share files between bazel test sandbox and docker container(running in root) with proper permissions and cleanup?

[Nian-Hsuan Tsai]

I am launching docker container within bazel tests to test services running in the docker container. And in my scenario, I need a shared storage location to which both my test process and the docker service under test can access. Our initial attempt was to create a shared directory in the bazel sandbox under $TEST_TMPDIR within the test process and bind mount the shared directory into the docker services. However, since the docker services run with root permission by default, we have two problems:

1. The docker service writes files with root permission to the shared directory. The test process does not have permission to modify these files.
2. The bazel sandbox cleanup fails to clean up the root files left behind. The cleanup problem is similar to this github issue.

Are there any general solutions to share files between the bazel test and the docker containers while avoiding the permission and cleanup issues?

Some assumptions here:

• I have no control over the docker files.
• The docker services run with root, and the test host process runs with an unprivileged user.

TL;DR

• We have a bazel test that launches a docker container.
• The service running in the docker container needs to write files at a location that the test process can later access.
• We mount a location, which is in the sandbox, when launching the container to share the directory in the two environments.
• The service writes the file but runs with root. The test process cannot access these files, and bazel is unable to clean the sandbox with root files left behind.

[Alexandre Rostovtsev]

Suggestions which I have not personally tried, but which I believe may work:

* use a docker bind mount pointing to a directory in an idmapped mount on the host (assuming the host is Linux with kernel 5.12 or newer, and you have root). See https://github.com/brauner/mount-idmapped

* use a docker samba or nfs volume, with the nfs / samba server having user mapping enabled

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c793d658-6d03-435e-8d84-68e66d895afan%40googlegroups.com.

[Nian-Hsuan Tsai]

Thanks for the response!

1. idmapped mounts: Unfortunately, we are running Linux kernel 5.4 which does not have support for mount_setattr().

2. nfs: To my understanding, if we go with this direction, we cannot create the directory within the sandbox, since I cannot run `sudo mount` within the bazel sandbox. We will have to create a directory on host first and mount the nfs to that directory, then make this directory available to the bazel sandbox with `--sandbox_writable_path`. Then I mount this directory to my docker containers. Is my understanding correct?

[David Turner]

Can you try running inside a rootless Docker container? I believe this should ensure the files are written with your UID on the real filesystem.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAHmzvzJC6MGB_%2B9F4-5tg-aEkXV_GosCfyNc2SUX0-4HrR8tGQ%40mail.gmail.com.

[Alexandre Rostovtsev]

Alternatively, is it possible to run `docker volume` inside your test, or does that trigger the sandbox? If it is possible, then inside your test you could launch a local nfs server and create a docker volume with type=nfs pointing at it.

[Nian-Hsuan Tsai]

Thanks for the response, Alexandre and David!

> Can you try running inside a rootless Docker container? I believe this should ensure the files are written with your UID on the real filesystem.

We did think of changing the running user of the docker container but unfortunately after some evaluation, we can't change it and need to run the containers as default, which is with root permission.

> Alternatively, is it possible to run `docker volume` inside your test, or does that trigger the sandbox? If it is possible, then inside your test you could launch a local nfs server and create a docker volume with type=nfs pointing at it.

Yes, we did try creating a docker nfs volume within the test and mounting the docker services to the docker volume to get access, however, the problem here is that the test process can't access the docker volumes. We also can't mount the nfs to a directory to access within the test because we can't run `sudo mount` in the bazel sandbox.

[Filip Filmar]

On Thu, Apr 20, 2023 at 10:22 AM 'Nian-Hsuan Tsai' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Are there any general solutions to share files between the bazel test and the docker containers while avoiding the permission and cleanup issues?

Yes. In general, use the `-u` option when invoking `docker run` to pass down the UID and GID of the user that the binaries inside the container run as. This will have them run as that UID:GID pair.

Mount the volumes you need into the same paths as the host, with appropriate `rw` permissions.

I made this approach into a helper that allows you to run a build step within a docker container entirely. See https://github.com/filmil/bazel-rules-bid/blob/main/build/docker_run.sh, and https://github.com/filmil/bazel-ebook as an example of its use.

F

[Nian-Hsuan Tsai]

Thanks for the response Filip!

We did consider changing the docker container uid to avoid the permission problems. However, we are trying to build a testing framework for test authors to test their docker services. Thus we do not have control over (1) the docker files and also (2) there are too many unknowns and risks involved in changing the docker container uid, as there might be services that rely on root permissions to work.

[Filip Filmar]

On Thu, Apr 20, 2023 at 8:45 PM Nian-Hsuan Tsai <nianhsu...@databricks.com> wrote:

We did consider changing the docker container uid to avoid the permission problems. However, we are trying to build a testing framework for test authors to test their docker services. Thus we do not have control over (1) the docker files and also (2) there are too many unknowns and risks involved in changing the docker container uid, as there might be services that rely on root permissions to work.

Hm, in that case perhaps bazel isn't a really good fit for your test fixture? I'm no bazel expert so take this with a grain of salt, but at the point where you actively need to juggle UIDs, you may be doing yourself a disservice by relying on bazel specifically.

Another way that comes to mind is to run a script at the very end of your container run that will (inside the container) revert the file permissions to `uid:gid`, thus allowing bazel to remove them once the container is down.  While you don't own the existing container (IIRC), you should be able to make a new container using your container under test as base.

F

[Nian-Hsuan Tsai]

Thanks again for the response!

We can probably do that for cleanup, however, the test framework still need to access the root written files during the test, which this method doesn't solve. We're also considering deploying an additional docker container that also mounts to the shared directory and runs in root to watch the shared directory and run chmod to allow access whenever there is a change. However, this seems more like a workaround than a good solution.