Running gVisor inside a docker container

[Lars Andringa]

Hi all,

Our team is developing a system that automatically tests code submissions for university assignments. We use gVisor + Docker to virtualise and protect those workloads.

We want to have automated integration tests. Our system however has multiple complex infrastructure requirements, and these integration tests must be able to be run locally, preferably with little setup, so we run a Docker-in-Docker setup.

The docker daemon running inside the top-level Docker container must therefore have gVisor installed. I attempted this, but ran into the following error:

message":"failed to create task for container: failed to create shim task: OCI runtime create failed: creating container: cannot create gofer process: gofer: fork/exec /proc/self/exe: resource temporarily unavailable: check whether /proc/sys/user/max_user_namespaces is set too low (gvisor.dev/issue/5964): unknown

I presume this has something to do with what was mentioned at https://github.com/google/gvisor/issues/4371. So rather than a low max_user_namespaces, it's likely to be more of an inside-container issue.

Is there any way to get gVisor to work on a Docker daemon running within a Docker container? The container is already running with --privileged, and security-wise we don't mind weakening things to make it work. It's just a local test environment.

Kind regards,
Lars

[Etienne Perot]

Hi Lars,

As you've noticed, this isn't a well-supported use-case.

You've mentioned being willing to weaken the security of this to make it work, so in your case I'd consider installing gVisor on the top-level Docker, and then mounting the top-level Docker daemon socket into the container where you'd currently be running the second-level Docker daemon (docker run ... -v /var/run/docker.sock:/var/run/docker.sock). So running the docker CLI within that container will actually talk to the top-level Docker daemon, so running `docker run --runtime=runsc ...` will run a top-level Docker container with gVisor. Some things will work counter-intuitively (e.g. volume paths passed to docker's -v flag will refer to path on the host filesystem, not path from the container you're running the CLI in; same deal with port forwarding), but depending on your setup this may be manageable.

If you've been looking at the gVisor commit history in the past few months, you may have noticed that there's been a bunch of activity around supporting Docker to be able to run it inside gVisor. I realize this is the literal inverse of what you're asking for, so it doesn't help you if you still have to run gVisor inside a Docker container). But if the daemon socket solution isn't an option for you, perhaps you could re-architect the assignment testing system such that gVisor is present at the top level, and then you'd have Docker-in-Docker running within gVisor. This would have a weaker degree of isolation between the second-level containers than if they were each running in their own gVisor container, but it would still provide gVisor's protection against compromising the host.

Hope this helps,

- Etienne

--
You received this message because you are subscribed to the Google Groups "gVisor Users [Public]" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gvisor-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/0b7f607a-8501-447e-a188-e09e99d36e7an%40googlegroups.com.

[Lars Andringa]

Dear Etienne,

Thank you for the response, I appreciate it.

We have considered using a top-level Docker daemon,  however the main problem that we're trying to solve with the docker container in integration testing is that our docker daemon has a lot of customisation requirements to make the system work. This currently in early development include:
- Having gVisor/runsc available

- The storage driver being backed by an XFS filesystem using prjquotas

- A custom daemon.json configuration

- Multiple services being present to run various services such as zombie process analysis. These services must run on the level that the daemon is running

As you can imagine, this is a lot to set up for someone who wants to run integration tests locally, especially considering that I for example am running Windows.

So the point was to be able to press a button in our Gradle configuration, and tada, you have your test results. If we move the daemon to the top-level, then all these requirements need to be done on the host, which defeats the purpose.

Interestingly, running "docker run --runtime=runsc {image}" Inside our DIND container actually seemed to work perfectly fine. It's only when our program is trying to run a gVisor container, is when it breaks.
I've included an image of our integration testing setup. The inner DIND daemon is passed to the companion container (which is the system we're testing) using the  -v /var/run/docker.sock:/var/run/docker.sock

Our docker-compose looks as follows:

services:
  companion:
    image: companion-container
    ports:
      - 9000:9000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    user: root
    privileged: true
  tester:
    image: integration-tester

[Etienne Perot]

> Interestingly, running "docker run --runtime=runsc {image}" Inside our DIND container actually seemed to work perfectly fine. It's only when our program is trying to run a gVisor container, is when it breaks.

I'm confused by this statement.

What exactly is the difference between running `docker run --runtime=runsc ...` from within the DIND container, vs "our program trying to run a gVisor container"?

Does your program run in the companion container, and talks to the second-level Docker daemon through the socket? And you're saying it doesn't work if it tries to create a container that way, but that if you run the Docker CLI in the docker dind container directly, it runs fine?

I created a diagram to show my understanding, please correct it if it is wrong.

If the diagram is not wrong, then that's surprising, because that would mean that gVisor is working fine inside a Docker container and that the problem lies elsewhere.

To view this discussion on the web visit https://groups.google.com/d/msgid/gvisor-users/eab85c8b-203a-476d-8801-6ed89b32707cn%40googlegroups.com.

[Lars Andringa]

That's a correct understanding yes.

My assumption was that the difference may lie in the fact that the companion ran inside a container, so there is something that happened here that did not happen with the CLI.

However I am now running it in a much simpler configuration, where I'm just running the companion unvirtualised on the host, without DIND or anything, and I'm getting similar issues (resource temporarily unavailable: check whether /proc/sys/user/max_user_namespaces is set too low (gvisor.dev/issue/5964)).
So I'm starting to suspect it's something in my program, rather than this setup. I'll have to debug it, but yeah our issues seem to be unrelated to the setup, and gVisor just fails in any situation, so there's something wrong with my code.

I'll update if I find out what it is, but yeah my apologies, it seems to be unrelated.

[Lars Andringa]

So the reason that it was broken was because we limit the amount of processes that can be made within a container, and gVisor exceeded that.