Running docker inside sandbox

[Paul Gross]

We have a few genrules and skylark command that rely on docker. We've done this to package up some tools we're using. For example:

    genrule(

      name = "convert",

      outs = ["output"],

      cmd = "docker run my_image:tag > \"$@\"",

    )

With the new sandboxing stuff turned on, these now fail. It looks like we can't use docker inside the sandbox. We get errors like:

    dial unix /var/run/docker.sock: no such file or directory. Are you trying to connect to a TLS-enabled daemon without TLS?" 

Is there a way to get sandboxing to support docker? Or is there a different workaround?

Thanks,

Paul

[Philipp Wollermann]

Hi Paul,

the sandbox currently does not mount /var into the sandbox, so the docker tool running inside it can't connect to the daemon.

As a quick fix, while I discuss with the team what the right approach here is, you could use one of the following:

1) Patch the code so that the sandbox mounts /var/run - just add it to the "mountUsualUnixDirs" method in LinuxSandboxedStrategy.java.

2) Use TCP/IP to connect to Docker: "docker -H localhost:2375 run my_image:tag ...", because network access is (still) allowed from inside the sandbox. You'd have to configure Docker to listen on localhost:2375 then.

Hope that helps,

Philipp

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAMXKpGCOOhFpxH%2BeUK--x%3DnuT_z9wvcthpPhj7N-0%3DfwD2uUAA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Google Germany GmbH
Dienerstraße 12
80331 München

Geschäftsführer: Graham Law, Christine Elizabeth Flores
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Paul Gross]

For now, we just turned off sandboxing using: build --spawn_strategy=standalone --genrule_strategy=standalone --ignore_unsupported_sandboxing

I'm curious to hear what you folks decide is the right approach.

Thanks,

Paul

[Damien Martin-guillerez]

We should add a tag ("non-hermetic"?) to genrule to be able to deactivate sandboxing for specific genrule. It totally not recommended though because docker run is totally non-hermetic and that can break in many ways.

We have the remote repository mechanism that is used for non-hermetic stuff usually but I believe you need some artifact from your build to run docker run? I am interested by what you are exactly trying to do (can this be run out of docker run?).

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAMXKpGCYadFo2BOkOtj-3T3ATxkD0%2BYmjTADnqgbstU9Y%3Dp%3DSQ%40mail.gmail.com.

[Paul Gross]

Our specific case is a little complicated. We use swagger (http://swagger.io/) yaml files for API specification, but we have tools that need all of the yaml files flattened into a single swagger.json. We wrote node code to do this (as the swagger node libraries seem to have the most features). Rather than require everyone to have node, npm, etc set up on their machine at just the right versions, we packaged it all up into a docker image. Then, we have a genrule that just runs that image with the yaml files as input and swagger.json as output.

We certainly could run the node conversion tool on the host machine. In fact, we were doing it this way before. But it's really nice to package up the whole node environment into an image and ensure everyone is running the same version of everything.

Thanks,

Paul

[Damien Martin-guillerez]

That's an interesting use case, we talked several time about that but we didn't knew it was actually used somewhere. In Google we check in all tool dependency in the depot to make sure everyone run the same version.

I see four ways to fix that:

  - Modify the sandbox-namespace program and provide your own to call Docker. That will leads a lot of volume mounting. Also I don't know how easy it is to specify your own sandbox wrapper.

  - Use the host tools

  - Use checked-in tools

  - We add a "non-deterministic" tag to genrule so people can escape the sandbox.

[Kamal Marhubi]

As an extension of "checked-in tools", could you supply a known-good version of all the components stored on S3 or similar, and bring them in via an http_repository rule? This would introduce network connectivity as an initial build requirement, though.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS_%3Do6CmCSO%2Bugr3xw%3DpM0vc21THX4hbb4TMy1k8zAjU6w%40mail.gmail.com.

[Paul Gross]

We could do this, but it's a little involved. We'd have to bundle node, npm, and the necessary packages and their dependencies. I think we'd be fine with a non-deterministic tag for now, and maybe we can look into vendoring all of the tools in the future.

Thanks,

Paul

[Ulf Adams]

You can already mark genrules and actions as 'local', and that should disable sandboxing. I'm not sure how to do that in Skylark off the top of my head.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAMXKpGDrEM%3DQy04euKmbs_GFKtzJAhBQS7FrCj6P8obMYHBdgA%40mail.gmail.com.

[Paul Gross]

Ulf,

The comment "// TODO(bazel-team): remove?" doesn't inspire confidence. :)

https://github.com/bazelbuild/bazel/blob/4603a78b748d0cb74239a3bac6a52600bb4339f4/src/main/java/com/google/devtools/build/lib/bazel/rules/genrule/BazelGenRuleRule.java#L154-L155

Is this still a recommended thing to do?

Thanks,

Paul

[Damien Martin-guillerez]

I think that comment should be more wether we should rename it as sandboxed execution is "local" but we treat it in our code as remote execution.

Well maybe we should stick with that "local" attribute and remove the TODO

[Ulf Adams]

Yeah, let's remove the comment. We don't have remote execution yet, but it seems as good a name as any.

[koe...@gmail.com]

> 1) Patch the code so that the sandbox mounts /var/run - just add it to the "mountUsualUnixDirs" method in LinuxSandboxedStrategy.java.

I tried this option:

diff --git a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedStrategy.java b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSa
index f4ccbdd..b19fec2 100644
--- a/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedStrategy.java
+++ b/src/main/java/com/google/devtools/build/lib/sandbox/LinuxSandboxedStrategy.java
@@ -315,6 +315,7 @@ public class LinuxSandboxedStrategy implements SpawnActionContext {
FileSystem fs = blazeDirs.getFileSystem();
mounts.put(fs.getPath("/bin"), fs.getPath("/bin"));
mounts.put(fs.getPath("/etc"), fs.getPath("/etc"));
+ mounts.put(fs.getPath("/var/run"), fs.getPath("/var/run"));
for (String entry : FilesystemUtils.readdir("/")) {
if (entry.startsWith("lib")) {
Path libDir = fs.getRootDirectory().getRelative(entry);


It works!

I use this to test docker instances generated by the docker_build skylark rule. Is there a better way to run tests on docker instances I'm not aware of?

Thanks,

Koen