extend sandbox environment to have network namespace

[Ming Zhao]

Hi,

I'd like to discuss the possibility of making the sandbox more complicated, :)

Is it possible to enable network namespace support(optional) so that
tests run inside sandbox won't affect other tests. For certain legacy
reason, some our tests have to use static TCP/UDP port, some of them
has to request privileged network operation like creating a new
bridge. These operations could be done much easily if the sandbox
creates a new network namespace. The trade off is the test inside the
sandbox can't make any external network communication. But that's kind
of what a reasonable test should behave like.

What do you guys think? I have done some quick experiment and it seems
to work. But I want to confirm with you guys before moving further.

Best,
Ming

[Austin Schuh]

Hi Ming,

Before sandboxing was enabled, we started running our tests in a namespace by creating one in tools/test/test-setup.sh.  This let us do exactly what you are talking about for some third-party tests we wanted to run.  It would be awesome if bazel could support that natively without having to disable the sandbox and use our own for those tests.

Austin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN0GiO04gxm7CcBKAj7aB%2BgdPALW1wsNm9XfCSaimE7SRiD78Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Damien Martin-guillerez]

+Philipp Wollermann

It's actually something we want to include in the evolution of the sandboxing mechanism. I don't know when we would be able to work on it though (Philip, any idea?)

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CABsbf%3DH3AP-ntVTXMsTx3qJOUnA_YpYp4W_CyO%2Bs9duCvzp6oA%40mail.gmail.com.

[Philipp Wollermann]

Hi Ming,

On Thu, Sep 10, 2015 at 2:19 AM, Ming Zhao <ming...@gmail.com> wrote:

I'd like to discuss the possibility of making the sandbox more complicated, :)

That's always welcome ;)

Is it possible to enable network namespace support(optional) so that
tests run inside sandbox won't affect other tests. For certain legacy
reason, some our tests have to use static TCP/UDP port, some of them
has to request privileged network operation like creating a new
bridge. These operations could be done much easily if the sandbox
creates a new network namespace. The trade off is the test inside the
sandbox can't make any external network communication. But that's kind
of what a reasonable test should behave like.

What do you guys think? I have done some quick experiment and it seems
to work. But I want to confirm with you guys before moving further.

I think this is a great idea and we should do it. However there might be some tests or build actions that deliberately need network access (for example, a genrule downloading something from the internet or an integration test running against a test server). So we have to solve the following issues:

- Should network access be enabled or disabled by default?

- Can we take away network access from normal build steps (compiling C / Java source etc.) as well, or should we only do this for tests?

- What would be a suitable way to toggle it for actions that need network access? I think we'd need an attribute on genrule and test_* rules and a way to do it in Skylark, too?

FYI, at Google we solve this for tests by specifying a "test size" of either small, medium or large and then have varying levels of restrictions for them: http://googletesting.blogspot.de/2010/12/test-sizes.html

What do you think, how would you like the feature to look like?

Input is very welcome. :)

Cheers,

Philipp

--

Google Germany GmbH
Dienerstraße 12
80331 München

Geschäftsführer: Graham Law, Christine Elizabeth Flores
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

[Erik Kuefler]

FYI, at Google we solve this for tests by specifying a "test size" of either small, medium or large and then have varying levels of restrictions for them: http://googletesting.blogspot.de/2010/12/test-sizes.html

I for one would be quite happy if Bazel automatically killed network access for small and medium tests - I know Bazel supports a "size" attribute already that affects things like timeout, so this seems like a natural extension.

[Ming Zhao]

As a follow up, I just sent out a prelimilary CL for comment:

https://bazel-review.googlesource.com/2101

Introduce two new options to Linux sandbox wrapper:
* -n: Create a new network namespace with only loopback interface.
* -r: set the uid/gid inside the sandbox to be root(instead of nobody)
so that setuid program like ping can still run when needed.

I don't have an good idea how to turn it on yet. Maybe we can make
"-n" be used by default, and introduce certain test attribute to turn
it off.

Please feel free to chime in. Thank you!

[Han-Wen Nienhuys]

Since we default to sandboxed execution wrt files, let's also make the
default cut off network access, with some option,

tags = [ "network-access" ]

to switch it on again.

Setting the UID to root is a little tricky: we have seen a bunch of
code that uses (getuid() == 0) and then assumes it has certain
capabilities.

Philipp, thoughts?

> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN0GiO2udFD-o4HvHCABeyFiCnGDiX-jDu8z5Yj%2BfEyorFqYzA%40mail.gmail.com.

> For more options, visit https://groups.google.com/d/optout.



--

Han-Wen Nienhuys
Google Munich
han...@google.com

[Ming Zhao]

On Tue, Oct 6, 2015 at 5:16 AM, Han-Wen Nienhuys <han...@google.com> wrote:
> Since we default to sandboxed execution wrt files, let's also make the
> default cut off network access, with some option,
>
> tags = [ "network-access" ]
>
> to switch it on again.
>
> Setting the UID to root is a little tricky: we have seen a bunch of
> code that uses (getuid() == 0) and then assumes it has certain
> capabilities.

According to Linux manual
page(http://man7.org/linux/man-pages/man7/user_namespaces.7.html) and
my test, creating a new namespace will give the user full
capabilities, but "Having a capability inside a user namespace permits
a process to perform operations (that require privilege) *only on
resources governed by that namespace*".

So without creating network namespace, the user won't be able to
perform privileged network operation, after being dropped in a new
network namespace, the user should be able perform privileged network
operation, like bringing up interface(which is done in the code too),
creating new virtual interface, capturing packet, etc. Since the check
is based on capability list, the UID/GID shouldn't really matter this
case, but I couldn't explain why ping program doesn't work as nobody,
unless setuid-ed program has special check based on whether it's
nobody.

[Brian Silverman]

Another example of something which doesn't work as nobody: building tarballs in which root owns the files. I was using fakeroot to do this before the sandbox, but fakeroot doesn't work inside Bazel's sandbox. I'm currently using a nested user namespace which maps 0-65535 out to nobody in Bazel's namespace, but I wouldn't mind dropping that. Maybe leave the default like it is now and have a tag to switch to running the program inside the sandbox as root?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN0GiO3Yfsn38XMnP9c%3D5BOVueu%2BkgeqX8Dg-_08YpWBkAk7Lg%40mail.gmail.com.

[Han-Wen Nienhuys]

For tar, just use --owner and --group.