namespace-sandbox segmentation faulted with vanilla build from source + cpp hello world

121 views
Skip to first unread message

denes....@gmail.com

unread,
Apr 17, 2016, 8:53:18 AM4/17/16
to bazel-discuss
Dear Bazel folks,

tl;dr I made namespace-sandbox.c:SetupDirectories() segfault from my vanilla bazel build. Lot of details follow:


I have a 64-bit Arch Linux box running kernel 4.5.0 and gcc 5.3.

I've decided to go my own way and compile Bazel for myself to completely manage my toolchain and not rely on what the distro offers. (Other people I may collaborate with don't use Arch.)

So I cloned the Git repo:
$ git clone https://github.com/bazelbuild/bazel.git
and ran
$ ./compile.sh.

The build went fine except for a bunch of warnings that were talking about _FORTIFY_SOURCE and I think a missing -O switch. :/ Then I created my WORKSPACE file and the following workspace contents:

===========================

/WORKSPACE
/examples
/examples/cpp
/examples/cpp/BUILD
/examples/cpp/hello.cc


hello.cc:
#include <iostream>

using namespace std;

int main() {
return 0;
}


BUILD:
cc_binary(
name = "helloworld",
srcs = ["hello.cc"],
)
===========================

Then attempted to execute from the workspace root:
===========================

$ bazel build examples/cpp:helloworld

INFO: Found 1 target...
ERROR: /home/denes/r2/examples/cpp/BUILD:1:1: C++ compilation of rule '//examples/cpp:helloworld' failed: namespace-sandbox failed: error executing command /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox ... (remaining 37 argument(s) skipped).
Target //examples/cpp:helloworld failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.209s, Critical Path: 0.07s
===========================

So I added --debug_sandbox and --verbose_failures:
===========================

[denes@rudanium r2 [master]]$ bazel build --verbose_failures --sandbox_debug examples/cpp:helloworldINFO: Found 1 target...
ERROR: /home/denes/r2/examples/cpp/BUILD:1:1: C++ compilation of rule '//examples/cpp:helloworld' failed:

Sandboxed execution failed, which may be legitimate (e.g. a compiler error), or due to missing dependencies. To enter the sandbox environment for easier debugging, run the following command in parentheses. On command failure, a bash shell running inside the sandbox will then automatically be spawned:

namespace-sandbox failed: error executing command
(cd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2 && \
exec env - \
PATH=/home/denes/bin:/home/denes/grub-2.00/bin:/home/denes/cross-compiler/build/cross/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/lib/mailman/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/denes/.r2bin:/scripts:/usr/share/java/apache-ant/bin/:/rudanium/bin:/home/denes/bin \
/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o' -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o).
src/main/tools/namespace-sandbox.c:761: sandbox root is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0
src/main/tools/namespace-sandbox.c:763: working dir is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2
src/main/tools/namespace-sandbox.c:392: unshare succeeded after 1 tries
Target //examples/cpp:helloworld failed to build
INFO: Elapsed time: 0.272s, Critical Path: 0.07s


===========================

Ok, so I thought't let's have a look at the sandbox:
===========================

[denes@rudanium r2 [master]]$ (cd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2 && \
> exec env - \
> PATH=/home/denes/bin:/home/denes/grub-2.00/bin:/home/denes/cross-compiler/build/cross/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/lib/mailman/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/denes/.r2bin:/scripts:/usr/share/java/apache-ant/bin/:/rudanium/bin:/home/denes/bin \
> /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o' -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o)
src/main/tools/namespace-sandbox.c:761: sandbox root is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0
src/main/tools/namespace-sandbox.c:763: working dir is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2
src/main/tools/namespace-sandbox.c:392: unshare succeeded after 1 tries
Segmentation fault (core dumped)

===========================
As you can see, namespace-sandbox segfaulted.

It turns out the fault comes from SetupDirectories. I added `gdb --args' to the command above to run namespace-sandbox in the debugger. Here's the output:
===========================

[denes@rudanium r2 [master]]$ (cd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2 && exec env - PATH=/home/denes/bin:/home/denes/grub-2.00/bin:/home/denes/cross-compiler/build/cross/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/lib/mailman/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/denes/.r2bin:/scripts:/usr/share/java/apache-ant/bin/:/rudanium/bin:/home/denes/bin gdb --args /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o' -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o)
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o
src/main/tools/namespace-sandbox.c:761: sandbox root is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/2636df84-c3ea-40a5-8512-ab6df5779aa4-0
src/main/tools/namespace-sandbox.c:763: working dir is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2
src/main/tools/namespace-sandbox.c:392: unshare succeeded after 1 tries
Error while mapping shared library sections:
Could not open `target:/usr/lib/libnss_files.so.2' as an executable file: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
0x00000000004034d0 in SetupDirectories ()
(gdb) where
#0 0x00000000004034d0 in SetupDirectories ()
#1 0x00000000004044e1 in main ()
(gdb)


===========================
I appreciate any help you can offer in debuging this!

It would be great for instance if I could create a debug build of Bazel and see which one was the offending line. However I was unable to figure out the standard way of creating such build of Bazel.

Thanks,
Dénes

denes....@gmail.com

unread,
Apr 17, 2016, 8:54:23 AM4/17/16
to bazel-discuss, denes....@gmail.com
1 more thing:

Using bazel release 0.2.1 (2016-03-21)

Brian Silverman

unread,
Apr 17, 2016, 9:13:36 AM4/17/16
to denes....@gmail.com, bazel-discuss
`EXTRA_BAZEL_ARGS=--compilation_mode=dbg ./compile.sh compile` should get you debugging symbols. Getting a line number for the segfault with that would be nice.

Also, that "Could not open `target:/usr/lib/libnss_files.so.2' as an executable file: Operation not permitted" message from GDB looks suspicious. SetupDirectories calls getpwuid which I think interacts with libnss_files.so.2 (I really don't understand how NSS works, other than that getpwuid showing up in `strings libnss_files.so.2` (on my Debian system at least) means something's going on). Does that maybe sound like something weird with your system?

Further debugging ideas: What do `ldd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox` and `ls -la /usr/lib/libnss_files.so.2 $(readlink -f /usr/lib/libnss_files.so.2)` print? Does a really simple test program that just calls getpwuid (or the one on the manpage that calls getpwnam_r) work?

On Sun, Apr 17, 2016 at 8:54 AM, <denes....@gmail.com> wrote:
1 more thing:

Using bazel release 0.2.1 (2016-03-21)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9b9ee47e-093b-48a7-bb2f-da9857d7ca82%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ALMÁSI Dénes

unread,
Apr 17, 2016, 10:08:02 AM4/17/16
to Brian Silverman, bazel-discuss
Hi Brian,

Thanks for your help! I recompiled bazel in debug and got the following when firing it up in gdb with the aformentioned setup:

[denes@rudanium r2 [master]]$ (cd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2 &&   exec env -     PATH=/home/denes/bin:/home/denes/grub-2.00/bin:/home/denes/cross-compiler/build/cross/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/lib/mailman/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/denes/.r2bin:/scripts:/usr/share/java/apache-ant/bin/:/rudanium/bin:/home/denes/bin gdb --args  /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/693c30ba-a16d-4529-8fbe-243a18667dab-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o' -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o)
GNU gdb (GDB) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
Find the GDB manual and other documentation resources online at:
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox...done.
(gdb) run

Starting program: /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/693c30ba-a16d-4529-8fbe-243a18667dab-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++0x -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o
src/main/tools/namespace-sandbox.c:761: sandbox root is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/693c30ba-a16d-4529-8fbe-243a18667dab-0
src/main/tools/namespace-sandbox.c:763: working dir is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2
src/main/tools/namespace-sandbox.c:392: unshare succeeded after 1 tries
Error while mapping shared library sections:
Could not open `target:/usr/lib/libnss_files.so.2' as an executable file: Operation not permitted

Program received signal SIGSEGV, Segmentation fault.
0x00000000004034d0 in SetupDirectories (opt=0x7fffffffe590) at src/main/tools/namespace-sandbox.c:533
533  char *homedir = getpwuid(getuid())->pw_dir;
(gdb) 

Indeed, that's related to libnss. However I was able to compile and run the test program which is on the man page of getpwnam_r. It just worked.

Here are the libnss_files libraries under /usr/lib:
[denes@rudanium ~]$ ls -l /usr/lib/libnss_files*
-rwxr-xr-x 1 root root 47584 Feb 20 00:48 /usr/lib/libnss_files-2.23.so
lrwxrwxrwx 1 root root    17 Feb 20 00:48 /usr/lib/libnss_files.so -> libnss_files.so.2
lrwxrwxrwx 1 root root    20 Feb 20 00:48 /usr/lib/libnss_files.so.2 -> libnss_files-2.23.so

As the error says operation not permitted, I thought it may be some permission issue but at least the unix permissions are fine here. Also, I don't remember anything special about this lib or the glibc package that provides it in my Arch installation.

Output of ldd on namespace-sandbox:
[denes@rudanium r2 [master]]$ ldd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox 
linux-vdso.so.1 (0x00007ffce8e8a000)
libm.so.6 => /usr/lib/libm.so.6 (0x00007fc46b162000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fc46ade0000)
libc.so.6 => /usr/lib/libc.so.6 (0x00007fc46aa3f000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc46b467000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fc46a829000)

I'm confused and will keep thinking about this now...

ALMÁSI Dénes

unread,
Apr 17, 2016, 10:32:08 AM4/17/16
to Brian Silverman, bazel-discuss
Executing namespace-sandbox from above with LD_DEBUG=all yields the following lines relevant to libnss_files immediately before segfaulting:

     15639: file=libnss_files.so.2 [0];  dynamically loaded by /usr/lib/libc.so.6 [0]
     15639: find library=libnss_files.so.2 [0]; searching
     15639: search cache=/etc/ld.so.cache
     15639:  trying file=/usr/lib/libnss_files.so.2
     15639:
     15639: file=libnss_files.so.2 [0];  generating link map
     15639:  dynamic: 0x00007f683f769da0  base: 0x00007f683f55f000   size: 0x0000000000211718
     15639:    entry: 0x00007f683f5611a0  phdr: 0x00007f683f55f040  phnum:                  7
     15639:
     15639: checking for version `GLIBC_2.3' in file /usr/lib/libc.so.6 [0] required by file /usr/lib/libnss_files.so.2 [0]
     15639: checking for version `GLIBC_2.14' in file /usr/lib/libc.so.6 [0] required by file /usr/lib/libnss_files.so.2 [0]
     15639: checking for version `GLIBC_PRIVATE' in file /usr/lib/libc.so.6 [0] required by file /usr/lib/libnss_files.so.2 [0]
     15639: checking for version `GLIBC_2.2.5' in file /usr/lib/libc.so.6 [0] required by file /usr/lib/libnss_files.so.2 [0]
     15639: object=/usr/lib/libnss_files.so.2 [0]
     15639: scope 0: /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox /usr/lib/libm.so.6 /usr/lib/libstdc++.so.6 /usr/lib/libc.so.6 /lib64/ld-linux-x86-64.so.2 /usr/lib/libgcc_s.so.1
     15639: scope 1: /usr/lib/libnss_files.so.2 /usr/lib/libc.so.6 /lib64/ld-linux-x86-64.so.2
     15639:
     15639:
     15639: relocation processing: /usr/lib/libnss_files.so.2 (lazy)
     15639: symbol=_res_hconf;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=_res_hconf;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=_res_hconf;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=_res_hconf;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `_res_hconf' [GLIBC_2.2.5]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_ITM_deregisterTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=errno;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=errno;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=errno;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=errno;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `errno' [GLIBC_PRIVATE]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__pthread_mutex_unlock;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__resp;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__resp;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__resp;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__resp;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `__resp' [GLIBC_PRIVATE]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__pthread_mutex_lock;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__gmon_start__;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=__gmon_start__;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=__gmon_start__;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_Jv_RegisterClasses;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libgcc_s.so.1 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: symbol=_ITM_registerTMCloneTable;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
     15639: symbol=__cxa_finalize;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__cxa_finalize;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__cxa_finalize;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__cxa_finalize;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `__cxa_finalize' [GLIBC_2.2.5]
     15639: object=/usr/lib/libnss_files.so.2 [0]
     15639: scope 0: /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox /usr/lib/libm.so.6 /usr/lib/libstdc++.so.6 /usr/lib/libc.so.6 /lib64/ld-linux-x86-64.so.2 /usr/lib/libgcc_s.so.1
     15639: scope 1: /usr/lib/libnss_files.so.2 /usr/lib/libc.so.6 /lib64/ld-linux-x86-64.so.2
     15639:
     15639: object=/usr/lib/libc.so.6 [0]
     15639: scope 0: /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox /usr/lib/libm.so.6 /usr/lib/libstdc++.so.6 /usr/lib/libc.so.6 /lib64/ld-linux-x86-64.so.2 /usr/lib/libgcc_s.so.1
     15639:
     15639: object=/lib64/ld-linux-x86-64.so.2 [0]
     15639: no scope
     15639:
     15639:
     15639: calling init: /usr/lib/libnss_files.so.2
     15639:
     15639: opening file=/usr/lib/libnss_files.so.2 [0]; direct_opencount=1
     15639:
     15639: symbol=_nss_files_getpwuid_r;  lookup in file=/usr/lib/libnss_files.so.2 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libnss_files.so.2 [0]: normal symbol `_nss_files_getpwuid_r'
     15639: symbol=fopen;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=fopen;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=fopen;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=fopen;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `fopen' [GLIBC_2.2.5]
     15639: symbol=fgets_unlocked;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=fgets_unlocked;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=fgets_unlocked;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=fgets_unlocked;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `fgets_unlocked' [GLIBC_2.2.5]
     15639: symbol=__ctype_b_loc;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=__ctype_b_loc;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=__ctype_b_loc;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=__ctype_b_loc;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `__ctype_b_loc' [GLIBC_2.3]
     15639: symbol=_nss_files_parse_pwent;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=_nss_files_parse_pwent;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=_nss_files_parse_pwent;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=_nss_files_parse_pwent;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `_nss_files_parse_pwent' [GLIBC_PRIVATE]
     15639: symbol=fclose;  lookup in file=/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox [0]
     15639: symbol=fclose;  lookup in file=/usr/lib/libm.so.6 [0]
     15639: symbol=fclose;  lookup in file=/usr/lib/libstdc++.so.6 [0]
     15639: symbol=fclose;  lookup in file=/usr/lib/libc.so.6 [0]
     15639: binding file /usr/lib/libnss_files.so.2 [0] to /usr/lib/libc.so.6 [0]: normal symbol `fclose' [GLIBC_2.2.5]
Segmentation fault (core dumped)



Brian Silverman

unread,
Apr 17, 2016, 10:36:33 AM4/17/16
to ALMÁSI Dénes, bazel-discuss
I bet it's segfaulting because getpwuid returns NULL on error, and then it just tries to dereference it... I96320287b1886347343c1a50d660c097534d91d2 prints it out nicely; what do you see with that patch applied?

Also, another thing which might help: try using strace instead of gdb. Might point to something in particular it's doing wrong.

ALMÁSI Dénes

unread,
Apr 17, 2016, 11:05:58 AM4/17/16
to Brian Silverman, bazel-discuss
You were right about getpwuid returning NULL. I applied your patch and see the following:

[denes@rudanium r2 [master]]$ (cd /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2 &&   exec env -     PATH=/home/denes/bin:/home/denes/grub-2.00/bin:/home/denes/cross-compiler/build/cross/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/lib/mailman/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/denes/.r2bin:/scripts:/usr/share/java/apache-ant/bin/:/rudanium/bin:/home/denes/bin   /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/_bin/namespace-sandbox @/home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/1aa591e7-995d-4a1a-9195-89a37e6c885a-0.params -- /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o' -MD -MF bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.d -fPIC -c examples/cpp/hello.cc -o bazel-out/local-fastbuild/bin/examples/cpp/_objs/helloworld/examples/cpp/hello.pic.o)
src/main/tools/namespace-sandbox.c:766: sandbox root is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2/bazel-sandbox/1aa591e7-995d-4a1a-9195-89a37e6c885a-0
src/main/tools/namespace-sandbox.c:768: working dir is /home/denes/.cache/bazel/_bazel_denes/d6d1931f967fee42e6ef7c8690ee6565/r2
src/main/tools/namespace-sandbox.c:392: unshare succeeded after 1 tries
getpwuid(getuid()): Success

So errno seems not to be set which implies what getuid() returns does not lead to a valid entry in the /etc/passwd file.

Then I checked this with strace and getuid() returns 65534 (which by convention means nobody - is that even correct in this sandbox situation?), but in my /etc/passwd nobody has UID of 99 so that explains why the entry is not found.

So I got a few questions:
- Is my environment misconfigured or is it Bazel making a false assumption?
- Should this binary even be ran by the user 'nobody' or is that already an error?
- Do you have any suggestion for a non-hacky fix? For a moment I had the idea of creating a user with that UID but I can only imagine problems at the end of that road.

Thanks!

Brian Silverman

unread,
Apr 17, 2016, 11:24:35 AM4/17/16
to ALMÁSI Dénes, bazel-discuss
Your environment is (obviously) different than what anybody else has tried. However, I have no idea if it's your environment or Bazel that's "wrong" yet, so let's keep debugging :)

namespace-sandbox switches to the actual "nobody" user's uid and gid later in its execution (found via getpwnam, in SetupUserNamespaceForNobody), but this early it shouldn't be.

Can you try inserting some getuid() calls in namespace-sandbox.c and figure out when it starts being 65534 vs your actual UID? SwitchToEuid in src/main/tools/process-tools.c looks like a good first guess. Also, regardless of whether SwitchToEuid is or isn't the problem, what does the geteuid call there return?

ALMÁSI Dénes

unread,
Apr 17, 2016, 3:53:20 PM4/17/16
to Brian Silverman, bazel-discuss
(Accidentally didn't send last mail to bazel-discuss)

Brian, moving the SetupDirectories call helped and Bazel was able to build the target.

Thank you! :)

On Sun, Apr 17, 2016 at 6:46 PM, Brian Silverman <bsilve...@gmail.com> wrote:
unshare deals with namespaces, which are really cool. I've found the kernel manpage and this LWN series to be good places to start.

Looks like the solution to #481 didn't quite work. Bazel switches to the real UID of nobody AFTER calling getpwuid with 65534... I think this fixes it without any regressions (running the tests now, and then I'm going to send this out):
diff --git a/src/main/tools/namespace-sandbox.c b/src/main/tools/namespace-sandbox.c
index 8c59a13..3206ad5 100644
--- a/src/main/tools/namespace-sandbox.c
+++ b/src/main/tools/namespace-sandbox.c
@@ -783,12 +783,12 @@ int main(int argc, char *const argv[]) {
   // outside environment.
   CHECK_CALL(mount("none", "/", NULL, MS_REC | MS_PRIVATE, NULL));
 
-  SetupDirectories(&opt);
   if (opt.fake_root) {
     SetupUserNamespace(uid, gid, 0, 0);
   } else {
     SetupUserNamespaceForNobody(uid, gid);
   }
+  SetupDirectories(&opt);
   ChangeRoot(&opt);
 
   SpawnCommand(opt.args, opt.timeout_secs, false);


On Sun, Apr 17, 2016 at 1:14 PM, ALMÁSI Dénes <denes....@gmail.com> wrote:
So I've put a bunch of printf's to the code and it seems the UID and EUID changes in CreateNamespace, more specifically during the successful call to unshare(...).
My printf's say:
getuid(): 1000 geteuid(): 1000 @src/main/tools/namespace-sandbox.c:389
getuid(): 65534 geteuid(): 65534 @src/main/tools/namespace-sandbox.c:394

SwitchToEuid()
 returns 1000 which is my used id.  Also, at the start of the program, euid == uid == 1000 holds.

Now I gonna learn a bit about unshare, I've never seen that function before.
Reply all
Reply to author
Forward
0 new messages