Build sandboxing disabled due to nsjail error —— Invalid argument

[Mengshui Chen]

This component is for issues with building an AOSP Platform build (Soong and Make), not for runtime issues (app bugs/crashes, ANRs, etc).

When trying to build AOSP, I'm getting the following warning:

2021/12/01 13:38:23.444890 build/soong/ui/build/build.go:165: Total RAM: 62.7GB

2021/12/01 13:38:26.100626 build/soong/ui/build/sandbox_linux.go:123: [prebuilts/build-tools/linux-x86/bin/nsjail -H android-build -e -u nobody -g nobody -R / -B /tmp -B /home/chenmengshui/aosp -B /home/chenmengshui/aosp/out --disable_clone_newcgroup -- /bin/bash -c if [ $(hostname) == "android-build" ]; then echo "Android" "Success"; else echo Failure; fi]

2021/12/01 13:38:26.109367 build/soong/ui/build/sandbox_linux.go:130: Build sandboxing disabled due to nsjail error.

2021/12/01 13:38:26.109398 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mode: STANDALONE_ONCE

2021/12/01 13:38:26.109412 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Jail parameters: hostname:'android-build', chroot:'', process:'/bin/bash', bind:[::]:0, max_conns:0, max_conns_per_ip:0, time_limit:0, personality:0, daemonize:false, clone_newnet:true, clone_newuser:true, clone_newns:true, clone_newpid:true, clone_newipc:true, clone_newuts:true, clone_newcgroup:false, clone_newtime:false, keep_caps:false, disable_no_new_privs:false, max_cpus:0

2021/12/01 13:38:26.109420 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/' flags:MS_RDONLY type:'tmpfs' options:'' dir:true

2021/12/01 13:38:26.109428 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/' -> '/' flags:MS_RDONLY|MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true

2021/12/01 13:38:26.109436 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/tmp' -> '/tmp' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true

2021/12/01 13:38:26.109443 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/home/chenmengshui/aosp' -> '/home/chenmengshui/aosp' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true

2021/12/01 13:38:26.109450 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/home/chenmengshui/aosp/out' -> '/home/chenmengshui/aosp/out' flags:MS_BIND|MS_REC|MS_PRIVATE type:'' options:'' dir:true

2021/12/01 13:38:26.109457 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Mount: '/proc' flags:MS_RDONLY type:'proc' options:'' dir:true

2021/12/01 13:38:26.109464 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Uid map: inside_uid:99 outside_uid:994 count:1 newuidmap:false

2021/12/01 13:38:26.109474 build/soong/ui/build/sandbox_linux.go:133: [I][2021-12-01T13:38:26+0800] Gid map: inside_gid:99 outside_gid:990 count:1 newgidmap:false

2021/12/01 13:38:26.109483 build/soong/ui/build/sandbox_linux.go:133: [W][2021-12-01T13:38:26+0800][18452] pid_t subproc::runChild(nsjconf_t *, int, int, int, int)():471 clone(flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNET) failed: Invalid argument

2021/12/01 13:38:26.109490 build/soong/ui/build/sandbox_linux.go:133: [E][2021-12-01T13:38:26+0800][18452] int nsjail::standaloneMode(nsjconf_t *)():272 Couldn't launch the child process

2021/12/01 13:38:26.109507 build/soong/ui/build/sandbox_linux.go:139: nsjail failed with exit status 255

This line points to the source of the problem

pid_t subproc::runChild(nsjconf_t *, int, int, int, int)():471 clone(flags=CLONE_NEWNS|CLONE_NEWUTS|CLONE_NEWIPC|CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNET) failed: Invalid argument

Then I found it in the source file: https://github.com/google/nsjail/blob/master/subproc.cc

pid_t runChild(nsjconf_t* nsjconf, int netfd, int fd_in, int fd_out, int fd_err) {

if (!net::limitConns(nsjconf, netfd)) {

return 0;

}

unsigned long flags = 0UL;

flags |= (nsjconf->clone_newnet ? CLONE_NEWNET : 0);

flags |= (nsjconf->clone_newuser ? CLONE_NEWUSER : 0);

flags |= (nsjconf->clone_newns ? CLONE_NEWNS : 0);

flags |= (nsjconf->clone_newpid ? CLONE_NEWPID : 0);

flags |= (nsjconf->clone_newipc ? CLONE_NEWIPC : 0);

flags |= (nsjconf->clone_newuts ? CLONE_NEWUTS : 0);

flags |= (nsjconf->clone_newcgroup ? CLONE_NEWCGROUP : 0);

flags |= (nsjconf->clone_newtime ? CLONE_NEWTIME : 0);

if (nsjconf->mode == MODE_STANDALONE_EXECVE) {

LOG_D("unshare(flags: %s)", cloneFlagsToStr(flags).c_str());

if (unshare(flags) == -1) {

PLOG_F("unshare(%s)", cloneFlagsToStr(flags).c_str());

}

subprocNewProc(nsjconf, netfd, fd_in, fd_out, fd_err, -1);

LOG_F("Launching new process failed");

}

LOG_D("Creating new process with clone flags:%s and exit_signal:SIGCHLD",

    cloneFlagsToStr(flags).c_str());

int sv[2];

if (socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0, sv) == -1) {

PLOG_E("socketpair(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC) failed");

return -1;

}

int child_fd = sv[0];

int parent_fd = sv[1];

pid_t pid = cloneProc(flags, SIGCHLD);

if (pid == 0) {

close(parent_fd);

subprocNewProc(nsjconf, netfd, fd_in, fd_out, fd_err, child_fd);

util::writeToFd(child_fd, &kSubprocErrorChar, sizeof(kSubprocErrorChar));

LOG_F("Launching child process failed");

}

close(child_fd);

if (pid == -1) {

auto saved_errno = errno;

PLOG_W("clone(flags=%s) failed", cloneFlagsToStr(flags).c_str());

close(parent_fd);

errno = saved_errno;

return pid;

}

addProc(nsjconf, pid, netfd);

if (!initParent(nsjconf, pid, parent_fd)) {

close(parent_fd);

return -1;

}

char rcvChar;

if (util::readFromFd(parent_fd, &rcvChar, sizeof(rcvChar)) == sizeof(rcvChar) &&

    rcvChar == kSubprocErrorChar) {

LOG_W("Received error message from the child process before it has been executed");

close(parent_fd);

return -1;

}

close(parent_fd);

return pid;

}

The value of pid is from the cloneProc(flags, SIGCHLD)

I'm building in a linux environment. And I suspect that many processes running concurrently is part of the issue, but I'm not sure. So How to Fix this nsjail error?