Maximum number of NaCl sandboxes within one process

[david.a...@ucl.ac.uk]

Hello,

Currently, we're building a program similar to the one in tests/custom_desc/ which consists of a host (trusted code) and guest (untrusted code). Host and guest run in one address space and communicate via shared memory. So, we're using the same communication primitives as in the custom_desc example. In contrast to the example, our host program sits in a while loop and creates sequentially a new NaCl sandbox, communicates with the sandbox, and waits for the sandbox's termination. (In other words, there is only one NaCl module / guest running at any point in time.) Our problem is that after 338 times creating and terminating the NaCl sandbox, we receive the following error message:

[12360,3070465920:16:32:57.091395] Native Client module will be loaded at base address 0x000001cf00000000
[12360,3070568192:16:32:57.096635] NaClHostDescFstat: already closed
POST-ABORT: LOG_FATAL abort exit
Aborted (core dumped)

The final lines of the output when running the program with NACLVERBOSITY level 5 are:

[11651,3254572928:16:13:13.659058] NaClWaitForMainThreadToExit: taking NaClApp lock
[11651,3254572928:16:13:13.659076]  waiting for exit status
[11651,3254675200:16:13:13.659082] NaClAppThreadLauncher: entered
[11651,3254675200:16:13:13.659108]       natp = 0x00007f5cc322d800
[11651,3254675200:16:13:13.659110]  prog_ctr  = 0x0000532000021860
[11651,3254675200:16:13:13.659112] stack_ptr  = 0x00005320fffeff68
[11651,3254675200:16:13:13.659113] ix 0: 0x00000000
[11651,3254675200:16:13:13.659132] found first not-all-ones ix 0
[11651,3254675200:16:13:13.659134] Set(0,0x7f5cc322d800) @ix 0: 0x00000000
[11651,3254675200:16:13:13.659136] After @ix 0: 0x00000001, avail_ix 0
[11651,3254675200:16:13:13.659139] Entering syscall 33: return address 0x532000021080
[11651,3254675200:16:13:13.659141] Entered NaClSysSysconf(7f5cc322d800x, 2, 0xfffefecc)
[11651,3254675200:16:13:13.659160] Returning from syscall 33: return value 0 (0x0)
[11651,3254675200:16:13:13.659162] Entering syscall 21: return address 0x532000020fe0
[11651,3254675200:16:13:13.659164] Entered NaClSysMmap(0x00000000,0x10000,0x3,0x22,-1,0xfffefee8)
[11651,3254675200:16:13:13.659167]  offset = 0x00000000
[11651,3254675200:16:13:13.659170] NaClSysMmap: FindMapSpace: page 0xfeff0
[11651,3254675200:16:13:13.659172] NaClSysMmap: new starting addr: 0xfeff0000
[11651,3254675200:16:13:13.659174] NaClSysMmap: NaClDescIoDescMap(,,0x5320feff0000,0x00010000,0x3,0x32,0x00000000)
[11651,3254675200:16:13:13.659176] NaClHostDescMap(0x00000000, 0x5320feff0000, 0x00010000, 0x3, 0x32, 0x00000000)
[11651,3254675200:16:13:13.659179] NaClHostDescMap: host_prot 0x3, host_flags 0x32
[11651,3254675200:16:13:13.659185] NaClHostDescMap: mmap returned 5320feff0000
[11651,3254675200:16:13:13.659187] NaClHostDescMap: returning 0x5320feff0000
[11651,3254675200:16:13:13.659201] NaClDescIoDescMap returning 5320feff0000
[11651,3254675200:16:13:13.659202] NaClVmmapUpdate(0x7f5cc2361e20, 0xfeff0, 0x10, 0x3, 0x32, 0, 0x0, 0x0)
[11651,3254675200:16:13:13.659205] NaClVmmapAdd(0x7f5cc2361e20, 0xfeff0, 0x10, 0x3, 0x32, 0x0, 0x0)
[11651,3254675200:16:13:13.659207] NaClVmmapEntryMake(0xfeff0,0x10,0x3,0x32,0x0,0x0)
[11651,3254675200:16:13:13.659209] entry: 0x7f5cbc005d80
[11651,3254675200:16:13:13.659210] NaClSysMmap: returning 0xfeff0000
[11651,3254675200:16:13:13.659212] Returning from syscall 21: return value -16842752 (0xfeff0000)
[11651,3254675200:16:13:13.659219] Entering syscall 82: return address 0x532000020660
[11651,3254675200:16:13:13.659221] Entered NaClSysTlsInit(0x7f5cc322d800, 0xfeff0488)
[11651,3254675200:16:13:13.659223] NaClSysTlsInit: thread_ptr 0xfeff0488, sys_tls 0x5320feff0488
[11651,3254675200:16:13:13.659225] Returning from syscall 82: return value 0 (0x0)
[11651,3254675200:16:13:13.659228] Entering syscall 17: return address 0x532000020c20
[11651,3254675200:16:13:13.659230] Entered NaClSysFstat(0x7f5cc322d800, 1, 0xfffef230)
[11651,3254675200:16:13:13.659232] NaClRefCountRef(0x7f5cc322d5c0).
[11651,3254675200:16:13:13.659233] NaClHostDescFstat: already closed
POST-ABORT: LOG_FATAL abort exit
Aborted (core dumped)

The error message "NaClHostDescFstat: already closed" gets printed in the file src/shared/platform/nacl_host_desc_common.c.
We currently don't understand why we get this error? Is it because any resource limits are reached? Is there an upper limit of NaCl modules we can create sequentially within one process?

A second question is how to effectively debug such problems using gdb (instead of printf() and log messages). When I use the normal host's gdb or toolchain/linux_x86/nacl_x86_newlib/bin/x86_64-nacl-gdb, I was unable to use the gdb command file <executable> because it said "no debugging symbols found". After inserting the line

CXXFLAGS='-g',

in the file SConstruct within pre_base_env = Environment(), it can read the symbols.
In the normal host's gdb I can even run the above program, but somehow I'm unable to set breakpoints.
When I try to run the program in toolchain/linux_x86/nacl_x86_newlib/bin/x86_64-nacl-gdb, I get the following error:
Don't know how to run.  Try "help target".
Neither can I run the program with the gdb command target exec <executable>.

I read the following web sites:
https://developer.chrome.com/native-client/devguide/devcycle/debugging
https://www.chromium.org/nativeclient/how-tos/debugging-documentation/debugging-with-debug-stub-recommended/getting-started-with-debug-stub
The first page rather describes how to debug using the Chrome browser (which we don't need for our program).
The second page says "Debugging command line NaCl applications is enabled by passing -g switch to sel_ldr." but we don't directly call sel_ldr in the command line.

So my question is whether someone could provide some tips how to use gdb to debug for example tests/custom_desc/?

Many thanks,
David