Build error - Bad NaCl helper startup ack

[Jon Butler]

Hi All,

I've been trying for a while to build chromium on Linux (latest attempt was on Ubuntu 13.10 x64). I'm doing a component build with ninja, and the build succeeds with only a few harfbuzz hidden symbol warnings. When I run the chrome binary, nothing happens, and after a Ctrl-C it prints:

"[xxx:ERROR:nacl_fork_delegate_linux.cc(226)] Bad NaCl helper startup ack (0 bytes)."

There has been a thread about this before, and I've checked everything that was discussed there (libelf-dev is installed, nacl_helper_bootstrap was built and seems functional). I'd rather not run it headless as was discussed in the previous thread, because I want it for more than running unit tests.

Has anyone else encountered this problem?

Thanks,

Jon

[Jon Butler]

As a followup, I did some debugging, and it appears the problem occurs before the NaCl error. Running chrome under strace shows the last operation as a successful open on /dev/urandom.

To me that raises more questions than it answers, but does anyone have any insight as to why chrome might be hanging after opening or while reading /dev/urandom?

Thanks,

Jon

[Lei Zhang]

Have you tried attaching a debugger to the chrome processes to see
what state they are in? You'll have much more insight into what the
browser is doing with gdb than with strace.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev

[Nick Bray]

I am not an expert in the code path, but I have a sneaking suspicion that the NSS library that comes with 13.10 is hosing the init of the helper process.

https://code.google.com/p/chromium/codesearch#chromium/src/chrome/nacl/nacl_helper_linux.cc&l=361

There seems to be some code skew between the helper and the zygote, it makes me suspicious code has evolved and NaCl was not updated.  Hmmm.  If you can tell how far you're getting in main() before hanging, that would help diagnose the problem.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Jon Butler]

Ok, I'll take a look under gdb later. I was a little sceptical that approach would get me any more information, because the process doesn't crash or hang, and I'm not even sure which thread to look at (or which renderer if any have been spawned yet).

Regarding NSS in 13.10, I've reproduced this behaviour both on older (supported) Ubuntus, and on Debian, so I don't believe it's distro-specific. I'm currently not even sure if it's a NaCl problem at all, it just seems that error message is printed after the Ctrl-C. More info later hopefully...

[Jon Butler]

Ok, here's the issue in GDB. As expected, it's definitely an issue getting randomness from /dev/urandom. I'm running in a VM here, might that cause some issues? There's a CHECK() immediately after the call to ReadFromFD, so if that had finished and failed I would expect to at least see some output...

chrome-dev@chrome-dev:~/src$ gdb ./out/Release/chrome
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/chrome-dev/src/out/Release/chrome...done.
(gdb) run --no-sandbox
Starting program: /home/chrome-dev/src/./out/Release/chrome --no-sandbox
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffda83c700 (LWP 60446)]
[New Thread 0x7fffd6cf9700 (LWP 60447)]
[New Thread 0x7fffd8279700 (LWP 60448)]
[New Thread 0x7fffd8238700 (LWP 60449)]
[New Thread 0x7fffd5be6700 (LWP 60450)]
[New Thread 0x7fffd365d700 (LWP 60451)]
[Thread 0x7fffd5be6700 (LWP 60450) exited]
[Thread 0x7fffd365d700 (LWP 60451) exited]
[New Thread 0x7fffd365d700 (LWP 60452)]
[New Thread 0x7fffda01c700 (LWP 60453)]
^C
Program received signal SIGINT, Interrupt.
ReadFromFD () at ../../base/file_util_posix.cc:347
347	        HANDLE_EINTR(read(fd, buffer + total_read, bytes - total_read));
(gdb) bt
#0  ReadFromFD () at ../../base/file_util_posix.cc:347
#1  0x00007ffff518ee13 in RandUint64 () at ../../base/rand_util_posix.cc:49
#2  0x00007ffff518d8f5 in RandGenerator () at ../../base/rand_util.cc:58
#3  0x00007ffff518d556 in RandInt () at ../../base/rand_util.cc:22
#4  0x00005555564ab12d in GenerateLowEntropySource ()
    at ../../chrome/browser/metrics/metrics_service.cc:308
#5  GetLowEntropySource () at ../../chrome/browser/metrics/metrics_service.cc:1101
#6  0x00005555564aa07c in CreateEntropyProvider ()
    at ../../chrome/browser/metrics/metrics_service.cc:540
#7  0x0000555556a1b447 in SetupMetricsAndFieldTrials ()
    at ../../chrome/browser/chrome_browser_main.cc:577
#8  0x0000555556a2214d in PreCreateThreadsImpl ()
    at ../../chrome/browser/chrome_browser_main.cc:1003
#9  0x0000555556a1ee6a in PreCreateThreads () at ../../chrome/browser/chrome_browser_main.cc:761
#10 0x00007fffe8d82cc8 in PreCreateThreads () at ../../content/browser/browser_main_loop.cc:534
#11 0x00007fffe946f8a0 in Run () at ../../base/callback.h:396
#12 RunAllTasksNow () at ../../content/browser/startup_task_runner.cc:45
#13 0x00007fffe8d83b1e in CreateStartupTasks ()
    at ../../content/browser/browser_main_loop.cc:603
#14 0x00007fffe8d8f14d in Initialize () at ../../content/browser/browser_main_runner.cc:109
#15 0x00007fffe8d7bf38 in BrowserMain () at ../../content/browser/browser_main.cc:22
#16 0x00007fffe8d34445 in RunNamedProcessTypeMain ()
    at ../../content/app/content_main_runner.cc:458
#17 0x00007fffe8d35eb9 in Run () at ../../content/app/content_main_runner.cc:777
#18 0x00007fffe8d32242 in ContentMain () at ../../content/app/content_main.cc:35
#19 0x0000555555e7a016 in ChromeMain () at ../../chrome/app/chrome_main.cc:39
#20 0x0000555555e79f3b in main () at ../../chrome/app/chrome_exe_main_gtk.cc:43

[Jon Butler]

Further lulz - It appears the problem is in HANDLE_EINTR, where it sits in an infinite loop. 64-bit calling conventions confuse me no end, so I can't work out if read() succeeds or not, but upon entering HANDLE_EINTR, the following code gets hit: 

0x7ffff50c3da0 <ReadFromFD()+16>()+16>:    jmp    0x7ffff50c3da0 <ReadFromFD()+16>

So it's basically looping unconditionally. How it got that from the code in chromium/src/base/posix/eintr_wrapper.h I don't know. There's a limit of 100 iterations in the source.

Perhaps a 64-bit bug? But then how has it not been spotted before? 

On 24 October 2013 21:26, Jon Butler <jbu...@chromium.org> wrote:

Tried it, and it completed pretty much instantly as expected.

[Jon Butler]

Solved.

For future reference, this was as a compiler bug when using -D_FORTIFY_SOURCE=2, which results in an infinite loop. The bug was reported upstream and duped with http://llvm.org/bugs/show_bug.cgi?id=16821, which is still not patched. For Chromium, a workaround would be to remove -D_FORTIFY_SOURCE=2 from build/common.gypi when clang is in use.