need help getting heapchecker running

[Antony Sargent]

I'm getting the error copied below when I try to run the heapchecker tool on unit_tests or browser_tests. I've tried setting the linux_use_heapchecker flag in my GYP defines, re-ran the build/install-build-deps.sh script, did a gclient runhooks followed by a clean build, and seem to get the same result. 

Anyone know what the problem might be?

The error output:

./tools/heapcheck/chrome_tests.sh -t unit --gtest_filter='ExtensionMenu*'

20:37:09 chrome_tests.py [INFO] running test unit

20:37:09 heapcheck_test.py [INFO] starting execution...

20:37:09 heapcheck_test.py [INFO] export G_SLICE=always-malloc

20:37:09 heapcheck_test.py [INFO] export NSS_DISABLE_ARENA_FREE_LIST=1

20:37:09 heapcheck_test.py [INFO] export GTEST_DEATH_TEST_USE_FORK=1

20:37:09 heapcheck_test.py [INFO] export HEAPCHECK=strict

20:37:09 heapcheck_test.py [INFO] export HEAP_CHECK_MAX_LEAKS=-1

20:37:09 heapcheck_test.py [INFO] export PPROF_PATH=/work/chrome/src/tools/heapcheck/../../third_party/tcmalloc/chromium/src/pprof

20:37:09 heapcheck_test.py [INFO] export LD_PRELOAD=/usr/lib/debug/libstdc++.so

20:37:09 common.py [INFO] running sh /work/chrome/src/tools/heapcheck/heapcheck_std.sh /work/chrome/src/out/Debug/unit_tests --gtest_print_time --gtest_filter=ExtensionMenu*, timeout 1200 sec

20:37:09 common.py [INFO] started subprocess

20:37:09 common.py [INFO] process ended, did not time out

20:37:09 common.py [INFO] flushing stdout

20:37:09 common.py [INFO] collecting result code

20:37:09 common.py [ERROR] sh exited with non-zero result code 127

/work/chrome/src/out/Debug/unit_tests: /usr/lib/debug/libstdc++.so: no version information available (required by /work/chrome/src/out/Debug/unit_tests)

/work/chrome/src/out/Debug/unit_tests: /usr/lib/debug/libstdc++.so: no version information available (required by /work/chrome/src/out/Debug/unit_tests)

/work/chrome/src/out/Debug/unit_tests: /usr/lib/debug/libstdc++.so: no version information available (required by /work/chrome/src/out/Debug/unit_tests)

/work/chrome/src/out/Debug/unit_tests: relocation error: /work/chrome/src/out/Debug/unit_tests: symbol _ZNSs4_Rep26_M_set_length_and_sharableEm, version GLIBCXX_3.4.5 not defined in file libstdc++.so.6 with link time reference

20:37:09 heapcheck_test.py [INFO] elapsed time: 00:00:00

20:37:09 heapcheck_test.py [INFO] For more information on the Heapcheck bot see http://dev.chromium.org/developers/how-tos/using-the-heap-leak-checker

Environment variables I set that may be relevant:

export GYP_GENERATORS="make,scons"

BINPATH="$CHROME_DIR/bin"

export CC="ccache distcc gcc '-B$BINPATH'"

export CXX="ccache distcc g++ '-B$BINPATH'"

export GYP_DEFINES="linux_use_heapchecker=1"

[Alexander Potapenko]

> Anyone know what the problem might be?

Looks like there are problems with preloading libstdc++.so on your
system. To check that, try running:

LD_PRELOAD=/usr/lib/debug/libstdc++.so /work/chrome/src/out/Debug/unit_tests

-- that should report the same relocation error. If that's the
problem, you can locally disable the LD_PRELOAD env var in
src/tools/heapcheck/heapcheck_test.py as a workaround.

Sorry for the long delay,
Alex

[Antony Sargent]

Disabling LD_PRELOAD worked.

In a recent check-in I introduced a leak that triggered the heapcheck and valgrind bots to go red (now fixed), but when I ran valgrind locally it didn't report the error. I started reverting my src/third_party/valgrind directory to older versions to see if an update to our valgrind binaries was causing the problem, and found that after the following check-in you did my copy of valgrind was no longer finding leaks: 

http://src.chromium.org/viewvc/chrome?view=rev&revision=48923

It seems that there is something about my machine configuration (a standard Googler ubuntu setup) which is different from both the bots and whatever machine you're using for testing, so it seems pretty important for us to understand what's going wrong here with both valgrind and heapcheck, and see if this is likely to be affecting other people. I'll contact you off-list, and then we can update this thread with our findings.

[Antony Sargent]

The revision of valgrind I mentioned was incorrect - it was actually 48921 that made things stop working: 

http://src.chromium.org/viewvc/chrome?view=rev&revision=48921

[Antony Sargent]

2 quick updates:

Valgrind: Evidence seems to point to a problem with intercepting tcmalloc in the newer valgrind binaries in third_party/valgrind. Building unit_tests with GYP_DEFINES=linux_use_tcmalloc=0 makes valgrind find leaks again for me (or rolling back my third_party/valgrind to r49172).

Heapcheck: For some reason, having LD_PRELOAD in my environment causes heapcheck to not work. Commenting out the line in tools/heapcheck/heapcheck_test.py that adds LD_PRELOAD at least lets heapcheck run, but it's not clear to me if it's catching leaks that it should be.

[Alexander Potapenko]

On Thu, Jul 1, 2010 at 9:26 PM, Antony Sargent <asar...@chromium.org> wrote:
> 2 quick updates:
> Valgrind: Evidence seems to point to a problem with intercepting tcmalloc in
> the newer valgrind binaries in third_party/valgrind. Building unit_tests
> with GYP_DEFINES=linux_use_tcmalloc=0 makes valgrind find leaks again for me
> (or rolling back my third_party/valgrind to r49172).

I'll try to investigate this. In fact the binaries submitted in that
revision were meant to intercept tcmalloc in a correct way.
Can you please post the list of REDIR lines printed by
tools/valgrind/valgrind.sh -v <your binary with proper gtest filter>
?

> Heapcheck: For some reason, having LD_PRELOAD in my environment causes
> heapcheck to not work. Commenting out the line in
> tools/heapcheck/heapcheck_test.py that adds LD_PRELOAD at least lets
> heapcheck run, but it's not clear to me if it's catching leaks that it
> should be.

It is. In fact we're just using the debug version of libstdc++ to get
better stacks in the memory leak reports.
Using the default libstdc++ with omitted frame pointers can cause some
reports to contain only the top frame, but the leaks are still
discoverable.
I'm able to reproduce the problem on one of my machines, but it has
nothing to do with heapcheck -- the system just can't replace one
libstdc++ with another one.