OutOfMemoryDeathTest.ViaSharedLibraries failures on Gentoo Linux

[Paweł Hajdan, Jr.]

While running base_unittests in Release mode on Gentoo Linux I'm getting the following failure:

[ RUN      ] OutOfMemoryDeathTest.ViaSharedLibraries

[WARNING] ../../testing/gtest/src/gtest-death-test.cc:825:: Death tests use fork(), which is unsafe particularly in a threaded context. For this test, Google Test couldn't detect the number 

of threads.

../../base/process/memory_unittest.cc:270: Failure

Death test: { SetUpInDeathAssert(); switch (0) case 0: default: if (const ::testing::AssertionResult gtest_ar = (::testing::internal:: EqHelper<(sizeof(::testing::internal::IsNullLiteralHelp

er(-1)) == 1)>::Compare("-1", "asprintf(&value, format.c_str(), 0)", -1, asprintf(&value, format.c_str(), 0)))) ; else ::testing::internal::AssertHelper(::testing::TestPartResult::kNonFatalF

ailure, "../../base/process/memory_unittest.cc", 270, gtest_ar.failure_message()) = ::testing::Message(); }

    Result: failed to die.

 Error msg:

[  DEATH   ] 

[  FAILED  ] OutOfMemoryDeathTest.ViaSharedLibraries (2 ms)

It seems that the test is important for security, so I'd like to find the root cause and fix it.

It's not obvious to me where to start, and what's different from Ubuntu, where is apparently works.

Some technical details from the Gentoo system:

I'm building using "build/gyp_chromium -Dwerror=" and "ninja -C out/Release base_unittests".

Current revision is src@238459. I run the test via "out/Release/base_unittests --gtest_filter=OutOfMemoryDeathTest.ViaSharedLibraries", and out/Release/base_unittests also leads to a failure.

I'm using gcc-4.7.3, binutils-2.23.1, and glibc-2.16.0.

Do you have recommendations for further debugging?

Paweł

[Paweł Hajdan, Jr.]

This is still happening as of src@243555. This is with gcc-4.7.3, binutils-2.23.2 and glibc-2.17.

Adam, could you help with this test that you added in https://codereview.chromium.org/533001?

Paweł

[Paweł Hajdan, Jr.]

The failing tests on Gentoo doesn't crash. On Ubuntu, it crashes as expected:

#0  tc_malloc (size=18446744073709539359) at ../../third_party/tcmalloc/chromium/src/debugallocation.cc:1167

#1  0x00007ffff5f1d0c3 in _IO_vfprintf_internal (s=<optimized out>, format=<optimized out>, ap=<optimized out>) at vfprintf.c:1527

#2  0x00007ffff5f460d4 in _IO_vasprintf (result_ptr=0x7fffffffd0b8, format=0x2d3305f32658 "%18446744073709539327d", args=0x7fffffffcf18) at vasprintf.c:64

#3  0x00007ffff5f26a97 in ___asprintf (string_ptr=<optimized out>, format=<optimized out>) at asprintf.c:37

#4  0x0000000000760a67 in OutOfMemoryDeathTest_ViaSharedLibraries_Test::TestBody (this=0x2d3305ee41a0) at ../../base/process/memory_unittest.cc:266

I've also stepped through the tc_malloc calls, and on Gentoo asprintf just doesn't perform this humongous allocation.

Since this is inside glibc, I think it can be attributed to different glibc. On Ubuntu it's 2.15, while on that Gentoo system it's 2.17 (2.16 was also failing).

I may experiment with maybe compiling different glibc versions locally on Ubuntu and running against that, and also looking what changed in asprintf implementation.

Do you have some suggestions about other ways to trigger huge memory allocations in glibc? Currently the test assumes that asprintf implementation will do the huge allocation, but it doesn't have to.

Paweł

On Wed, Jan 8, 2014 at 8:04 PM, Adam Langley <a...@chromium.org> wrote:

On Wed, Jan 8, 2014 at 9:56 AM, Paweł Hajdan, Jr.
<phajd...@chromium.org> wrote:
> Adam, could you help with this test that you added in
> https://codereview.chromium.org/533001?

You can take the g_try_malloc call out of the ASSERT_DEATH macro and
single step. It should enter Chromium's malloc functions and crash.


Cheers

AGL

[Mike Frysinger]

just leak memory instead ?

while (1)

  malloc(1024 * 1024);

-mike

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Roland McGrath]

On Thu, Jan 9, 2014 at 2:13 PM, Mike Frysinger <vap...@chromium.org> wrote:
> just leak memory instead ?
> while (1)
> malloc(1024 * 1024);

That fails to test what the test intends to test. It's specifically a test
to verify that libc's internal calls to an allocator use the interposed
malloc rather than libc's own malloc.

To add some more information without necessarily solving the problem:

What changed in libc is that *printf got more robust checking for integer
overflow in a variety of cases. In particular, it was decided that field
width and precision values are always 'int'. This was deemed to be
sensible given that when using %* or %.*, the dynamic field width or
precision is an 'int' argument. It now uniformly detects any such value
that tries to exceed INT_MAX, which is impossible with the dynamic
arguments but easy with static format strings like the one in this test.
Now *printf will fail with EOVERFLOW for this case before anything attempts
any kind of allocation.

So this could be addressed in a variety of ways:

1. Change the test setup so that the Chromium-supplied malloc is one that
fails characteristically with a size of INT_MAX or greater and then use
INT_MAX as the field width.

2. Use asprintf but verify that it's calling the right malloc in some way
other than by expecting it to fail because the size is too large. A
variety of options come to mind, but everything I've thought of just now
entails modifying the malloc function linked into the test program.

3. Use some other libc function that calls malloc and that can be reliably
(i.e. guaranteed by the documented API) to attempt a malloc call of a
very large size that you control.

I don't know off hand that there is such an interface to use or, if so,
what it is. Someone (like Mike or I) who is comfortable trolling
through the libc code could put some effort into finding one.


Thanks,
Roland

[Paweł Hajdan, Jr.]

This is now failing on Ubuntu 14.04 LTS Trusty Tahr.

Since we'd like to run it one the infrastructure, any help with this test would be welcome.

Paweł