Investigating Android native memory leak

[Lambros Lambrou]

Hi,

I'm trying to find the cause of a memory leak in the Chromoting Android app, and I'm looking for pointers to appropriate tools that can help.

The leak is in the native heap, not the Dalvik heap, so I need something that can report native allocations that are not freed.

I have already tried adding logs to some ctor/dtor pairs that I thought were responsible, but had no success.

I tried setting up ASAN on Android, but I didn't see anything reported in the device log. Looking deeper, it seems that LeakSanitizer is only supported on x86_64 Linux: https://www.chromium.org/developers/testing/leaksanitizer .

I found this: https://developer.chrome.com/devtools/docs/heap-profiling

but it seems to be geared toward Chrome itself, so I'm looking for something I could apply to Chromoting.apk.

Many thanks,

Lambros

[Newton Allen]

+Yusuf, who's investigated several memory leaks on Android

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Maria Khomenko]

You may be be interested in http://www.chromium.org/developers/memory-bloat

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Lambros Lambrou]

Thanks for the link. I found this page: http://www.chromium.org/developers/deep-memory-profiler

and I'm trying to do a build with

GYP_DEFINES='OS=android profiling=1 disable_debugallocation=1 android_full_debug=1 use_allocator="tcmalloc"'

but I'm running into this error:

FAILED: ..blah-blah/third_party/android_tools/ndk//toolchains/arm-linux-androideabi-4.9/prebuilt/linux-x86_64/bin/arm-linux-androideabi-g++ -MMD -MF obj/third_party/skia/src/opts/skia_opts.SkBlitRow_opts_arm.o.d -DV8_DEPRECATION_WARNINGS -D_FILE_OFFSET_BITS=64 -DDISABLE_NACL -DCHROMIUM_BUILD -DCR_CLANG_REVISION=233105-1 -DUSE_LIBJPEG_TURBO=1 -DENABLE_PROFILING=1 -DENABLE_WEBRTC=1 -DUSE_PROPRIETARY_CODECS -DENABLE_BROWSER_CDMS -DENABLE_CONFIGURATION_POLICY -DENABLE_NOTIFICATIONS -DSYSTEM_NATIVELY_SIGNALS_MEMORY_PRESSURE -DDONT_EMBED_BUILD_METADATA -DENABLE_AUTOFILL_DIALOG=1 -DCLD_VERSION=1 -DENABLE_PRINTING=1 -DENABLE_BASIC_PRINTING=1 -DENABLE_SUPERVISED_USERS=1 -DVIDEO_HOLE=1 -DV8_USE_EXTERNAL_STARTUP_DATA -DSK_SUPPORT_GPU=1 -DSK_LEGACY_DRAWPICTURECALLBACK -DSK_SUPPORT_LEGACY_OPTIONLESS_GET_PIXELS -DSK_BUILD_FOR_ANDROID '-DSK_DEFAULT_FONT_CACHE_LIMIT=(1*1024*1024)' -DSK_GAMMA_APPLY_TO_A8 -DSK_GAMMA_EXPONENT=1.4 -DSK_GAMMA_CONTRAST=0.0 -DSK_ARM_HAS_OPTIONAL_NEON -DUSE_LIBPCI=1 -DUSE_OPENSSL=1 -DUSE_OPENSSL_CERTS=1 -DANDROID -D__GNU_SOURCE=1 -DUSE_STLPORT=1 -D_STLP_USE_PTR_SPECIALIZATIONS=1 '-DCHROME_BUILD_ID=""' -DHAVE_SYS_UIO_H -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -D_DEBUG -Igen -I../../third_party/skia/include/core -I../../third_party/skia/include/effects -I../../third_party/skia/include/utils -I../../third_party/skia/src/core -I../../third_party/skia/src/opts -I../../third_party/skia/src/utils -I../.. -I../../skia/config -fstack-protector --param=ssp-buffer-size=4 -fno-strict-aliasing -Wno-unused-parameter -Wno-missing-field-initializers -fvisibility=hidden -pipe -fPIC -Wno-unused-local-typedefs -Wno-format -march=armv7-a -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -fno-tree-sra -fno-caller-saves -Wno-psabi -mthumb-interwork -fno-optimize-sibling-calls -ffunction-sections -funwind-tables -g -fstack-protector -fno-short-enums -finline-limit=64 --sysroot=../../third_party/android_tools/ndk//platforms/android-14/arch-arm -isystem../../third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport -O0 -g -gdwarf-4 -funwind-tables -fno-exceptions -fno-rtti -fno-threadsafe-statics -fvisibility-inlines-hidden -Wno-deprecated -Wno-abi -std=gnu++11 -Wno-narrowing -Wno-literal-suffix  -c ../../third_party/skia/src/opts/SkBlitRow_opts_arm.cpp -o obj/third_party/skia/src/opts/skia_opts.SkBlitRow_opts_arm.o

../../third_party/skia/src/opts/SkBlitRow_opts_arm.cpp: In function 'void S32A_D565_Opaque(uint16_t*, const SkPMColor*, int, U8CPU, int, int)':

../../third_party/skia/src/opts/SkBlitRow_opts_arm.cpp:114:1: error: r7 cannot be used in asm here

 }

 ^

Any ideas?

[Lambros Lambrou]

This seems to hint at the cause of the problem:

https://code.google.com/p/chromium/codesearch#chromium/src/skia/skia_library_opts.gyp&q=armv7_sources&sq=package:chromium&type=cs&l=78

Could someone please help me figure out how to get this to work? Or maybe point me to a different memory-leak tool?

[Mike Klein]

That code is pretty weird, and uses the frame pointer as a general purpose register.  To be able to build it, you need to make sure -fomit-frame-pointer ends up on the commandline when compiling that file.

[Lambros Lambrou]

On Fri, Apr 17, 2015 at 2:06 PM, Mike Klein <mtk...@google.com> wrote:

That code is pretty weird, and uses the frame pointer as a general purpose register.  To be able to build it, you need to make sure -fomit-frame-pointer ends up on the commandline when compiling that file.

Thanks, but I'm concerned that the flag might interfere with the memory-analysis machinery somehow. Instead, I tweaked skia_library_opts.gyp:67 conditional to use 'none_sources' instead of 'armv7_sources'.

Cutting a long story short, I needed to add "allocator.gyp:allocator" to my dependencies to get my build to link.

Also, I couldn't get it to work with tcmalloc at all:

In file included from ../../third_party/android_tools/ndk//platforms/android-14/arch-arm/usr/include/../include/signal.h:42:0,

                 from ../../third_party/android_tools/ndk//sources/cxx-stl/stlport/stlport/signal.h:28,

                 from ../../third_party/tcmalloc/chromium/src/profiler.cc:38:

../../third_party/android_tools/ndk//platforms/android-14/arch-arm/usr/include/sys/ucontext.h:77:3: error: conflicting declaration 'typedef struct ucontext ucontext_t'

 } ucontext_t;

   ^

In file included from ../../third_party/tcmalloc/chromium/src/profiler.cc:37:0:

../../third_party/tcmalloc/chromium/src/getpc.h:178:32: note: previous declaration as 'typedef struct _Unwind_Context ucontext_t'

 typedef struct _Unwind_Context ucontext_t;

                                ^

[19/41] CXX obj/third_party/tcmalloc/chromium/src/allocator.thread_cache.o

ninja: build stopped: subcommand failed.

Apparently, "ctmalloc" is the new hotness, and I was able to get a successful build with this setting:

GYP_DEFINES='OS=android profiling=1 disable_debugallocation=1 android_full_debug=1 use_allocator="ctmalloc"'

The app runs fine, and it shows the memory-leak as before, but I haven't been able to get any heap dumps at all (following the directions at http://www.chromium.org/developers/deep-memory-profiler)

Maybe there's some C++ function we have to call on startup, to initiate heap-monitoring? Probably Chrome does something like this, which is missing in our app?

Or maybe it's because I'm using ctmalloc instead of tcmalloc?

Lambros

[Will Harris]

Deep memory profiling relies on tcmalloc so if you're disabling tcmalloc you're not going to get any memory profiling.

Will

[Lambros Lambrou]

Thanks! In that case, I need to get tcmalloc working. For reference, here's what I've done:

Tweaked Skia to not use assembler.

Clobbered the build (rm -rf out).

Added '../base/allocator/allocator.gyp:allocator' to my dependencies.

export GYP_DEFINES='OS=android profiling=1 disable_debugallocation=1 android_full_debug=1 use_allocator="tcmalloc"'

gclient runhooks

ninja -C out/Debug remoting_apk

And I get the compilation error shown below.

Many thanks,

Lambros

[Will Harris]

Sorry, I can't be much more help on DMP but there is a mailing list for it here:

https://groups.google.com/a/chromium.org/forum/#!forum/dmprof

and several posts on there about Android DMP support.

HTH,

Will

[Lambros Lambrou]

Nearly there!

I've got the app to build with tcmalloc (with the help of a patch from the dmprof forum). And the app is trying to create dumps (I added a call to base::debug::StartProfiling("chromoting") during startup).

The problem is that the app is denied write access:

V/gperftools(10970): Dumping heap profile to /data/local/tmp/heap/dumptest.10970.0064.heap (2 sec since the last dump)

E/gperftools(10970): Failed dumping heap profile to /data/local/tmp/heap/dumptest.10970.0064.heap

W/Thread-1703(10992): type=1400 audit(0.0:3440): avc: denied { write } for name="heap" dev="dm-0" ino=122165 scontext=u:r:untrusted_app:s0 tcontext=u:object_r:shell_data_file:s0 tclass=dir

I have root on the device (it's a userdebug image). Could someone please tell me how I can run my APK such that it can write the dumps?

[Lambros Lambrou]

I just wanted to close off this thread in case anyone else is trying to do the same thing.

I was finally able to track down the memory-leak in the Chromoting app - the video decoder was not being destroyed after disconnecting from the host.

The write-denied problem was solved by putting the device into SELinux "permissive" mode: adb shell setenforce 0

Analyzing the dumps was tricky at first, but was solved once I realized a couple of things:

For non-Chrome binaries, you have to use the "--alternative-dirs" option to dmprof. And you have to do that in every dmprof command, particularly the first, otherwise dmprof will cache some bogus data and use it in subsequent commands.

You can follow the examples given in the docs, but you have to adjust both sides of the mapping. You have to explore the files on your device, and the files in your local output directory, to figure out the correct mapping.

The other problem was that "policies" are not explained very well in the docs. It seems they are specific to debugging Chrome. Since I was debugging something that wasn't Chrome, I simply ignored the policies and just chose a random one (such as "policy.android.browser") whenever a dmprof command required me to specify one. I think the policies just control how the allocations are bucketed. The result was that all the allocations ended up in a single generic bucket, but that didn't matter to me.

[Nico Weber]

Can you update whatever wiki page you expected this to describe this with what you learned?