Very slow lldb debug on macos

[Quyen Le]

Hi,

I'm not sure this is mac related issues or not but I am experiencing very slow Chromium debugging with lldb. A simple bt command takes minutes to finish for example (I feel like it could freeze forever). Not to mention if I use Visual Studo Code's C++ extension to debug, it will hang indefinitely and only forced restarting computer can solve it. 

[Erik Chen]

Loading symbols can be slow on first start. But I wouldn't expect slowness on every command. My first guess [with no other information to work on] is that your device has insufficient RAM to hold the contents of the decoded symbol files, resulting in thrashing of the compressor/swap.

On Mon, May 4, 2020 at 5:38 PM Quyen Le <le.ho...@gmail.com> wrote:

Hi,

I'm not sure this is mac related issues or not but I am experiencing very slow Chromium debugging with lldb. A simple bt command takes minutes to finish for example (I feel like it could freeze forever). Not to mention if I use Visual Studo Code's C++ extension to debug, it will hang indefinitely and only forced restarting computer can solve it. 

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/2312f725-99a7-444c-bd00-0b8c891be449%40chromium.org.

[Ken Russell]

Seeing this behavior as well. Simple calls in the debugger like SkPixmap::width() take minutes to complete against a Debug build of Chromium with basically default GN args.

lldb is in fact taking a huge amount of memory; >26 GB on a machine (2017 MacBook Pro) with 16 GB physical memory.

Is anyone else debugging successfully on a similar configuration? Reducing symbol_level globally probably won't work in this particular case, as I need to inspect Blink and Skia data structures.

Thanks,

-Ken

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAEYHnr0rHUvK3LtKCzkVQAB59ZH80B0CLZAsip0cCe_ciTYoUg%40mail.gmail.com.

[Erik Chen]

> >26 GB on a machine (2017 MacBook Pro) with 16 GB physical memory.

Ouch. :(

I haven't debugged on macOS recently, but it's possible that symbol_level=1 will yield smaller debug files, if you're currently using symbol_level=2 ?

[Ken Russell]

Before I rebuild - do you know whether that will allow good local variable inspection and/or calling methods against objects?

[Erik Chen]

> Before I rebuild - do you know whether that will allow good local variable inspection and/or calling methods against objects?

I believe you will lose both.

[Ken Russell]

Unfortunately for the debugging task I had at hand, that would have defeated the purpose.

Is there any way to make progress on this issue? Should I file a bug on crbug.com?

I think attempting to selectively reduce Chromium's debug symbol size is likely to break debugging in unpredictable ways, so maybe the best path forward would be to file upstream bugs against lldb?

Do you think Chromium folks who work on tools/ would be able to help with that?

-Ken

[Erik Chen]

+Nico Weber will know best. I don't know if this is an architectural limitation of lldb [e.g. needing to preload all symbols] or just a missing optimization. Likewise, I don't know if chromium's massive symbol sizes is WAI or if we can do reasonable optimizations passes there.

[Ken Russell]

Could I get a little feedback on this issue? It's very easy to reproduce. Just launch a debug Chromium build on macOS, attach to its GPU process, and execute:

b renderergl_utils.cpp:1867

(trying to hit the first line of executable code in SupportsNativeRendering)

Even without swapping, this breakpoint takes minutes to set. It's really unacceptable performance for a critical tool.

How can bugs be filed effectively against lldb?

Thanks,

-Ken

[Quyen Le]

I gave up on debugging chromium. Currently, if I want to debug ANGLE only then I will just use a release version of chromium, then build a debug version of ANGLE, copy its libEGL.dylib & libGLESv2.dylib to chromium's libraries folder. Later I can attach to chromium gpu process and debug ANGLE library smoothly. I won't be able to see the chromium's caller source code though.

Just wondering if increasing RAM to 32 GB would solve the issue. It is still absurd though.

[Erik Chen]

+Elly Fong-Jones : does the Mac team have a way of dealing with this right now?

[Ken Russell]

I hope that this is just a bug that can be fixed in lldb. Today while attempting to print a variable in ANGLE ("nativeInfo", here:

https://source.chromium.org/chromium/chromium/src/+/master:third_party/angle/src/libANGLE/renderer/gl/renderergl_utils.cpp;l=1899?q=SupportsNativeRendering

) the debugger hung irrevocably.

Any further help / input would be greatly appreciated. I realize Chromium's a huge project and generates massive amounts of debug information, but a functioning debugger is an absolutely essential tool.

On Wed, Oct 14, 2020 at 9:50 AM Elly Fong-Jones <elly...@google.com> wrote:

lldb should not be that slow or require tons of RAM to use, no.

rsesek@ (cced) might remember this problem - I have a distant memory that it has come up before.

-- elly

[Aleks Totic]

Linux datapoint, gdb memory usage is comparable to lldb. It loads in
10s on my linux 96GB ram box.

PID VIRT RES %CPU %MEM TIME+ COMMAND
2673342 24.8g 21.8g 0.0 11.6 17:11.20 gdb

The new 16" pros support up to 64G RAM. And the desktop mac pro would
be comparable to my linux workstation.

Aleks

[K. Moon]

Presumably this is the copy of lldb that comes with macOS, so this would have to be something Apple or someone upstream fixes? Other than efforts to just not have so much debug information, anyway...

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
    http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMdyzDuDzCss7jWR_i1V_4o0qLc076iEeidNgXNsackF9eMgvg%40mail.gmail.com.

[Ken Russell]

Good point - yes, this is the lldb in /usr/bin:

$ lldb --version
lldb-1103.0.22.10
Apple Swift version 5.2.4 (swiftlang-1103.0.32.9 clang-1103.0.32.53)

If a newer version of lldb would solve this performance problem, and if it were possible for Chromium's presubmit hooks to download that version along with the clang binary, that would be fantastic.

[Quyen Le]

I remember 6 years ago, my windows laptop with only 8GB can debug chromium fine using visual studio. Did chromium project become that big in these years that 16 GB ram is not even enough to debug it?

I understand this is not important, but there might be bloating issues somewhere.