Proposal: Use -fdebug-prefix-map by default for goma on Linux build

[Takuto Ikuta]

Hi chromium devs,

If you are using goma on linux for chromium dev, please continue to read.

TL;DR

I'm considering to enable strip_absolute_paths_from_debug_symbols config by default for better cache hit rate of goma build.

This will give you x2 or more faster build time by sharing goma's backend cache between bots and other goma users more widely.

But this replaces abs path to relative path of source files in objs used by debugger.

Please add following script to your ~/.gdbinit if you do source level debugging in gdb.

define hook-run  python import os for i in gdb.objfiles():  # Add source dir relative to build dir.  compile_dir = os.path.dirname(i.filename)  gdb.execute("dir %s" % compile_dir)  end end

If you want to see what happens beforehand, consider to use args.gn from buildbot. You can see buildbot's args.gn from `generate_build_files (with patch)` step.

Detailed description is here.

https://docs.google.com/document/d/1OX4jY_bOCeNK7PNjVRuBQE9s6BQKS8XRNWGK8FEyh-E/edit?usp=sharing

If you have some concern/feedback, please feel free to reply.

Thanks, Takuto

[Matthew Larionov]

Hello, Takuto!

In attached doc you mention that compile time drops from 4 minutes to 1.5. But when I try to find[1] these results on waterfall I'm able to see that only 6k targets get compiled in 1.6 minutes, but not all of them. Are the times you're talking about are for incremental builds? If not, is there a way to find ninja traces for full builds (from scratch) that complete in 1.5 minutes?

1. https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Linux%20Builder/102550

[Takuto Ikuta]

Hi Matthew,

The bot runs compile with -j500. But when we build chrome in local machine, we use -j2000 with goma.

In such build we can build chrome around 1.5 minutes when almost all of compile results come from cache.

I attached the trace of my local build (1m39s).

2018年5月17日(木) 18:45 Matthew Larionov <matth...@gmail.com>:

--

Takuto Ikuta

Software Engineer in Tokyo

Chrome Infrastructure (goma team)

[Takuto Ikuta]

I landed the CL.

Please follow this instruction if you want to do source level debugging on gdb. 

2018年5月18日(金) 13:49 Takuto Ikuta <tik...@google.com>:

[Aleks Totic]

Just in case you missed this announcement. 

IF YOU DO NOT DO THIS, your gdb WILL NOT FIND ANY OF YOUR SOURCE FILES!

> Please follow this instruction if you want to do source level debugging on gdb. 

I missed the announcement, and spend a little bit of time figuring this out.

Happy debugging,

Aleks

[Christian Biesinger]

I think something else is wrong, even with sourcing that file.

I get a lot of errors like:

warning: Could not find DWO CU
obj/third_party/blink/renderer/core/clipboard/clipboard/data_transfer.dwo(0x4b8d2c95891e5e46)
referenced by CU at offset 0x1c24c6 [in module
/src/chromium/src/out/Debuggn/./libblink_core.so]

and basic features don't work:

867 LayoutObject* layout_object = frame->ContentLayoutObject();
(rr)
868 if (!layout_object || !layout_object->IsBox())
(rr) p layout_object->ShowLayoutTreeForThis()
No symbol "layout_object" in current context.

I think I'll have to disable this for now...

Christian

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
> ---
> You received this message because you are subscribed to the Google Groups
"Chromium-dev" group.
> To view this discussion on the web visit
https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAMdyzDstWottMYLoV5F0it65uPV5Giwo9P7Y0W8uknWAjTcsLg%40mail.gmail.com
.

[Aleks Totic]

Everything works for me if I start gdb inside out/Debug directory.

Aleks

[Christian Biesinger]

Ah... maybe I need to get into the habit of doing that :(

Christian

[Jakob Kummerow]

I was under the impression that the stuff in gdbinit was precisely intended to avoid the need to run GDB from the out/Debug directory? I can confirm that that isn't working though.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CAPTJ0XFhqmUkVZQurabsMObUC7zN%3DR2jmSeaOnrTydJyvwV82w%40mail.gmail.com.

[Takuto Ikuta]

2018年5月22日(火) 7:36 Jakob Kummerow <jkum...@chromium.org>:

I was under the impression that the stuff in gdbinit was precisely intended to avoid the need to run GDB from the out/Debug directory? I can confirm that that isn't working though.

Yes, can I ask your debugging workflow?

gdbinit I wrote supposed that binary is always run from gdb.

I miss other workflows.

 

What happens if you use following gdbinit?

python import os def load_relative_path(): for i in gdb.objfiles(): # Add source dir relative to build dir. compile_dir = os.path.dirname(i.filename) gdb.execute("dir %s" % compile_dir) load_relative_path() end define hook-run python load_relative_path() end end define hook-break python load_relative_path() end end define hook-attach python load_relative_path() end end

[Jakob Kummerow]

My typical workflow is to start binaries with:

gdb -ex run -args out/path/binary --parameters --for --binary

I do see symbols in GDB's source view, so the custom gdbinit is doing something, but I can't print any local variables. Like Christian reported, all I get is:

(gdb) p isolate
No symbol "isolate" in current context.

The longer gdbinit you suggested in your last email doesn't help. Adding a hook-stop doesn't help either. Changing the cwd before starting GDB helps:

(cd out/path; gdb -ex run -args ./binary --parameters --for --binary)

but that's inconvenient e.g. when loading files from the current directory, because I need a bunch of ../../ then.

[Takuto Ikuta]

I could not reproduce No symbol behavior.

I can see local variables in gdb like below.

tikuta@tikuta:~/chromium/src$ gdb out/Release/base_unittests 

GNU gdb (GDB) 7.9-gg19

Copyright (C) 2015 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software: you are free to change and redistribute it.

There is NO WARRANTY, to the extent permitted by law.  Type "show copying"

and "show warranty" for details.

This GDB was configured as "x86_64-grtev4-linux-gnu".

Type "show configuration" for configuration details.

<http://go/gdb-home FAQ: http://go/gdb-faq Email: c-toolchain-team IRC: gdb>

Find the GDB manual and other documentation resources online at:

<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".

Type "apropos word" to search for commands related to "word".

Reading symbols from out/Release/base_unittests...done.

Unable to determine compiler version.

Skipping loading of libstdc++ pretty-printers for now.

Non-google3 binary detected.

(gdb) b main

Breakpoint 1 at 0xc906ce: file ../../base/test/run_all_base_unittests.cc, line 11.

(gdb) r

Starting program: /usr/local/google/home/tikuta/chromium/src/out/Release/base_unittests 

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/ef/9474f02d7f7a3959b6dae0afe8e6acee1ac5ba.debug]

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/18/ad48fb80336dd6554ce9d61ecb3ccb7f6befa3.debug]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/45/2d8247a2b64710d413dfc84b6d7c8924aca427.debug]

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/7b/5f153d3f24f886129354bf9e6a32d01607e87a.debug]

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/cb/88f1fe155855433a969d01f052923b299701b8.debug]

Dwarf Error: wrong version in compilation unit header (is 0, should be 2, 3, or 4) [in module /usr/lib/debug/.build-id/76/1d39eb84070706076d19230afb6301781637cf.debug]

Breakpoint 1, main (argc=1, argv=0x7fffffffdba8) at ../../base/test/run_all_base_unittests.cc:11

11   base::TestSuite test_suite(argc, argv);

(gdb) list

6 #include "base/test/launcher/unit_test_launcher.h"

7 #include "base/test/test_suite.h"

8 #include "build/build_config.h"

9

10 int main(int argc, char** argv) {

11   base::TestSuite test_suite(argc, argv);

12   return base::LaunchUnitTests(

13       argc, argv,

14       base::Bind(&base::TestSuite::Run, base::Unretained(&test_suite)));

15 }

(gdb) n

14       base::Bind(&base::TestSuite::Run, base::Unretained(&test_suite)));

(gdb) p test_suite

$1 = {_vptr$TestSuite = 0xd54d78 <vtable for base::TestSuite+16>, at_exit_manager_ = {__ptr_ = {<std::__1::__compressed_pair_elem<base::AtExitManager*, 0, false>> = {

        __value_ = 0x1b961daa8a80}, <std::__1::__compressed_pair_elem<std::__1::default_delete<base::AtExitManager>, 1, true>> = {<std::__1::default_delete<base::AtExitManager>> = {<No data fields>}, <No data fields>}, <No data fields>}}, trace_to_file_ = {path_ = {static kSeparators = 0x7ffff7e81270 <base::FilePath::kSeparators> "/", static kSeparatorsLength = 2, static kCurrentDirectory = <same as static member of an already seen type>, 

      static kParentDirectory = <same as static member of an already seen type>, static kExtensionSeparator = 46 '.', path_ = {<std::__1::__basic_string_common<true>> = {<No data fields>}, static __short_mask = 1, 

        static __long_mask = 1, __r_ = {<std::__1::__compressed_pair_elem<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__rep, 0, false>> = {__value_ = {{__l = {__cap_ = 0, __size_ = 0, 

                  __data_ = 0x0}, __s = {{__size_ = 0 '\000', __lx = 0 '\000'}, __data_ = '\000' <repeats 22 times>}, __r = {__words = {0, 0, 

                    0}}}}}, <std::__1::__compressed_pair_elem<std::__1::allocator<char>, 1, true>> = {<std::__1::allocator<char>> = {<No data fields>}, <No data fields>}, <No data fields>}, static npos = 18446744073709551615}}, 

    started_ = false}, initialized_command_line_ = true, scoped_feature_list_ = {original_feature_list_ = {__ptr_ = {<std::__1::__compressed_pair_elem<base::FeatureList*, 0, false>> = {__value_ = 

    0x0}, <std::__1::__compressed_pair_elem<std::__1::default_delete<base::FeatureList>, 1, true>> = {<std::__1::default_delete<base::FeatureList>> = {<No data fields>}, <No data fields>}, <No data fields>}}, field_trial_override_ = {

      ptr_ = 0x0}, field_trial_list_ = {__ptr_ = {<std::__1::__compressed_pair_elem<base::FieldTrialList*, 0, false>> = {

          __value_ = 0x0}, <std::__1::__compressed_pair_elem<std::__1::default_delete<base::FieldTrialList>, 1, true>> = {<std::__1::default_delete<base::FieldTrialList>> = {<No data fields>}, <No data fields>}, <No data fields>}}}, 

  printer_ = 0x0, assert_handler_ = {__ptr_ = {<std::__1::__compressed_pair_elem<logging::ScopedLogAssertHandler*, 0, false>> = {

        __value_ = 0x0}, <std::__1::__compressed_pair_elem<std::__1::default_delete<logging::ScopedLogAssertHandler>, 1, true>> = {<std::__1::default_delete<logging::ScopedLogAssertHandler>> = {<No data fields>}, <No data fields>}, <No data fields>}}}

Can I ask concrete example to reproduce your debug workflow?

* what is your args.gn?

* what is your testing binary/function?

[Takuto Ikuta]

Sorry for inconvenience.

I made a CL to load dwo files correctly.

https://chromium-review.googlesource.com/c/chromium/src/+/1068785

If you can't load symbols or see strange behavior even with above gdb script, please let me know again.

Thanks, Takuto

2018年5月23日(水) 9:37 Takuto Ikuta <tik...@google.com>:

[les...@chromium.org]

Any reason to load symbols on run/attach/break rather than connecting a new_objfile event handler?

[Takuto Ikuta]

2018年5月23日(水) 20:32 <les...@chromium.org>:

Any reason to load symbols on run/attach/break rather than connecting a new_objfile event handler?

Just I didn't know such feature. And it looks suitable than hook.

Thanks

 

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/5f7174a9-9308-49e3-9173-8d0e56ab16d0%40chromium.org.

[Takuto Ikuta]

I landed gdbinit fix.

If you see debugging difficulty yet, please let me know again.

Thank you for your feedback!

2018年5月24日(木) 16:36 Takuto Ikuta <tik...@google.com>:

[Christian Biesinger]

Thank you! It seems to be working now.

Christian