ninja gn_all fails with GN build, w/ header issues in WebRTC being built under PNaCl..?

[Wez]

Hallo Chromium-dev!

I'm trying out a spiffy new Linux/Debug/GN build and thought I'd try the gn_all target, only to find that the build goes all 'splodey with what look like header issues somewhere in WebRTC, in code being built for [P]NaCl.

Is the gn_all target supposed to build...?

Thanks!

Wez

[Daniel Cheng]

I hit the same issue on Monday this week and gave up and switched back to gyp. It worked the previous week, but I haven't had time to bisect and find the guilty change yet.

Daniel

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Wez]

Excellent - thanks.  I had assumed I was Doing Something Wrong.

Perhaps if we can get it working again we could switch the _gn bots to actually build gn_all and protect ourselves from this...?

[Dirk Pranke]

The 'gn_all' target (and 'all') are supposed to build.

The fact that the bots are not currently doing this is an unfortunate side effect of some

confusion and bot-juggling as we flip Linux over from GYP to GN.

There were some issues w/ pNaCl + WebRTC this week, but a roll earlier today should (?) have 

fixed those things. I have another CL in flight that should fix another issue.

I am not aware of any other issues, but I haven't tried to build a Linux build yet today, either. I will do so now.

Otherwise, feel free to file a bug and assign it to me.

-- Dirk

[Dirk Pranke]

Okay, I hit one other issue w/ 'gn_all' that I just CQ'ed a fix for, but otherwise it linked for me.

'all', on the other hand, is still broken, and I probably won't get to that until Monday or possibly some time

over the weekend.

-- Dirk

[Wez]

Thanks Dirk.  Does this mean we can/should switch some/all of the GN bots to build gn_all?

I'm unable to gclient sync right now because of some new credential issue, so I can't try the build, but I'll try again tomorrow...

[Dirk Pranke]

On Fri, Nov 6, 2015 at 9:13 PM, Wez <w...@chromium.org> wrote:

Thanks Dirk.  Does this mean we can/should switch some/all of the GN bots to build gn_all?

Yes. We should always be able to build 'all'. 

It actually looks like we've lost debug coverage of 'all' in multiple configurations for both GYP and GN. I'm working on it with some of the infra folks and others.

-- Dirk

[Wez]

Oh no, that's no good!  Thanks for looking into it, Dirk. :)

[Daniel Cheng]

Sorry to bring this up again but...

I run 'ninja' with no explicit argument pretty frequently, and 2x in the last 24 hours, I've had my local build break. The changes that broke all targets were changes that landed successfully via the CQ. I notice the CQ bots are still passing a hardcoded set of targets to ninja: can we just have them run ninja without any explicit target argument?

Daniel

[Dirk Pranke]

Which platform did you see breakages on? I would normally expect 'all' to work and be kept

working by the bots, so I'd like to dig into failures to see what we're missing.

We do build 'all' on the waterfall on a number of configurations, but it's possible we don't have 

any CQ bots that do so. We should have one for Linux, but that got dropped from the CQ at some point

and hasn't yet been re-added, though it should be and we just haven't gotten around to it.

-- Dirk

[Christian Biesinger]

Please no... Trybots are slow enough as it is.

[Dirk Pranke]

On Linux, building all in the CQ should not affect cycle times (as I said before, we used to do this, and it was fine). 

We intentionally do not build 'all' on Mac or Win, though, because that would be much slower.

-- Dirk

[Daniel Cheng]

This is just a plain Linux GN build. I have a few things in gn args, but nothing particularly esoteric (use_goma=false enable_oilpan=true dcheck_always_on=true is_component_build=true).

cbiesinger@: the CQ would run a lot faster if we skipped all the tests too =)

Daniel

[Paweł Hajdan, Jr.]

+infra-dev

CQ is designed to mirror main waterfall. If something's not tested on main waterfall, I don't expect that not to be broken.

I consider requests for more things to be tested on main waterfall and CQ to be evaluated by chrome-eng-review, and then of course infra has to see if we already have capacity or would need to add some more.

Paweł

[Daniel Cheng]

Pawel: I'll ask chrome-eng-review@ about building more things. However, at the very least, can we make the ninja command the bots use to build easier to copy and paste?

Right now, it looks like this:

ninja -w dupbuild=err -C /b/build/slave/Linux_Builder/build/src/out/Release accessibility_unittests accessibility_unittests_run app_list_unittests app_list_unittests_run app_shell_unittests app_shell_unittests_run aura_unittests aura_unittests_run base_unittests base_unittests_run battor_agent battor_agent_unittests battor_agent_unittests_run blimp blimp_unittests blimp_unittests_run browser_tests browser_tests_run cacheinvalidation_unittests cacheinvalidation_unittests_run cast_unittests cast_unittests_run cc_unittests cc_unittests_run chrome chromedriver_unittests chromedriver_unittests_run chromium_swarm_tests components_browsertests components_browsertests_run components_unittests components_unittests_run compositor_unittests compositor_unittests_run content_browsertests content_browsertests_run content_unittests content_unittests_run crypto_unittests crypto_unittests_run dbus_unittests device_unittests device_unittests_run display_unittests display_unittests_run events_unittests events_unittests_run extensions_browsertests extensions_browsertests_run extensions_unittests extensions_unittests_run gcm_unit_tests gcm_unit_tests_run gfx_unittests gfx_unittests_run gin_unittests gin_unittests_run gn_unittests gn_unittests_run google_apis_unittests google_apis_unittests_run gpu_unittests gpu_unittests_run html_viewer_unittests interactive_ui_tests interactive_ui_tests_run ipc_mojo_unittests ipc_tests ipc_tests_run jingle_unittests jingle_unittests_run mandoline:all mash:all mash_unittests media_blink_unittests media_blink_unittests_run media_unittests media_unittests_run midi_unittests midi_unittests_run mojo_apptests mojo_common_unittests mojo_common_unittests_run mojo_js_integration_tests mojo_js_unittests mojo_public_application_unittests mojo_public_bindings_unittests mojo_public_bindings_unittests_run mojo_public_environment_unittests mojo_public_environment_unittests_run mojo_public_system_unittests mojo_public_system_unittests_run mojo_public_utility_unittests mojo_public_utility_unittests_run mojo_runner_host_unittests mojo_shell_unittests mojo_surfaces_lib_unittests mojo_system_unittests mojo_view_manager_lib_unittests nacl_loader_unittests nacl_loader_unittests_run net_unittests net_unittests_run ppapi_unittests printing_unittests printing_unittests_run remoting_all remoting_unittests remoting_unittests_run resource_provider_unittests sandbox_linux_unittests sandbox_linux_unittests_run skia_unittests skia_unittests_run sql_unittests sql_unittests_run sync_integration_tests sync_integration_tests_run sync_unit_tests sync_unit_tests_run telemetry_gpu_unittests_run telemetry_perf_unittests_run telemetry_unittests_run ui_base_unittests ui_base_unittests_run ui_touch_selection_unittests ui_touch_selection_unittests_run unit_tests unit_tests_run url_unittests url_unittests_run views_unittests window_manager_unittests wm_unittests wm_unittests_run -j50

But in order to actually build the same targets locally, I need to strip out all the targets with a _run suffix (I have no idea how to get ninja to generate targets for these). Since it's interspersed with all the regular targets, I need to paste it into an editor, search and replace out _run targets, and then run the result.

Daniel

[Dirk Pranke]

The _run targets should be automatically defined for you in a GN build (regardless of the gn args), and 

in a GYP build you need to set the test_isolation_mode=prepare in GYP_DEFINES.

In either of those cases, copy/pasting the command should just work. 

Of course, if you don't match the gn args or the gyp defines that the bot is setting, there no reason to

generally expect the compile line to work, because the list of targets defined may differ.

-- Dirk

[Paweł Hajdan, Jr.]

Daniel, do you have any specific suggestions about how to make the command line easier to copy and paste?

I guess it might have something to do with making it shorter. Please note we tell ninja to only compile targets affected by the patch, so the list is rather fine grained.

You can see the list of targets requested by recipe in the "analyze" step, e.g. for https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/174852 (cached logs https://chrome-logs-storage.appspot.com/browser/logs/tryserver.chromium.linux/linux_chromium_rel_ng/0174852/) see https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/174852/steps/analyze/logs/analyze_details:

  "additional_compile_targets": [
    "battor_agent", 
    "blimp", 
    "chromium_swarm_tests", 
    "mandoline:all", 
    "mash:all", 
    "remoting_all"
  ], 

And then "test_targets" contains explicit targets that are then tested by the bots.

Paweł

[Daniel Cheng]

It sounds like I was just missing test_isolation_mode=prepare from my GYP_DEFINES at one point, which gave me the impression it didn't work at all.

As you said, my other thought is just around how much stuff needs to be copy and pasted. The default view doesn't wrap long lines at a reasonable place, so I end up scrolling quite far sideways (I try not to include newlines in the copied text by default, since I sometimes have off-by-one errors on the ends, like only selecting inja instead of ninja). But I think I'm just going to write a script to make it easier to duplicate a bot build.

Daniel

[Paweł Hajdan, Jr.]

On Tue, Feb 2, 2016 at 3:49 PM, Daniel Cheng <dch...@chromium.org> wrote:

It sounds like I was just missing test_isolation_mode=prepare from my GYP_DEFINES at one point, which gave me the impression it didn't work at all.

I wonder why don't we make that mode the default.

 

As you said, my other thought is just around how much stuff needs to be copy and pasted. The default view doesn't wrap long lines at a reasonable place, so I end up scrolling quite far sideways (I try not to include newlines in the copied text by default, since I sometimes have off-by-one errors on the ends, like only selecting inja instead of ninja). But I think I'm just going to write a script to make it easier to duplicate a bot build.

If it's just about duplicating entire build, see https://build.chromium.org/p/tryserver.chromium.linux/builders/linux_chromium_rel_ng/builds/174852/steps/setup_build/logs/run_recipe