GYP->GN Migration update, August 2015

[Dirk Pranke]

Hi all!

We're long-overdue for another update on the GYP->GN migration :)

The default Linux configurations are believed to be more-or-less-complete (barring some tests failing for NaCl, and some isolate/test target work for a couple of GPU targets). We're on the verge of flipping the main Linux builders over to use GN by default (https://codereview.chromium.org/1291843002/).

*As of now, we do not plan to continue building GYP on platforms where GN is the default* (on the bots, that is). 

This means that building the Linux platform in the default configs w/ GYP will be deprecated. However, there are a number of other Linux variants (like Linux ASAN) that aren't complete yet and that will be switched as separate steps. 

When a configuration has been switched over, GYP patches for that configuration are Best Effort (like maintaining the disable_nacl flag which people use but isn't "officially supported"). In practice, since most of the Linux variants are basically the full build with a couple extra flags, most Linux GYP changes will continue to be required until all of the Linux bots have been converted. We will probably need to keep things building through a full release cycle as well before we can really let things break.

If you have build flags that you use in development that aren't tested on the bots, now is a good time to make sure that it's implemented and works in GN, and to implement it or fix it.

I will be updating the default "getting started" docs on dev.chromium.org to make sure the build instructions tell people to use GN instead of GYP. 

The GN quick start guide is: https://chromium.googlesource.com/chromium/src/tools/gn/+/HEAD/docs/quick_start.md and the GYP conversion cookbook is: https://chromium.googlesource.com/chromium/src/tools/gn/+/HEAD/docs/cookbook.md

The very quick start is:

  % gn gen out/mybuild  # you only need to ever run this once per checkout

  % ninja -C out/mybuild chrome

I will also send out another note when we do actually flip the defaults with more detailed instructions.

We are actively closing in on completeness for the Win, CrOS, and Android ports as well, though there are still a number of bugs and small features left on each platform. Developers should be able to use GN for their day-to-day work on those platforms with no issues. 

We have started to make real progress on the Mac and iOS platforms as well, but they are still a ways off from being ready for day-to-day use.

We all recognize that maintaining two build systems is a hassle. Ideally I'd like to be done with all of this by the end of the year, but I'm not sure if we'll hit that yet. The more volunteers we get to get us over the line, the better off we'll all be! If you'd like to help, drop me a line.

Feedback welcome!

-- Dirk

[Nico Weber]

I'd recommend switching peripheral linux bots (asan and whatnot) first before switching the main configs used by everyone. That gives you the opportunity to work out some kinks without impacting everyone.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Dirk Pranke]

Duly noted :).

-- Dirk

[John Abd-El-Malek]

Very exciting!

On Wed, Aug 12, 2015 at 3:55 PM, Dirk Pranke <dpr...@chromium.org> wrote:

[Paweł Hajdan, Jr.]

That's great progress! Excited about the update.

Just checking - there are some questions important for me:

1. What are your plans for the official builds?

2. Could you take a look at debugallocation regression https://code.google.com/p/chromium/issues/detail?id=520894 ? Could you confirm whether you consider it blocking the actual switch?

Paweł

[Dirk Pranke]

On Fri, Aug 14, 2015 at 3:41 AM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

That's great progress! Excited about the update.

Just checking - there are some questions important for me:

1. What are your plans for the official builds?

I plan to convert them as we need to, just like everything else. I'm not sure if you had a more specific question? We'll have to work out the exact timing w/ the release managers.

 

2. Could you take a look at debugallocation regression https://code.google.com/p/chromium/issues/detail?id=520894 ? Could you confirm whether you consider it blocking the actual switch?

I don't know if that would block the actual switch of the bots, but I would expect that to be fixed before we consider Linux done.

I fully expect to switch the main configurations over knowing that there will still be minor bugs and tasks left; the main criteria we're using is that all the tests on the bots should pass and that all of the common targets and flags have been converted.

-- Dirk

[Scott Violet]

I just happened across a handful of tests that weren't being with the
gn build ( https://codereview.chromium.org/1296623003/ ). Have we
tried comparing the list of files built with gyp to that of gn to make
sure we aren't missing other tests?

-Scott

On Wed, Aug 12, 2015 at 3:55 PM, Dirk Pranke <dpr...@chromium.org> wrote:

[John Abd-El-Malek]

We should compare the output of a try (or buildbot) run of both gyp/gn to ensure that all the tests are the same. Checking .cc files won't catch the case where different defines disable tests.

[Brett Wilson]

I've checked such things in the past but it tends to bitrot. I'll do so again. Hopefully moving the main builders on GN will reduce the chances of this happening.

Brett

On Fri, Aug 14, 2015 at 9:16 AM, Scott Violet <s...@chromium.org> wrote:

[Steve Kobes]

My usual workflow involves "gclient runhooks", which seems to be calling GYP behind the scenes.  Will that be switched over at some point?  Is there some way of telling it to use GN instead?

On Wed, Aug 12, 2015 at 3:55 PM, Dirk Pranke <dpr...@chromium.org> wrote:

[Scott Violet]

GYP_CHROMIUM_NO_ACTION=1
But be careful. I ran into an issue today where because I have
disable_nacl=1 in my GYP_DEFINES, but gn uses a separate args files
gclient sync wasn't pulling down the nacl bits and my build was
failing. Setting enable_nacl=false in my gn args fixed the issue.
Nuking my GYP_DEFINES would have also fixed it.

-Scott

[Dirk Pranke]

One of the main differences between GYP and GN is that Ninja is able to re-run GN to regenerate the build files when it needs to,

and so we don't need to run an equivalent of gyp_chromium on every sync / runhooks invocation.

You do need to run GN at least once, however, to configure the initial build, so it makes more sense to think of this as running GN right after you run 'fetch chromium'.

You can (today) disable GYP by setting GYP_CHROMIUM_NO_ACTION=1 in your environment. You should be able to set this in chromium.gyp_env, but it looks like that doesn't work at the moment (a bug to be fixed).

If you wanted to re-create the default environment one gets w/ GYP if you have no GYP_DEFINES set, you could do:

% export GYP_CHROMIUM_NO_ACTION=1

% fetch chromium

% gn gen //out/Debug  # GN builds are debug static builds by default

% gn gen //out/Release --args='is_debug=false'

If you normally set various GYP_DEFINEs, look at //tools/mb/mb_config.pyl or 

https://chromium.googlesource.com/chromium/src/tools/gn/+/HEAD/docs/cookbook.md#Variable-mappings

for how to map a given GYP define to the GN equivalent.

As Scott says, you do need to be a little careful how you do things or you can shoot yourself in the foot if you try to stray off the beaten path with your defines. Any suggestions people might have for how to make this easier are welcome.

I will be updating the docs for all of this as we actually start to flip things over.

-- Dirk

On Fri, Aug 14, 2015 at 1:55 PM, Steve Kobes <sko...@chromium.org> wrote:

[Paweł Hajdan, Jr.]

On Fri, Aug 14, 2015 at 5:15 PM, Dirk Pranke <dpr...@chromium.org> wrote:

2. Could you take a look at debugallocation regression https://code.google.com/p/chromium/issues/detail?id=520894 ? Could you confirm whether you consider it blocking the actual switch?

I don't know if that would block the actual switch of the bots, but I would expect that to be fixed before we consider Linux done.

I fully expect to switch the main configurations over knowing that there will still be minor bugs and tasks left; the main criteria we're using is that all the tests on the bots should pass and that all of the common targets and flags have been converted.

Please note it's much easier to prevent regressions than to chase them afterwards. Also, this is the current default for Debug config, so IMHO pretty major thing. Since debugallocation is stricter than "regular" allocator, I think you wouldn't detect any regression by looking for failing tests. It's really easy to miss if one doesn't pay close attention to it, which is why I'm trying to emphasize this.

Also, it shouldn't be hard to migrate this. I'd be happy to work on patches. Quick question: how do I check inside GN file whether mode is Release or Debug? The idea is to have a flag to override it, but the default would be on for Debug and off for Release.

Paweł

[Colin Blundell]

if (is_debug) {

  ...

}

 

Paweł

[Dana Jansens]

I tried to use GN today, so I went to https://chromium.googlesource.com/chromium/src/tools/gn/ .

The quick start link is a 404: https://chromium.googlesource.com/chromium/src/tools/gn/docs/quick_start.md

As is the FAQ: https://chromium.googlesource.com/chromium/src/tools/gn/docs/faq.md

Well.. all the links at the bottom are 404s I guess.

[Dana Jansens]

On Mon, Aug 17, 2015 at 10:33 AM, Khannan <k...@chromium.org> wrote:

This is the link you are looking for, https://chromium.googlesource.com/chromium/src/+/master/tools/gn/docs/

Thanks. FWIW https://chromium.googlesource.com/chromium/src/tools/gn/ is the first result you get when searching google for "chromium gn".

 

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Brett Wilson]

Thanks, we recently moved this stuff from the wiki. I'll be sure to do a full pass through the docs soon.

Brett

[Sunny Sachanandani]

Those URLs don't have the revision number in them. Try https://chromium.googlesource.com/chromium/src/+/master/tools/gn/docs/quick_start.md.

[Dirk Pranke]

I think this is actually a problem in the Gitiles' markdown rendering; it's attempting to find the default 'README.md' and render it when you do a directory listing, but that doesn't really work correctly for relative URLs.

I'll file a bug w/ them and follow up.

Relatively soon, hopefully, we'll have a better top-level doc site for GN and web search will return better URLs, and I'll see if there are other SEO-type things we can do.

[Dana Jansens]

This is probably me shooting myself in said feet with fancy settings in GN, but I don't know how to resolve this. When I build I get this:

[1208/19860] ACTION //chrome/common:mojo_bindings_dart(//build/toolchain/linux:clang_x64)

FAILED: python ../../third_party/mojo/src/mojo/public/tools/gn/zip.py --base-dir=gen/dart-gen --zip-inputs=\[\] --output=obj/chrome/common/mojo_bindings.dartzip --link-inputs=\[\"gen/chrome/common/resource_usage_reporter.mojom.dart\"\]

Traceback (most recent call last):

  File "../../third_party/mojo/src/mojo/public/tools/gn/zip.py", line 66, in <module>

    sys.exit(main())

  File "../../third_party/mojo/src/mojo/public/tools/gn/zip.py", line 63, in main

    DoZip(inputs, link_inputs, zip_inputs, output, base_dir)

  File "../../third_party/mojo/src/mojo/public/tools/gn/zip.py", line 27, in DoZip

    outfile.write(realf, file_name)

  File "/usr/lib/python2.7/zipfile.py", line 1118, in write

    st = os.stat(filename)

OSError: [Errno 2] No such file or directory: '/usr/local/google/home/danakj/s/c/src/out_desktop/Release/gen/chrome/out_desktop/Release/gen/dart-gen/mojom/lib/resource_usage_reporter.mojom.dart'

I did a pull/gclient sync/runhooks before this. My GN args are:

is_component_build = true

is_debug = false

enable_nacl = false

use_goma = true

symbol_level = 1

is_chrome_branded = false

is_msan = false

is_lsan = false

is_tsan = false

enable_google_now = false

enable_remoting = false

is_clang = true

My GYP_DEFINES when I did runhooks were:

GYP_DEFINES= disable_nacl=1 python_ver=2.7 use_goma=1 ffmpeg_branding=Chrome clang=1 clang_use_chrome_plugins=1 component=shared_library

Any hints?

[Dana Jansens]

sunnyps@ solved it. Nuking my out dirs made things work instead of building on top of where I used to use gyp.

[Brett Wilson]

On Wed, Aug 12, 2015 at 3:55 PM, Dirk Pranke <dpr...@chromium.org> wrote:

Hi all!

We're long-overdue for another update on the GYP->GN migration :)

The default Linux configurations are believed to be more-or-less-complete (barring some tests failing for NaCl, and some isolate/test target work for a couple of GPU targets). We're on the verge of flipping the main Linux builders over to use GN by default (https://codereview.chromium.org/1291843002/).

The main "Linux Builder" bot on the waterfall has been flipped.

Brett

[John Abd-El-Malek]

Congrats! This is a huge milestone.

Can you guys share a public timeline for when Android/Windows/Mac/ChromeOS will be switched?

--

[John Abd-El-Malek]

and iOS of course :)

[Dirk Pranke]

The timeline is pretty rough, but I'm hoping to get Android, Windows, and ChromeOS in Q4 and Mac and iOS in Q1. 

-- Dirk

[Daniel Cheng]

I couldn't find the answer for this: is there an equivalent to gyp's release_extra_cflags? It's nice to have more easily debuggable stack trace with asan.

Daniel

[Brett Wilson]

Nobody has done that yet to my knowledge. My opinion is that there should be a flag to set optimizations for sanitizers that does the right thing rather than having people type random flags like this. 

Brett

[Bence Béky]

Hi,

I have a build dependency issue which seems to be specific to GN,
build working as intended with GYP. I filed https://crbug.com/541158,
but have no idea what are the best labels to tag or what other
preferred ways are there to report such issues. Please let me know if
there's a mailing list or issue tracker label appropriate for this (or
if I overlooked something trivial).

Thank you,

Bence

On Thu, Aug 13, 2015 at 12:55 AM, Dirk Pranke <dpr...@chromium.org> wrote:
> Hi all!
>
> We're long-overdue for another update on the GYP->GN migration :)
>
> The default Linux configurations are believed to be more-or-less-complete
> (barring some tests failing for NaCl, and some isolate/test target work for
> a couple of GPU targets). We're on the verge of flipping the main Linux
> builders over to use GN by default
> (https://codereview.chromium.org/1291843002/).
>

[Thiago Farina]

On Thu, Oct 8, 2015 at 11:18 AM, Bence Béky <b...@chromium.org> wrote:

Hi,

I have a build dependency issue which seems to be specific to GN,
build working as intended with GYP.  I filed https://crbug.com/541158,
but have no idea what are the best labels to tag or what other
preferred ways are there to report such issues.  Please let me know if
there's a mailing list or issue tracker label appropriate for this (or
if I overlooked something trivial).

AFAIK, we have these labels: Build=Tools-GN, Proj=GN-Migration. And gn-dev mailing list.

--

Thiago Farina

[Dirk Pranke]

For the record, Build=Tools-GN is for bugs in GN itself, and Proj=GN-Migration is for

issues related to the chromium build files and migrating things from GYP.

-- Dirk