Please try experimental GYP replacement
378 views
Skip to first unread message
Brett Wilson
Oct 8, 2013, 2:05:54 PM10/8/13
to Chromium-dev
If you like waiting for gclient runhooks and nested conditions in GYP
files, you can stop reading now.
I have been working on an experimental GYP replacement creatively
called "Generate Ninja" (GN) that aims to be faster and simpler (and
somewhat inspired by the internal Google build tools). There are many
open questions and there is no decision about whether we should move
the Chrome build over to use it. However, it is stable enough for
other people to contribute.
I'm looking for volunteers to be involved to tell me when I'm crazy,
to find missing features and bugs, and to help convert the build. The
best way to get started is to find a 3rd party library with no
dependencies and try to write a build file. Note that the current goal
is not to convert as much of the build as possible, since maintaining
a parallel build requires a lot of upkeep.
Page on how to contribute:
https://code.google.com/p/chromium/wiki/gn
There is also a mailing list:
https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
It's currently self-hosting plus a smattering of other things that
don't change often (base, test support stuff, ipc, parts of net, some
third party stuff). The current partial build consists of 38 build
files and generates 127 targets.
For the above build the generation time is 32ms on Linux and 45ms on
Windows plus ~5000ms of running Grit :). koz volunteered to work on
the Grit problem (theoretically this can be ~0 so I count it
separately). In contrast all of Chrome has about 500 build files and
1000 outputs. If you extrapolate this will give a ~500ms generation
time (I'm guessing it will be a bit slower but I hope we can hit <1
second).
Brett
files, you can stop reading now.
I have been working on an experimental GYP replacement creatively
called "Generate Ninja" (GN) that aims to be faster and simpler (and
somewhat inspired by the internal Google build tools). There are many
open questions and there is no decision about whether we should move
the Chrome build over to use it. However, it is stable enough for
other people to contribute.
I'm looking for volunteers to be involved to tell me when I'm crazy,
to find missing features and bugs, and to help convert the build. The
best way to get started is to find a 3rd party library with no
dependencies and try to write a build file. Note that the current goal
is not to convert as much of the build as possible, since maintaining
a parallel build requires a lot of upkeep.
Page on how to contribute:
https://code.google.com/p/chromium/wiki/gn
There is also a mailing list:
https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
It's currently self-hosting plus a smattering of other things that
don't change often (base, test support stuff, ipc, parts of net, some
third party stuff). The current partial build consists of 38 build
files and generates 127 targets.
For the above build the generation time is 32ms on Linux and 45ms on
Windows plus ~5000ms of running Grit :). koz volunteered to work on
the Grit problem (theoretically this can be ~0 so I count it
separately). In contrast all of Chrome has about 500 build files and
1000 outputs. If you extrapolate this will give a ~500ms generation
time (I'm guessing it will be a bit slower but I hope we can hit <1
second).
Brett
Mostyn Bramley-Moore
Oct 8, 2013, 2:38:17 PM10/8/13
to chromi...@chromium.org
On 10/08/2013 08:05 PM, Brett Wilson wrote:
> If you like waiting for gclient runhooks and nested conditions in GYP
> files, you can stop reading now.
>
> I have been working on an experimental GYP replacement creatively
> called "Generate Ninja" (GN) that aims to be faster and simpler (and
> somewhat inspired by the internal Google build tools). There are many
> open questions and there is no decision about whether we should move
> the Chrome build over to use it. However, it is stable enough for
> other people to contribute.
>
>
> I'm looking for volunteers to be involved to tell me when I'm crazy,
> to find missing features and bugs, and to help convert the build. The
> best way to get started is to find a 3rd party library with no
> dependencies and try to write a build file. Note that the current goal
> is not to convert as much of the build as possible, since maintaining
> a parallel build requires a lot of upkeep.
>
> Page on how to contribute:
> https://code.google.com/p/chromium/wiki/gn
> There is also a mailing list:
> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
There doesn't seem to be a "Join group" button, is this restricted to
> If you like waiting for gclient runhooks and nested conditions in GYP
> files, you can stop reading now.
>
> I have been working on an experimental GYP replacement creatively
> called "Generate Ninja" (GN) that aims to be faster and simpler (and
> somewhat inspired by the internal Google build tools). There are many
> open questions and there is no decision about whether we should move
> the Chrome build over to use it. However, it is stable enough for
> other people to contribute.
>
>
> I'm looking for volunteers to be involved to tell me when I'm crazy,
> to find missing features and bugs, and to help convert the build. The
> best way to get started is to find a 3rd party library with no
> dependencies and try to write a build file. Note that the current goal
> is not to convert as much of the build as possible, since maintaining
> a parallel build requires a lot of upkeep.
>
> Page on how to contribute:
> https://code.google.com/p/chromium/wiki/gn
> There is also a mailing list:
> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
chromium.org accounts?
> It's currently self-hosting plus a smattering of other things that
> don't change often (base, test support stuff, ipc, parts of net, some
> third party stuff). The current partial build consists of 38 build
> files and generates 127 targets.
>
> For the above build the generation time is 32ms on Linux and 45ms on
> Windows plus ~5000ms of running Grit :). koz volunteered to work on
> the Grit problem (theoretically this can be ~0 so I count it
> separately). In contrast all of Chrome has about 500 build files and
> 1000 outputs. If you extrapolate this will give a ~500ms generation
> time (I'm guessing it will be a bit slower but I hope we can hit <1
> second).
-Mostyn.
--
Mostyn Bramley-Moore
mos...@opera.com
Brett Wilson
Oct 8, 2013, 3:56:54 PM10/8/13
to mos...@opera.com, Chromium-dev
On Tue, Oct 8, 2013 at 11:38 AM, Mostyn Bramley-Moore <mos...@opera.com> wrote:
> On 10/08/2013 08:05 PM, Brett Wilson wrote:
>>
>> If you like waiting for gclient runhooks and nested conditions in GYP
>> files, you can stop reading now.
>>
>> I have been working on an experimental GYP replacement creatively
>> called "Generate Ninja" (GN) that aims to be faster and simpler (and
>> somewhat inspired by the internal Google build tools). There are many
>> open questions and there is no decision about whether we should move
>> the Chrome build over to use it. However, it is stable enough for
>> other people to contribute.
>>
>>
>> I'm looking for volunteers to be involved to tell me when I'm crazy,
>> to find missing features and bugs, and to help convert the build. The
>> best way to get started is to find a 3rd party library with no
>> dependencies and try to write a build file. Note that the current goal
>> is not to convert as much of the build as possible, since maintaining
>> a parallel build requires a lot of upkeep.
>>
>> Page on how to contribute:
>> https://code.google.com/p/chromium/wiki/gn
>> There is also a mailing list:
>> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
>
>
> There doesn't seem to be a "Join group" button, is this restricted to
> chromium.org accounts?
Sorry, I fixed the permissions.
> On 10/08/2013 08:05 PM, Brett Wilson wrote:
>>
>> If you like waiting for gclient runhooks and nested conditions in GYP
>> files, you can stop reading now.
>>
>> I have been working on an experimental GYP replacement creatively
>> called "Generate Ninja" (GN) that aims to be faster and simpler (and
>> somewhat inspired by the internal Google build tools). There are many
>> open questions and there is no decision about whether we should move
>> the Chrome build over to use it. However, it is stable enough for
>> other people to contribute.
>>
>>
>> I'm looking for volunteers to be involved to tell me when I'm crazy,
>> to find missing features and bugs, and to help convert the build. The
>> best way to get started is to find a 3rd party library with no
>> dependencies and try to write a build file. Note that the current goal
>> is not to convert as much of the build as possible, since maintaining
>> a parallel build requires a lot of upkeep.
>>
>> Page on how to contribute:
>> https://code.google.com/p/chromium/wiki/gn
>> There is also a mailing list:
>> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev
>
>
> There doesn't seem to be a "Join group" button, is this restricted to
> chromium.org accounts?
Brett
Jiang Jiang
Oct 9, 2013, 4:27:16 AM10/9/13
to bre...@chromium.org, Nico Weber, Chromium-dev
Hi Brett,
On Tue, Oct 8, 2013 at 8:05 PM, Brett Wilson <bre...@chromium.org> wrote:
> It's currently self-hosting plus a smattering of other things that
> don't change often (base, test support stuff, ipc, parts of net, some
> third party stuff). The current partial build consists of 38 build
> files and generates 127 targets.
>
> For the above build the generation time is 32ms on Linux and 45ms on
> Windows plus ~5000ms of running Grit :). koz volunteered to work on
> the Grit problem (theoretically this can be ~0 so I count it
> separately). In contrast all of Chrome has about 500 build files and
> 1000 outputs. If you extrapolate this will give a ~500ms generation
> time (I'm guessing it will be a bit slower but I hope we can hit <1
> second).
I wonder why the generation time is so much longer on my Mac with a
fresh checkout of chromium tree.
First:
$ out/Release/gn
ERROR at //skia/BUILD.gn:1343:5: Assertion failed.
assert(false, "TODO(brettw) write this")
^-----
After I commented out this line,
$ out/Release/gn
Done. Wrote 126 targets from 36 files in 10070ms
Then I tried another run,
$ out/Release/gn
Done. Wrote 126 targets from 36 files in 9999ms
so it's more or less the same. If I run the gyp part,
$ out/Release/gn gyp
Running GYP...
Running GN...
Done. Wrote 126 targets from 36 files in 10627ms (GYP took 36291ms)
Then run the gn part only:
$ out/Release/gn gyp --no-gyp
Running GN...
Done. Wrote 126 targets from 36 files in 9926ms (GYP took 0ms)
Still takes more or less the same time (10s).
Also, debug build takes almost the same time:
$ out/Debug/gn
Done. Wrote 126 targets from 36 files in 10098ms
Am I missing something or is the Mac build not optimized at this point?
- Jiang
On Tue, Oct 8, 2013 at 8:05 PM, Brett Wilson <bre...@chromium.org> wrote:
> It's currently self-hosting plus a smattering of other things that
> don't change often (base, test support stuff, ipc, parts of net, some
> third party stuff). The current partial build consists of 38 build
> files and generates 127 targets.
>
> For the above build the generation time is 32ms on Linux and 45ms on
> Windows plus ~5000ms of running Grit :). koz volunteered to work on
> the Grit problem (theoretically this can be ~0 so I count it
> separately). In contrast all of Chrome has about 500 build files and
> 1000 outputs. If you extrapolate this will give a ~500ms generation
> time (I'm guessing it will be a bit slower but I hope we can hit <1
> second).
fresh checkout of chromium tree.
First:
$ out/Release/gn
ERROR at //skia/BUILD.gn:1343:5: Assertion failed.
assert(false, "TODO(brettw) write this")
^-----
After I commented out this line,
$ out/Release/gn
Done. Wrote 126 targets from 36 files in 10070ms
Then I tried another run,
$ out/Release/gn
Done. Wrote 126 targets from 36 files in 9999ms
so it's more or less the same. If I run the gyp part,
$ out/Release/gn gyp
Running GYP...
Running GN...
Done. Wrote 126 targets from 36 files in 10627ms (GYP took 36291ms)
Then run the gn part only:
$ out/Release/gn gyp --no-gyp
Running GN...
Done. Wrote 126 targets from 36 files in 9926ms (GYP took 0ms)
Still takes more or less the same time (10s).
Also, debug build takes almost the same time:
$ out/Debug/gn
Done. Wrote 126 targets from 36 files in 10098ms
Am I missing something or is the Mac build not optimized at this point?
- Jiang
Jiang Jiang
Oct 9, 2013, 4:36:48 AM10/9/13
to bre...@chromium.org, Nico Weber, Chromium-dev
On Wed, Oct 9, 2013 at 10:27 AM, Jiang Jiang <jia...@opera.com> wrote:
> Am I missing something or is the Mac build not optimized at this point?
Does it look like the grit part is taking twice as long on Mac? I
> Am I missing something or is the Mac build not optimized at this point?
tried profiling a gn run and it seemed to confirm it.
$ out/Release/gn --time
File parse times: (time in ms, name)
1.63 //skia/BUILD.gn
1.40 //net/BUILD.gn
0.97 //base/BUILD.gn
0.53 //third_party/icu/BUILD.gn
0.47 //build/config/compiler/BUILD.gn
0.47 //build/config/BUILDCONFIG.gn
0.47 //third_party/leveldatabase/BUILD.gn
0.38 //chrome/BUILD.gn
0.26 //crypto/BUILD.gn
0.24 //third_party/sfntly/BUILD.gn
0.19 //third_party/libxml/BUILD.gn
0.18 //tools/gn/BUILD.gn
0.16 //ipc/BUILD.gn
0.14 //tools/grit/grit_rule.gni
0.13 //testing/BUILD.gn
0.11 //third_party/libusb/BUILD.gn
0.11 //third_party/zlib/BUILD.gn
0.10 //net/third_party/nss/BUILD.gn
0.09 //build/toolchain/mac/BUILD.gn
0.08 //sdch/BUILD.gn
0.08 //build/toolchain/nacl/BUILD.gn
0.07 //third_party/libevent/BUILD.gn
0.07 //third_party/re2/BUILD.gn
0.07 //crypto/ssl/BUILD.gn
0.07 //third_party/snappy/BUILD.gn
0.07 //url/BUILD.gn
0.06 //build/config/BUILD.gn
0.04 //third_party/mach_override/BUILD.gn
0.04 //third_party/expat/BUILD.gn
0.03 //base/third_party/nspr/BUILD.gn
0.03 //base/allocator/BUILD.gn
0.03 //crypto/ssl/flags.gni
0.02 //BUILD.gn
0.02 //base/third_party/dynamic_annotations/BUILD.gn
0.02 //build/config/mac/BUILD.gn
0.02 //third_party/modp_b64/BUILD.gn
File execute times: (total time in ms, # executions, name)
9940.77 1 //chrome/BUILD.gn
433.41 1 //net/BUILD.gn
22.22 2 //base/BUILD.gn
13.50 1 //third_party/leveldatabase/BUILD.gn
10.99 1 //skia/BUILD.gn
7.50 1 //ipc/BUILD.gn
6.26 1 //third_party/libevent/BUILD.gn
6.23 1 //tools/gn/BUILD.gn
5.83 1 //third_party/icu/BUILD.gn
5.41 1 //testing/BUILD.gn
5.40 1 //crypto/BUILD.gn
5.25 2 //build/config/compiler/BUILD.gn
4.68 1 //third_party/zlib/BUILD.gn
3.68 1 //url/BUILD.gn
3.29 1 //third_party/snappy/BUILD.gn
2.91 1 //third_party/libusb/BUILD.gn
2.12 2 //base/third_party/dynamic_annotations/BUILD.gn
2.07 1 //third_party/sfntly/BUILD.gn
2.06 1 //third_party/re2/BUILD.gn
1.98 1 //sdch/BUILD.gn
1.94 1 //net/third_party/nss/BUILD.gn
1.80 2 //base/third_party/nspr/BUILD.gn
1.62 1 //third_party/expat/BUILD.gn
1.62 2 //base/allocator/BUILD.gn
1.50 2 //third_party/modp_b64/BUILD.gn
1.34 2 //build/config/BUILD.gn
1.21 1 //BUILD.gn
1.19 1 //build/toolchain/mac/BUILD.gn
1.03 1 //build/toolchain/nacl/BUILD.gn
0.68 1 //third_party/libxml/BUILD.gn
0.64 1 //third_party/mach_override/BUILD.gn
0.56 1 //crypto/ssl/BUILD.gn
0.56 2 //build/config/BUILDCONFIG.gn
0.21 1 //build/config/mac/BUILD.gn
Script execute times: (total time in ms, # executions, name)
10329.69 38 //tools/grit/grit_info.py
Done. Wrote 126 targets from 36 files in 9947ms
- Jiang
Brett Wilson
Oct 9, 2013, 11:14:13 AM10/9/13
to Jiang Jiang, Nico Weber, Chromium-dev
Yes, that time is almost all grit. You can run with --no-exec to
disable forking out to other processes (which bypasses grit).
Brett
disable forking out to other processes (which bypasses grit).
Brett
Paweł Hajdan, Jr.
Oct 9, 2013, 2:08:07 PM10/9/13
to Brett Wilson, k...@chromium.org, Jiang Jiang, Nico Weber, Chromium-dev
Wow, this is impressive. Once we optimize grit this is going to be super-fast!
Do you have more info about the plan to make grit faster? Maybe I could help somehow if I can find some spare cycles.
Paweł
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
Nico Weber
Oct 9, 2013, 2:41:24 PM10/9/13
to Paweł Hajdan, Jr., Jiang Jiang, Brett Wilson, k...@chromium.org, Chromium-dev
James Kozianski
Oct 10, 2013, 9:59:33 PM10/10/13
to Nico Weber, Paweł Hajdan, Jr., Jiang Jiang, Brett Wilson, Chromium-dev
A bit more info on this: currently we call grit at generate time to determine the inputs and outputs of a grit step so we can refer to them elsewhere, but this requires parsing every grd file twice at generation time. The CL above adds the option to grit to generate a depfile when it runs, which will remove the need to specify inputs at generation time, which should lead to a 50% speed up.
Outputs are trickier because they are referenced by name by other targets in the build. One solution could be to specify the expected outputs explicitly in the generation rule, but this might be unwieldy as there can be many outputs of a grd (eg: one for each supported language).
James
Brett Wilson
Dec 14, 2013, 2:04:32 AM12/14/13
to Chromium-dev
To follow-up on this build system stuff:
So far only one person has tried to check in build files. I don't know
how many people tried and failed that I don't know about, but I
haven't heard of any, and nobody has sent particularly detailed
feedback about any aspect of the design.
I asked one person today about it who responded "yeah I should set
that up". Did you know that there's no setup? You can use the pure GN
build of gn itself, base, and some random other stuff using your tree
today right now, and it won't conflict with what you're doing in your
other compiles. From the src directory:
gn
ninja -C out\Default gn
Please see the wiki on how to get started writing a build file:
https://code.google.com/p/chromium/wiki/gn
This project can not be successful with just me working on it. If you
like Gyp well enough and don't see the point, even that would be
valuable feedback.
If you want to help but aren't sure what to do, I'd pick a leaf target
in third_party and see if you can write a build file for it. If you
have any specific interests or knowledge (Chrome OS, cross-compiling,
weird test configurations, running some kind of IDL compiler) see if
it works or if you can get it working. Ask on the mailing list
https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev about
anything that's unclear.
Brett
So far only one person has tried to check in build files. I don't know
how many people tried and failed that I don't know about, but I
haven't heard of any, and nobody has sent particularly detailed
feedback about any aspect of the design.
I asked one person today about it who responded "yeah I should set
that up". Did you know that there's no setup? You can use the pure GN
build of gn itself, base, and some random other stuff using your tree
today right now, and it won't conflict with what you're doing in your
other compiles. From the src directory:
gn
ninja -C out\Default gn
Please see the wiki on how to get started writing a build file:
https://code.google.com/p/chromium/wiki/gn
This project can not be successful with just me working on it. If you
like Gyp well enough and don't see the point, even that would be
valuable feedback.
If you want to help but aren't sure what to do, I'd pick a leaf target
in third_party and see if you can write a build file for it. If you
have any specific interests or knowledge (Chrome OS, cross-compiling,
weird test configurations, running some kind of IDL compiler) see if
it works or if you can get it working. Ask on the mailing list
https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev about
anything that's unclear.
Brett
Thiago Farina
Dec 14, 2013, 8:03:34 AM12/14/13
to Brett Wilson, Chromium-dev
On Sat, Dec 14, 2013 at 5:04 AM, Brett Wilson <bre...@chromium.org> wrote:
> If you want to help but aren't sure what to do, I'd pick a leaf target
> in third_party and see if you can write a build file for it. If you
> have any specific interests or knowledge (Chrome OS, cross-compiling,
> weird test configurations, running some kind of IDL compiler) see if
> it works or if you can get it working. Ask on the mailing list
> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev about
> anything that's unclear.
>
I think the major problem is that most of gyp files have complex
> If you want to help but aren't sure what to do, I'd pick a leaf target
> in third_party and see if you can write a build file for it. If you
> have any specific interests or knowledge (Chrome OS, cross-compiling,
> weird test configurations, running some kind of IDL compiler) see if
> it works or if you can get it working. Ask on the mailing list
> https://groups.google.com/a/chromium.org/forum/#!forum/gn-dev about
> anything that's unclear.
>
conditions and depend already in a bunch of other targets and to
translate that to gn is a non-trivial task and most people are busy
working on fixing real bugs or implementing new features to stop and
write BUILD.gn for their projects.
In third_party I flac is one that it won't be too hard to convert to gn.
So far I have identified some targets in //components that are simple
enough to be converted to gn in a straightforward way and because for
their dependencies there are already gn equivalents.
* auto_login_parser (depends on base and net)
* navigation_metrics (depends on base and url)
* encryptor (depends on base and crypto)
* startup_metric_utils (depends on base)
* onc_component (depends on base)
* translate_common (depends on base and url)
* url_matcher (depends on base, dynamic_annotations, re2, url)
* wifi_component (depends on base, onc_component and libxml)
* wifi_test (depends on wifi_component, base and oc_component)
I will try to pick some targets of this list and convert to gn, if I
succeed I will send to you for review. startup_metric_utils and
onc_component are the easiest and the simplest, so I'll probably pick
them first.
--
Thiago
Reply all
Reply to author
Forward
0 new messages