Proposal: Deserialize build dependency between chrome and V8

65 views
Skip to first unread message

Takuto Ikuta

unread,
Apr 19, 2018, 4:11:43 AM4/19/18
to v8-...@googlegroups.com, Chromium-dev
Hi chromium folks and v8 foks.


When I build chrome, I see serialized dependency between chrome and V8 like below build trace.

This serialized dependency makes build time slow and slowness by this serialized dependency becomes relatively large in highly parallelizable build.
I want to reduce this serialized dependency by changing BUILD.gn.

More description of my proposal is here.
If you have something, please give me feedback.

Thanks, Takuto

Andrew Grieve

unread,
Apr 19, 2018, 4:49:28 AM4/19/18
to tik...@chromium.org, v8-...@googlegroups.com, Chromium-dev
v8 build times were recently brought up as an issue (https://bugs.chromium.org/p/v8/issues/detail?id=7629), so I think your timing here is great!

I'm not on v8, but read through your proposal and it sounds good to me!

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CALNjmMrbWafsnCbqYm%3DK9c18%2BsYzOmCqHW--rdvihZG-VVZS9g%40mail.gmail.com.

Takuto Ikuta

unread,
Apr 19, 2018, 5:33:23 AM4/19/18
to Jakob Gruber, v8-...@googlegroups.com, Chromium-dev


2018-04-19 17:56 GMT+09:00 Jakob Gruber <jgr...@google.com>:
I commented on the doc. Unfortunately, I don't think it will be possible to
turn mksnapshot into a data_dep due to embedded builtins (mksnapshot
generates embedded.cc, required to build libv8.so). 


Thank you for comment! Yeah, it is unfortunate that v8 generates embedded.cc.
I'll reconsider the way. Introducing new dependency type like link_deps to GN might be necessary.

 
But looking at the graph above, it seems like splitting up code-stub-assembler.cc may be
something to investigate?


I think so. Some files in v8 took very long compile time. I hope those files are separated to utilize parallelism.
 

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.




--
Takuto Ikuta
Software Engineer in Tokyo
Chrome Infrastructure (goma team)

Daniel Bratell

unread,
Apr 19, 2018, 8:49:04 AM4/19/18
to Jakob Gruber, 'Takuto Ikuta' via Chromium-dev, tik...@google.com, v8-...@googlegroups.com, Mostyn Bramley-Moore
This is something I also would like to see solved. The long phase in the middle can be even more pronounced in Chromium jumbo builds and from a high level it looks like it ought to be fixable.

I would look for a solution that does not require adding another dependency type though. Brett Wilson has  stated many times that the coarse dependency system in gn is intentional because there were so many subtle dependency bugs with gyp's more fine grained dependencies.

/Daniel
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Takuto Ikuta
Software Engineer in Tokyo
Chrome Infrastructure (goma team)
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-dev/CALNjmMoDA3hyO-0G_esiG%3Dn4k8FJu%3Dhfd_ucBG5hRusL1OAuxA%40mail.gmail.com.



--
/* Opera Software, Linköping, Sweden: CEST (UTC+2) */

Takuto Ikuta

unread,
Apr 19, 2018, 9:27:13 AM4/19/18
to Daniel Bratell, Jakob Gruber, 'Takuto Ikuta' via Chromium-dev, v8-...@googlegroups.com, Mostyn Bramley-Moore
2018-04-19 21:47 GMT+09:00 Daniel Bratell <bra...@opera.com>:
This is something I also would like to see solved. The long phase in the middle can be even more pronounced in Chromium jumbo builds and from a high level it looks like it ought to be fixable.

I would look for a solution that does not require adding another dependency type though. Brett Wilson has  stated many times that the coarse dependency system in gn is intentional because there were so many subtle dependency bugs with gyp's more fine grained dependencies.

Currently it looks there are no way to introduce dependency containing action target that is linked but does not block compiling for shared_library and static_library in GN.
If we don't want to introduce another dependency type, I think we can do by using source_set for target needs to be linked with libv8.so instead of component and move v8 dependency to final executable.
But I expect this will cause long link time.

I don't know gyp's era and how its fine grained dependencies were not good, but link only dependency will give us short build time.
link only dependency will also be applicable to yasm_assemble action in addition to code generator.
 
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Takuto Ikuta
Software Engineer in Tokyo
Chrome Infrastructure (goma team)
--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
---
You received this message because you are subscribed to the Google Groups "Chromium-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev+unsubscribe@chromium.org.

Marja Hölttä

unread,
Apr 19, 2018, 10:13:37 AM4/19/18
to v8-...@googlegroups.com, Daniel Bratell, Jakob Gruber, 'Takuto Ikuta' via Chromium-dev, Mostyn Bramley-Moore
Slightly tangential: Just by looking at the trace, it would already help if some builders got started with the big files (code-stub-assembler.cc and objects.cc) right at the beginning, instead of first doing some smaller tasks and only then getting started with the bigger ones. Is that doable?

(That is orthogonal to the problem that when mksnapshot is running, nothing else gets done.)


Google Germany GmbH

Erika-Mann-Straße 33

80636 München


Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg


Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

Takuto Ikuta

unread,
Apr 19, 2018, 11:41:57 PM4/19/18
to ma...@google.com, v8-...@googlegroups.com, Daniel Bratell, Jakob Gruber, 'Takuto Ikuta' via Chromium-dev, Mostyn Bramley-Moore
2018-04-19 23:12 GMT+09:00 Marja Hölttä <ma...@chromium.org>:
Slightly tangential: Just by looking at the trace, it would already help if some builders got started with the big files (code-stub-assembler.cc and objects.cc) right at the beginning, instead of first doing some smaller tasks and only then getting started with the bigger ones. Is that doable?

(That is orthogonal to the problem that when mksnapshot is running, nothing else gets done.)

Yes. There is some effort for better scheduling on ninja side.
 

Takuto Ikuta

unread,
Apr 20, 2018, 7:28:35 AM4/20/18
to Jakob Gruber, Marja Hölttä, v8-...@googlegroups.com, Daniel Bratell, 'Takuto Ikuta' via Chromium-dev, Mostyn Bramley-Moore
I sent proposal of link_deps to gn-dev. The proposal will fix the issue next time.


2018-04-20 15:39 GMT+09:00 Jakob Gruber <jgr...@chromium.org>:
On Fri, Apr 20, 2018 at 5:40 AM, Takuto Ikuta <tik...@google.com> wrote:
2018-04-19 23:12 GMT+09:00 Marja Hölttä <ma...@chromium.org>:
Slightly tangential: Just by looking at the trace, it would already help if some builders got started with the big files (code-stub-assembler.cc and objects.cc) right at the beginning, instead of first doing some smaller tasks and only then getting started with the bigger ones. Is that doable?

(That is orthogonal to the problem that when mksnapshot is running, nothing else gets done.)

Yes. There is some effort for better scheduling on ninja side.

That link is about explicit build orders - would it make sense to try and distribute compile work evenly over builders, taking e.g. file size as a metric to approximate compile time?

I want to do some investigation for this, but ninja's owner does not actively review PRs. But if I could sure that there is large improvement, I will ping them.


Jakob Gruber

Software Engineer

jgr...@google.com

Jakob Gruber

unread,
Apr 20, 2018, 10:38:33 AM4/20/18
to Takuto Ikuta, Marja Hölttä, v8-...@googlegroups.com, Daniel Bratell, 'Takuto Ikuta' via Chromium-dev, Mostyn Bramley-Moore
On Fri, Apr 20, 2018 at 5:40 AM, Takuto Ikuta <tik...@google.com> wrote:
2018-04-19 23:12 GMT+09:00 Marja Hölttä <ma...@chromium.org>:
Slightly tangential: Just by looking at the trace, it would already help if some builders got started with the big files (code-stub-assembler.cc and objects.cc) right at the beginning, instead of first doing some smaller tasks and only then getting started with the bigger ones. Is that doable?

(That is orthogonal to the problem that when mksnapshot is running, nothing else gets done.)

Yes. There is some effort for better scheduling on ninja side.

That link is about explicit build orders - would it make sense to try and distribute compile work evenly over builders, taking e.g. file size as a metric to approximate compile time?
 

Jakob Gruber

Software Engineer

jgr...@google.com

Dirk Pranke

unread,
Apr 20, 2018, 3:44:02 PM4/20/18
to Takuto Ikuta, Jakob Gruber, Marja Hölttä, v8-...@googlegroups.com, Daniel Bratell, 'Takuto Ikuta' via Chromium-dev, Mostyn Bramley-Moore, Nico Weber
On Fri, Apr 20, 2018 at 4:26 AM, Takuto Ikuta <tik...@chromium.org> wrote:
I sent proposal of link_deps to gn-dev. The proposal will fix the issue next time.


2018-04-20 15:39 GMT+09:00 Jakob Gruber <jgr...@chromium.org>:
On Fri, Apr 20, 2018 at 5:40 AM, Takuto Ikuta <tik...@google.com> wrote:
2018-04-19 23:12 GMT+09:00 Marja Hölttä <ma...@chromium.org>:
Slightly tangential: Just by looking at the trace, it would already help if some builders got started with the big files (code-stub-assembler.cc and objects.cc) right at the beginning, instead of first doing some smaller tasks and only then getting started with the bigger ones. Is that doable?

(That is orthogonal to the problem that when mksnapshot is running, nothing else gets done.)

Yes. There is some effort for better scheduling on ninja side.

That link is about explicit build orders - would it make sense to try and distribute compile work evenly over builders, taking e.g. file size as a metric to approximate compile time?

I want to do some investigation for this, but ninja's owner does not actively review PRs. But if I could sure that there is large improvement, I will ping them.

+thakis@

The owner has been actively reviewing PRs of late, so don't give up hope completely :). However, let's keep that separate from the proposal to reduce the serialization, which we can discuss in the GN thread that you've created.

-- Dirk

Reply all
Reply to author
Forward
0 new messages