Build System Project - Update from the last 2 weeks

[David Burns]

Below is a highlight of all work the build peers have done in the last 2
weeks as part of their work to modernise the build infrastructure.

Since the last report[1] a large number of improvements have landed in
Mozilla Central.

We have landed some more build improvements that have brought down the
build times for Windows PGO to be more in line with Linux64 opt builds[2]
in automation. This is helping us edge closer to it taking 11 hours less to
release Firefox to Windows users[3]. These same changes should also improve
local build times in most configurations, most noticeably on Windows.

We have also started looking at how we can use a global compiler cache on
local builds and not just in automation. This will allow artifact-like
builds for those who are doing C++ development.

Tests are now installed incrementally so the build system only installs
what you need when you need it. This dramatically speeds the test part of
the development cycle locally. On top of this, there are investigations to
see if all-tests.json can be reduced in size. Initial investigations show
that we can do some simple optimizations and get some sizeable wins.

We are continuing to remove configure, m4 code, and Makefiles from
mozilla-central. As mentioned in the previous status email, this will allow
us to replace the build backend with a more performant tool.

David

[1]
https://groups.google.com/d/msg/mozilla.dev.platform/7kCM7aJ80rs/Mgq7jrmNBAAJ

[2]
https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-inbound,04b9f1fd5577b40a555696555084e68a4ed2c28f,1%5D&series=%5Bmozilla-inbound,c0018285639940579da345da71bb7131d372c41e,1%5D&series=%5Bmozilla-inbound,65e0ddb3dc085864cbee77ab034dead6323a1ce6,1%5D

[3] https://rail.merail.ca/posts/release-build-promotion-overview.html

[Nicolas B. Pierron]

Unrelated, Do we have news for clang blockers on Windows? In particular, I
am thinking about the various Sanitizers.

On 04/20/2016 11:00 AM, David Burns wrote:
> We have also started looking at how we can use a global compiler cache on
> local builds and not just in automation. This will allow artifact-like
> builds for those who are doing C++ development.

I am particularly scared about this topic for multiple reasons, including
that my system does not have a /lib directory. Is there a location where I
can learn more and contribute back to this?

--
Nicolas B. Pierron

[Gregory Szorc]

> On Apr 20, 2016, at 08:16, Nicolas B. Pierron <nicolas....@mozilla.com> wrote:
>
> Unrelated, Do we have news for clang blockers on Windows? In particular, I am thinking about the various Sanitizers.

We haven't really talked about Clang on Windows in our build meeting/plannings. That's not to say someone else hasn't been working on it. But I haven't seen much bug traffic indicating that's the case.

If you make a case for Clang on Windows improving developer productivity or improving stability, that's how you get something prioritized. Can you start a thread on dev-builds listing the benefits?

>
>> On 04/20/2016 11:00 AM, David Burns wrote:
>> We have also started looking at how we can use a global compiler cache on
>> local builds and not just in automation. This will allow artifact-like
>> builds for those who are doing C++ development.
>
> I am particularly scared about this topic for multiple reasons, including that my system does not have a /lib directory. Is there a location where I can learn more and contribute back to this?
>
> --
> Nicolas B. Pierron

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform

[Armen Zambrano G.]

Do we know what happen with the data points of the end of the graph?

https://treeherder.mozilla.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-inbound,04b9f1fd5577b40a555696555084e68a4ed2c28f,1%5D&series=%5Bmozilla-inbound,65e0ddb3dc085864cbee77ab034dead6323a1ce6,1%5D&series=%5Bmozilla-inbound,c0018285639940579da345da71bb7131d372c41e,1%5D&zoom=1460493349005.865,1461176501000,0,10000

If I read it correctly it feels that we're having this week less pgo
builds (2/day versus 9-10/day)

This link without opt makes it more obvious:
https://treeherder.mozilla.org/perf.html#/graphs?series=%5Bmozilla-inbound,c0018285639940579da345da71bb7131d372c41e,1%5D&series=%5Bmozilla-inbound,04b9f1fd5577b40a555696555084e68a4ed2c28f,0%5D&series=%5Bmozilla-inbound,65e0ddb3dc085864cbee77ab034dead6323a1ce6,1%5D

regards,
Armen

--
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca

[Christopher Manchester]

Hi Armen,

Thanks for noticing that, I think you found an issue with how we're
submitting data caused by something I landed last week. I filed
https://bugzilla.mozilla.org/show_bug.cgi?id=1266183 to track a fix.

Chris

On Wed, Apr 20, 2016 at 12:01 PM, Armen Zambrano G. <arm...@mozilla.com>
wrote:

[Bobby Holley]

I believe Ehsan was close to having Clang static analysis running on
Windows.

On Wed, Apr 20, 2016 at 8:44 AM, Gregory Szorc <gsz...@mozilla.com> wrote:

>
>
> > On Apr 20, 2016, at 08:16, Nicolas B. Pierron <
> nicolas....@mozilla.com> wrote:
> >

> > Unrelated, Do we have news for clang blockers on Windows? In
> particular, I am thinking about the various Sanitizers.
>

> We haven't really talked about Clang on Windows in our build
> meeting/plannings. That's not to say someone else hasn't been working on
> it. But I haven't seen much bug traffic indicating that's the case.
>
> If you make a case for Clang on Windows improving developer productivity
> or improving stability, that's how you get something prioritized. Can you
> start a thread on dev-builds listing the benefits?
>
> >

> >> On 04/20/2016 11:00 AM, David Burns wrote:
> >> We have also started looking at how we can use a global compiler cache
> on
> >> local builds and not just in automation. This will allow artifact-like
> >> builds for those who are doing C++ development.
> >
> > I am particularly scared about this topic for multiple reasons,
> including that my system does not have a /lib directory. Is there a
> location where I can learn more and contribute back to this?
> >
> > --
> > Nicolas B. Pierron

[Nicholas Nethercote]

On Thu, Apr 21, 2016 at 2:13 AM, Bobby Holley <bobby...@gmail.com> wrote:
> I believe Ehsan was close to having Clang static analysis running on
> Windows.

Nathan Froyd has also been involved, and should be included in any
discussion there.

Nick