Now measuring Firefox size per-commit. What else should we be tracking?

[William Lachance]

Hey, so as described here:

http://wrla.ch/blog/2015/11/perfherder-onward/

... I recently added tracking for Firefox installer size inside
Perfherder. This should let us track how bloated (or not) Firefox is on
our various supported platforms, on a per-commit basis:

https://treeherder.mozilla.org/perf.html#/graphs?series=[mozilla-inbound,4eb0cde5431ee9aeb5eb14512ddb3da6d4702cf0,1]&series=[mozilla-inbound,80cac7ef44b76864458627c574af1a18a425f338,1]&series=[mozilla-inbound,0060252bdfb7632df5877b7594b4d16f1b5ca4c9,1]

As I mentioned in the blog post, it's now *very* easy (maybe too easy?
heh) to submit "performance" (read: quantitative) data for any job
reporting to treeherder by outputting a line called "PERFHERDER_DATA" to
the log.

Is there anything we could be tracking as part of our build or test jobs
that we should be? Build times are one thing that immediately comes to
mind. Is there anything else?

In order to be a good candidate for measurement in this kind of system,
a metric should be:

1. Relatively deterministic.
2. Something people actually care about and are willing to act on, on a
per-commit basis. If you're only going to look at it once a quarter or
so, it doesn't need to be in Perfherder.

Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
add stuff incrementally, to make sure Perfherder can handle the load,
but I'd love to hear all your ideas.

Will

[Chris AtLee]

This is really great, thanks for adding support for this!

I'd like to see the size of the complete updates measured as well, in
addition to the installer sizes.

Do we have alerts for these set up yet?

Cheers,
Chris

On Wed, Nov 4, 2015 at 10:55 AM, William Lachance <wlac...@mozilla.com>
wrote:

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[William Lachance]

On 2015-11-04 11:15 AM, Chris AtLee wrote:
> This is really great, thanks for adding support for this!
>
> I'd like to see the size of the complete updates measured as well, in
> addition to the installer sizes.

Where are the updates generated? As part of the build process? If so,
can you self-serve your way to victory here, by adding something to this
log line?

http://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/mozharness/mozilla/building/buildbase.py#l1628

I'm happy to provide guidance where needed/helpful.

> Do we have alerts for these set up yet?

Not yet, alert/sheriffing support for Perfherder is being tracked in
this bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1150152

I'm hoping to have something working by the end of the quarter.

Will

[Andreas Tolfsen]

On 4 November 2015 at 15:55, William Lachance <wlac...@mozilla.com> wrote:
> Is there anything we could be tracking as part of our build or test jobs
> that we should be? Build times are one thing that immediately comes to mind.

Measuring build time would almost certainly be of big value, because
it has a tendency to slip from time to time.

When we start measuring this we should make sure to store the metrics
for different compiler/linker versions and options separately so we
can compare slowdown/speedups for different toolchains.

[William Lachance]

On 2015-11-04 10:55 AM, William Lachance wrote:
>
> 1. Relatively deterministic.
> 2. Something people actually care about and are willing to act on, on a
> per-commit basis. If you're only going to look at it once a quarter or
> so, it doesn't need to be in Perfherder.
>
> Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
> add stuff incrementally, to make sure Perfherder can handle the load,
> but I'd love to hear all your ideas.

Someone mentioned "test times" to me in private email.

I do think test times are worth tracking, but probably not in
Perfherder: test times might not be deterministic depending on where /
how they're running (which makes it difficult to automatically detect
regressions and sheriff them on a per commit basis) and regardless
there's too much data to really be manageable by Perfherder's intended
interface even if that problem were magically solved.

As a possible alternative, I believe Kyle Lahnakoski's ActiveData
project (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData)
already *does* track this type of data but last I heard he was looking
for more feedback on how to alert/present it to the platform community.
If you have any ideas on this, please let him know (he's CC'ed). :)

Will

[Michael Layzell]

I'd love it if we could get better stats for build times. Not only is it
handy for making sure they don't grow out of control, but we can also use
it to make sure that our static analysis doesn't add an unacceptable level
of overhead on compilation.

On Wed, Nov 4, 2015 at 12:10 PM, Andreas Tolfsen <a...@mozilla.com> wrote:

> On 4 November 2015 at 15:55, William Lachance <wlac...@mozilla.com>
> wrote:

> > Is there anything we could be tracking as part of our build or test jobs
> > that we should be? Build times are one thing that immediately comes to
> mind.
>

> Measuring build time would almost certainly be of big value, because
> it has a tendency to slip from time to time.
>
> When we start measuring this we should make sure to store the metrics
> for different compiler/linker versions and options separately so we
> can compare slowdown/speedups for different toolchains.

[L. David Baron]

On Wednesday 2015-11-04 12:46 -0500, William Lachance wrote:
> On 2015-11-04 10:55 AM, William Lachance wrote:
> >
> >1. Relatively deterministic.
> >2. Something people actually care about and are willing to act on, on a
> >per-commit basis. If you're only going to look at it once a quarter or
> >so, it doesn't need to be in Perfherder.
> >
> >Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
> >add stuff incrementally, to make sure Perfherder can handle the load,
> >but I'd love to hear all your ideas.
>
> Someone mentioned "test times" to me in private email.

That was me. (I didn't feel like sending a late-at-night
one-sentence email to the whole list, and figured there was a decent
chance that somebody else would mention it as well.)

I think they're worth tracking because we've had substantial
performance regressions (I think including as bad as a doubling in
times) that weren't caught quickly, and led to substantially worse
load on our testing infrastructure.

> I do think test times are worth tracking, but probably not in Perfherder:
> test times might not be deterministic depending on where / how they're
> running (which makes it difficult to automatically detect regressions and
> sheriff them on a per commit basis) and regardless there's too much data to
> really be manageable by Perfherder's intended interface even if that problem
> were magically solved.

It seems like if we're running the same tests on different sorts of
machines, we could track different perf numbers for the test run on
different machine classes.

We'd also want to measure the test time and *not* the time spent
downloading the build.

And we'd probably want to measure the total time across chunks so
that we don't count redistribution between chunks as a set of
regressions and improvements.

So that does make it a bit difficult, but it does seem doable.

> As a possible alternative, I believe Kyle Lahnakoski's ActiveData project
> (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData) already *does*
> track this type of data but last I heard he was looking for more feedback on
> how to alert/present it to the platform community. If you have any ideas on
> this, please let him know (he's CC'ed). :)

Perhaps, but I have no idea how to use it or what that would look
like. The wiki page explicitly says it's for automated clients and
not by humans; it would be useful to see an example of such an
automated client to have an idea of how this would work.

-David

--
𝄞 L. David Baron http://dbaron.org/ 𝄂
𝄢 Mozilla https://www.mozilla.org/ 𝄂
Before I built a wall I'd ask to know
What I was walling in or walling out,
And to whom I was like to give offense.
- Robert Frost, Mending Wall (1914)

[Sebastian Hengst]

- Build: Memory footprint
Background: 8 GB is our recommended minimum for building on Windows. New
contributors don't necessarily have high-end developer machines. By
noticing bumps we can evaluate the cost/benefit of these and protract
further bumps of the recommended minimum.

- Tests: Memory footprint during or at end of selected tests
To detect changes in memory allocation. We run the tests anyway.

Sebastian

-------- Original-Nachricht --------
Betreff: Now measuring Firefox size per-commit. What else should we be
tracking?
Von: William Lachance <wlac...@mozilla.com>
Datum: 2015-11-04 16:55

[Andrew Halberstadt]

ActiveData is just a very large data base. An automated-client would be
something that periodically runs a query, formats the data and plugs it
into a graph. Here's an example of a client side JS tool that runs a
query to determine which tests are enabled or skipped:
https://github.com/mozilla/test-informant/blob/master/js/report.js#L105

But I think trying to track runtime at the job (aka chunk) level is
going to be way too noisy. Something more useful might be (total suite
runtime)/(total number of tests) across all chunks.

-Andrew

[Kyle Lahnakoski]

Al,,

ActiveData[1] is warehouse of our test times, build times, and whatever
other properties or measures I can get my hands on. It has a primitive
query tool[2] that a human can use to send queries to the public service
[3]. The `unittest` table has results from all tests that emit
structured logs, and is the largest at 4 billion records.

Getting a feel for what is in ActiveData will take a little time. I
would first suggest a short list of some example documents. `{"from":
"unittest"}` will give you some example records. `{"from":"unittest",
"groupby":"build.platform"}` will give you the cardinality of the
platforms. There are also some starter docs on the query tool page [2].

Please forgive the low speed, it is scaled down for minimal cost.

Some other tables:
* {"from":"unittest.result.subtests"} a table of all nested unittest
documents: only has the failure messages, for now
* {"from":"jobs"} the buildbot job properties and timing (builds are not
properly categorized, he Treeherder code should be used for this),
* {"from":"jobs.action.timings"} each buildbot step, each mozharness
step, and their timing
* {"from":"orange_factor"} copy of the orangefactor data, plus `push_date`
* {"from":"perf"} all the Talos results, including replicates

I have only started to ingest the TaskCluster information.

[1] https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
[2] http://activedata.allizom.org/tools/query.html
[3] http://activedata.allizom.org/query

On 2015-11-05 09:18, L. David Baron wrote:
> On Wednesday 2015-11-04 12:46 -0500, William Lachance wrote:

>> On 2015-11-04 10:55 AM, William Lachance wrote:
>>> 1. Relatively deterministic.
>>> 2. Something people actually care about and are willing to act on, on a
>>> per-commit basis. If you're only going to look at it once a quarter or
>>> so, it doesn't need to be in Perfherder.
>>>
>>> Anyway, just thought I'd open the floor to brainstorming. I'd prefer to
>>> add stuff incrementally, to make sure Perfherder can handle the load,
>>> but I'd love to hear all your ideas.
>> Someone mentioned "test times" to me in private email.

> That was me. (I didn't feel like sending a late-at-night
> one-sentence email to the whole list, and figured there was a decent
> chance that somebody else would mention it as well.)
>
> I think they're worth tracking because we've had substantial
> performance regressions (I think including as bad as a doubling in
> times) that weren't caught quickly, and led to substantially worse
> load on our testing infrastructure.
>

>> I do think test times are worth tracking, but probably not in Perfherder:
>> test times might not be deterministic depending on where / how they're
>> running (which makes it difficult to automatically detect regressions and
>> sheriff them on a per commit basis) and regardless there's too much data to
>> really be manageable by Perfherder's intended interface even if that problem
>> were magically solved.

> It seems like if we're running the same tests on different sorts of
> machines, we could track different perf numbers for the test run on
> different machine classes.
>
> We'd also want to measure the test time and *not* the time spent
> downloading the build.
>
> And we'd probably want to measure the total time across chunks so
> that we don't count redistribution between chunks as a set of
> regressions and improvements.
>
> So that does make it a bit difficult, but it does seem doable.
>

>> As a possible alternative, I believe Kyle Lahnakoski's ActiveData project
>> (https://wiki.mozilla.org/Auto-tools/Projects/ActiveData) already *does*
>> track this type of data but last I heard he was looking for more feedback on
>> how to alert/present it to the platform community. If you have any ideas on
>> this, please let him know (he's CC'ed). :)

[Mark Finkle]

I also think measuring build times, and other build related stats, would be
useful. I'd like to see Mozilla capturing those stats for developer builds
though. I'm less interested in build times for the automation. That data is
already looked at by the automation team.

Build stats for developers is a black hole though. We don't have a good
idea of the amount of time it takes developers, and this is likely
something that could end up driving productivity down and keep new
contributors from getting involved.

On Wed, Nov 4, 2015 at 12:10 PM, Andreas Tolfsen <a...@mozilla.com> wrote:

> On 4 November 2015 at 15:55, William Lachance <wlac...@mozilla.com>
> wrote:

> > Is there anything we could be tracking as part of our build or test jobs
> > that we should be? Build times are one thing that immediately comes to
> mind.
>

[William Lachance]

On 2015-11-06 5:56 PM, Mark Finkle wrote:
> I also think measuring build times, and other build related stats, would be
> useful. I'd like to see Mozilla capturing those stats for developer builds
> though. I'm less interested in build times for the automation. That data is
> already looked at by the automation team.

Chris Manchester has volunteered to look into submitting build times in
automation to perfherder here:

https://bugzilla.mozilla.org/show_bug.cgi?id=1222549

I actually do think that perfherder has some advantages over the
existing grafana system, in that it has the capability of being
sheriffed easily down to the per-commit level by anyone (not just
releng/ateam).

We'll see I guess! The proof will be in bugs filed and fixed when
regressions occur. I really think developer build times are strongly
correlated with build times in automation, so my hope is that there will
be a trickle-down effect if this system proves useful.

> Build stats for developers is a black hole though. We don't have a good
> idea of the amount of time it takes developers, and this is likely
> something that could end up driving productivity down and keep new
> contributors from getting involved.

I've heard the same from developers in the Toronto office. I'm honestly
not sure how helpful tracking developer build times via some
telemetry-like system would be: everyone knows that things have gotten
worse over the last year or so, the question is how do we address that?

Will

[Chris AtLee]

On Mon, Nov 9, 2015 at 6:39 PM, William Lachance <wlac...@mozilla.com>
wrote:

> On 2015-11-06 5:56 PM, Mark Finkle wrote:
>
>> I also think measuring build times, and other build related stats, would
>> be
>> useful. I'd like to see Mozilla capturing those stats for developer builds
>> though. I'm less interested in build times for the automation. That data
>> is
>> already looked at by the automation team.
>>
>
> Chris Manchester has volunteered to look into submitting build times in
> automation to perfherder here:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1222549
>
> I actually do think that perfherder has some advantages over the existing
> grafana system, in that it has the capability of being sheriffed easily
> down to the per-commit level by anyone (not just releng/ateam).
>
> We'll see I guess! The proof will be in bugs filed and fixed when
> regressions occur. I really think developer build times are strongly
> correlated with build times in automation, so my hope is that there will be
> a trickle-down effect if this system proves useful.
>

Yes, I agree this is a better approach. Having the build system produce
logs or artifacts than can be ingested by perfherder is a more flexible
model than the one we're currently using.

[Andreas Tolfsen]

On 9 November 2015 at 18:39, William Lachance <wlac...@mozilla.com> wrote:
> Chris Manchester has volunteered to look into submitting build times in
> automation to perfherder here:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=1222549

Another thing it might be interesting to look at is ccache utilisation
over time, and how our changes to the build system (unified builds
come into mind) are affecting hits and misses. I'm not sure if this
is feasible to test.

[ishi...@yk.rim.or.jp]

2015-11-10 04:12 に Andreas Tolfsen さんは書きました:

> On 9 November 2015 at 18:39, William Lachance <wlac...@mozilla.com>
> wrote:

>> Chris Manchester has volunteered to look into submitting build times
>> in
>> automation to perfherder here:
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1222549
>

> Another thing it might be interesting to look at is ccache utilisation
> over time, and how our changes to the build system (unified builds
> come into mind) are affecting hits and misses. I'm not sure if this
> is feasible to test.

I think simply grepping "Result:" in ccache log of the day will give us
some ideas:
The ratios as follows will be important.

- Hits / the total number of Result: lines
- Misses / the total number of Result: lines
- others / the total number of Result: lines
(ccache produces other results: some compilation and linking are
recognized as artifact for
configure and other build-specific compilations and they are not handled
by ccache.)

Daily changes tracked over time should give us the feel for major build
infra change such as the introduction of unified compilation and other
future changes.

Just a thought.

BTW, does the version of ccache in the mozilla compilation farm
support --split-dwarf now?
According to https://ccache.samba.org/releasenotes.html
> ccache 3.2.3
>
> Release date: 2015-08-16
> New features and improvements
>
> Added support for compiler option -gsplit-dwarf.

I checked Debian packages and Debian now provides 3.2.4 and so
-gpslit-dwarf is supported.

"--gsplit-dwarf" certainly makes a difference on local linux build.
I helped the incorporation of the feature into ccache, and I am glad
that
it makes local build much faster! I can't live without it.
I found I could build TB even under 32-bit linux using the modified
ccache and gcc --split-dwarf and GNU gold linker during ccache patch
development although I usually use the modified ccache under 64-bit
linux.

The support of -gsplit-dwarf would/should lessen the CPU load on the
mozilla compilation farm.

This has been a plug from the ccache patch author :-)

CI