Next Week - Running Talos in row order vs column order

[jmaher]

Next week the automation and tools team will be changing the way we
run talos (tp5 for now) to be row order instead of column order. You
can get some background in this blog post:
https://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/.

Column order means that we take a list of web pages (tp5 has 100
pages), and load each page one time, then cycle through the test.

Row order means that for each page in the list of web pages, we load
that page X times (number of cycles), then move onto the next page.

A concern expressed in the above mentioned blog post is we are not
measuring what the end user sees. This is true, but we have not been
measuring what the end user sees as we have an empty profile that has
specific preferences set and a webserver on the local machine instead
of loading pages over the internet. Historically we have always
dropped the highest value in the series which for tp5 means the first
page load. So all measurements we have taken to date only measure
page load times after the page has been loaded.

The result will be more uniform page load times. We are going to run
25 cycles and drop the first 5 numbers (instead of the first one).
This will give us a repeatable number for the given web page we are
testing.

Next quarter we will be rolling out a new set of changes that collect
all the raw numbers (this is already being collected on our staging
branch 'Pine' into a development database) instead of the median
value for each page. This upcoming work will allow us to view test
results page by page with a UI to drill down. By collecting raw
numbers we will have the ability to view the first load times and
report on those in addition to our normal reported data.

[Armen Zambrano G.]

Hi all,
We will be switching to run tp in rows rather than column order tomorrow
morning.
Please read below for more information.

This won't affect mozilla-1.9.2, mozilla-esr10, mozilla-aurora,
mozilla-beta and mozilla-release.
On merge days the change will be uplifted to m-a/m-b and m-r.

Here is the bug where the work is being done:
https://bugzilla.mozilla.org/show_bug.cgi?id=737458

cheers,
Armen

[Kyle Huey]

Hey Armen,

The DOM team plans to land some performance-sensitive work tomorrow to meet
one of our Q1 goals. (Bug 740069 for those playing along at hime) Given
that this may change the Talos numbers (and we apparently already tried to
do this last week and it broke the tree?) can we hold off on this until
Monday so that we can be sure that we can land and see any performance
regressions we cause?

- Kyle

On Thu, Mar 29, 2012 at 10:44 AM, Armen Zambrano G. <arm...@mozilla.com>wrote:

> Hi all,
> We will be switching to run tp in rows rather than column order tomorrow
> morning.
> Please read below for more information.
>
> This won't affect mozilla-1.9.2, mozilla-esr10, mozilla-aurora,
> mozilla-beta and mozilla-release.
> On merge days the change will be uplifted to m-a/m-b and m-r.
>
> Here is the bug where the work is being done:

> https://bugzilla.mozilla.org/**show_bug.cgi?id=737458<https://bugzilla.mozilla.org/show_bug.cgi?id=737458>
>
> cheers,
> Armen
>

[Armen Zambrano G.]

I can keep tpr_responsiveness running for mozilla-central, larch and
mozilla-inbound.

Would that help?

[Armen Zambrano G.]

I can keep tpr_responsiveness running for mozilla-central, larch and
mozilla-inbound.

Would that help?


On 12-03-29 1:57 PM, Kyle Huey wrote:

[Kyle Huey]

m-c and larch would be sufficient, I think.

Thanks!

- Kyle

>>> https://bugzilla.mozilla.org/****show_bug.cgi?id=737458<https://bugzilla.mozilla.org/**show_bug.cgi?id=737458>
>>> <https:**//bugzilla.mozilla.org/show_**bug.cgi?id=737458<https://bugzilla.mozilla.org/show_bug.cgi?id=737458>
>>> >
>>>
>>> cheers,
>>> Armen
>>>
>>>
>
>