tp5 deployment and what it means for wait times

[alice nodelman]

The work in constructing tp5 is now complete, green and ready for
deployment:

https://bugzilla.mozilla.org/show_bug.cgi?id=601798 (create tp5 pageset)

tp5 is a replacement for tp4. tp4 runs 100 pages copied from the live
web during February of 2009. tp5 is a set of 100 pages copied from the
live web April 8th, 2011. Other then the newness of the pages in tp5
(which will better reflect the state of web design this year) much
effort was put into tp5 to make the pages selected for the set better
represent real browsing behavior. Instead of the Google home page it
has a page of Google search results, instead of the eBay home page it
has an eBay product page, etc.

To deploy we need to run tp5 side by side with tp4 for 2 weeks. This
will allow us to 1) generate an initial baseline result for tp5 and 2)
prove that tp5 is as effective as tp4 for catching regressions.

The downside here is that this will increase the load on the talos
performance testing pool of machines. tp5 adds an extra 10-15 minutes
of actual test time, plus the overhead of downloading and installing the
new page set. While the inconvenience will be short lived, it could
cause further delays in reporting test results.

I'd like to go ahead with rolling out tp5 as soon as possible to get the
most value out of the new set; the longer that it sits the more aged the
pages in it become.

If there is any reason not to go ahead with roll out please contact me
as soon as possible, either through newsgroups or in the bug.

Thanks,
alice.

[Nicholas Nethercote]

On Tue, May 24, 2011 at 12:05 PM, alice nodelman <anod...@mozilla.com> wrote:
> The work in constructing tp5 is now complete, green and ready for
> deployment:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=601798 (create tp5 pageset)
>
> tp5 is a replacement for tp4.  tp4 runs 100 pages copied from the live web
> during February of 2009.  tp5 is a set of 100 pages copied from the live web
> April 8th, 2011.  Other then the newness of the pages in tp5 (which will
> better reflect the state of web design this year) much effort was put into
> tp5 to make the pages selected for the set better represent real browsing
> behavior.  Instead of the Google home page it has a page of Google search
> results, instead of the eBay home page it has an eBay product page, etc.

Sounds like a nice improvement :)

I'm a bit confused about what Tp4 and Tp5 actually measure, though. I read
https://wiki.mozilla.org/Buildbot/Talos#Talos_Tests, it's clear they
are memory tests, and on each platform several measurements are made
every 20 seconds. For example, on Windows 7 we measure four things
for Tp4:

- Private Bytes
- Working Set (tp4_memset)
- Modified Page List Bytes
- % CPU

That seems clear, but the graphs at http://graphs-new.mozilla.org/
show a single number for Tp4, and there's no units on the y-axis. I
guess that number is one of the above four metrics, and I guess it's
the average of all the measurements taken during the run?

Any clarification here would be welcome! Thanks.

Nick

[Shawn Wilsher]

On 5/24/2011 5:06 PM, Nicholas Nethercote wrote:
> I'm a bit confused about what Tp4 and Tp5 actually measure, though. I read
> https://wiki.mozilla.org/Buildbot/Talos#Talos_Tests, it's clear they
> are memory tests, and on each platform several measurements are made
> every 20 seconds. For example, on Windows 7 we measure four things
> for Tp4:

tp4 is primarily a measure of page load times.

> That seems clear, but the graphs at http://graphs-new.mozilla.org/
> show a single number for Tp4, and there's no units on the y-axis. I
> guess that number is one of the above four metrics, and I guess it's
> the average of all the measurements taken during the run?

To see the additional data, you need to use a custom chart (like
http://graphs-new.mozilla.org/graph.html#tests=[[39,1,12],[39,1,1]]&sel=none&displayrange=7&datatype=running
for tp4 private bytes on windows).

Cheers,

Shawn

[Robert Kaiser]

Nicholas Nethercote schrieb:

> I'm a bit confused about what Tp4 and Tp5 actually measure, though.

Tp (in all its versions) always has primarily been a performance measure
- "Time pageload" is where the name comes from, IIRC.
AFAIK, in recent times we have started to gather some other info around
that as well, though, and are gathering memory and CPU data in addition
to the page load time measure it has been made for.

Robert Kaiser

--
Note that any statements of mine - no matter how passionate - are never
meant to be offensive but very often as food for thought or possible
arguments that we as a community should think about. And most of the
time, I even appreciate irony and fun! :)