tp5 deployment and what it means for wait times

11 views
Skip to first unread message

alice nodelman

unread,
May 24, 2011, 3:05:51 PM5/24/11
to
The work in constructing tp5 is now complete, green and ready for
deployment:

https://bugzilla.mozilla.org/show_bug.cgi?id=601798 (create tp5 pageset)

tp5 is a replacement for tp4. tp4 runs 100 pages copied from the live
web during February of 2009. tp5 is a set of 100 pages copied from the
live web April 8th, 2011. Other then the newness of the pages in tp5
(which will better reflect the state of web design this year) much
effort was put into tp5 to make the pages selected for the set better
represent real browsing behavior. Instead of the Google home page it
has a page of Google search results, instead of the eBay home page it
has an eBay product page, etc.

To deploy we need to run tp5 side by side with tp4 for 2 weeks. This
will allow us to 1) generate an initial baseline result for tp5 and 2)
prove that tp5 is as effective as tp4 for catching regressions.

The downside here is that this will increase the load on the talos
performance testing pool of machines. tp5 adds an extra 10-15 minutes
of actual test time, plus the overhead of downloading and installing the
new page set. While the inconvenience will be short lived, it could
cause further delays in reporting test results.

I'd like to go ahead with rolling out tp5 as soon as possible to get the
most value out of the new set; the longer that it sits the more aged the
pages in it become.

If there is any reason not to go ahead with roll out please contact me
as soon as possible, either through newsgroups or in the bug.

Thanks,
alice.

Shawn Wilsher

unread,
May 24, 2011, 5:56:55 PM5/24/11
to dev-tree-...@lists.mozilla.org
On 5/24/2011 12:05 PM, alice nodelman wrote:
> If there is any reason not to go ahead with roll out please contact me
> as soon as possible, either through newsgroups or in the bug.
Can you please make sure it's possible to continue to run tp4 on the try
server for a time with the try chooser syntax? I'd hate to have to
regenerate any data points for bug 655930 [1] again (where I've already
sent 104 changesets to the try server) because I need to rebase on tp5.

Cheers,

Shawn

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=655930

alice nodelman

unread,
May 24, 2011, 6:33:09 PM5/24/11
to Lukas Blakk
I believe that lsblakk constructed the try chooser interface and can say
what is possible there.

alice.

Lukas Blakk

unread,
May 24, 2011, 6:37:43 PM5/24/11
to dev-tree-...@lists.mozilla.org
If it's in this list of SUITES :
http://hg.mozilla.org/build/buildbot-configs/file/14be0062035b/mozilla-tests/config.py#l44

It's able to be asked for specifically by the trychooser syntax.

Cheers,
Lukas

On 11-05-24 2:56 PM, Shawn Wilsher wrote:
> On 5/24/2011 12:05 PM, alice nodelman wrote:

>> If there is any reason not to go ahead with roll out please contact me
>> as soon as possible, either through newsgroups or in the bug.

> Can you please make sure it's possible to continue to run tp4 on the
> try server for a time with the try chooser syntax? I'd hate to have
> to regenerate any data points for bug 655930 [1] again (where I've
> already sent 104 changesets to the try server) because I need to
> rebase on tp5.
>
> Cheers,
>
> Shawn
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=655930
>
>
>

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management

Shawn Wilsher

unread,
May 24, 2011, 6:47:32 PM5/24/11
to dev-tree-...@lists.mozilla.org
On 5/24/2011 12:05 PM, alice nodelman wrote:
> The downside here is that this will increase the load on the talos
> performance testing pool of machines. tp5 adds an extra 10-15 minutes of
> actual test time, plus the overhead of downloading and installing the
> new page set. While the inconvenience will be short lived, it could
> cause further delays in reporting test results.
When we added ts_paint and tpaint, there was discussion about removing
ts and twinopen/txul because the new tests were better. Could we
possibly disable those older tests (on mozilla-central), so we have more
cycles for this?

Cheers,

Shawn

Dao

unread,
May 25, 2011, 2:35:57 AM5/25/11
to

What exactly do ts_paint and tpaint measure?

Boris Zbarsky

unread,
May 25, 2011, 2:51:59 AM5/25/11
to
On 5/25/11 2:35 AM, Dao wrote:
> What exactly do ts_paint and tpaint measure?

The time it takes to respectively start the browser and open a window
and then paint said browser or window on the screen.

The Ts/Txul tests just measure the time it takes to fire the load event,
which in many cases happens before the window has actually been painted.

So ts_paint and tpaint are closer to the user-perceived startup and
window open times.

-Boris

Dao

unread,
May 25, 2011, 8:39:27 AM5/25/11
to

What I see on Windows is that the window is rendered with a blank tab,
then the browser freezes for a second or two, and then my tabs appear. I
believe Jimm made it work this way, I frankly find it pretty annoying.

If measuring the first paint would mean not measuring the following
process (e.g. the freeze) anymore, then this seems like a bad idea. So I
guess you were referring to the /content/ window being painted?

Boris Zbarsky

unread,
May 25, 2011, 10:54:03 AM5/25/11
to
On 5/25/11 8:39 AM, Dao wrote:
> If measuring the first paint would mean not measuring the following
> process (e.g. the freeze) anymore, then this seems like a bad idea. So I
> guess you were referring to the /content/ window being painted?

I believe this is correct, but I have not verified that.

-Boris

alice nodelman

unread,
May 26, 2011, 12:00:39 PM5/26/11
to Shawn Wilsher, dev-tree-...@lists.mozilla.org
This is considered a full replacement for tp4, so will be run anywhere
that tp4 currently is - which should be all actively tested branches.

alice.

On 5/24/11 3:50 PM, Shawn Wilsher wrote:
> (sorry, I keep coming up with more questions)


>
> On 5/24/2011 12:05 PM, alice nodelman wrote:

>> The work in constructing tp5 is now complete, green and ready for
>> deployment:

> Will this be deployed to all trees, or just mozilla-central and project
> branches following it?
>
> Cheers,
>
> Shawn
>

alice nodelman

unread,
May 26, 2011, 12:11:09 PM5/26/11
to Shawn Wilsher, dev-tree-...@lists.mozilla.org
The roll out of tpaint is a separate issue and is not tied to tp4/tp5.
We need to be sure that we trust the tpaint numbers and have a
discussion about if there is any value in maintaining ts.

alice.

jmaher

unread,
May 26, 2011, 3:17:51 PM5/26/11
to

When we landed and turned on ts_paint and tpaint, it was only for
specific branches because the support for moz_after_paint is not on
all branches. I suspect we would be fine turning this off for mozilla-
central and and branches from there (the next migration to Aurora).
Turning off ts/txul will save a little bit of time, probably combined
enough to run tp5 side by side.

alice nodelman

unread,
May 26, 2011, 6:04:55 PM5/26/11
to jmaher
jmaher - if you are prepared to retire ts/txul on tpaint test enabled
branches can we get a bug filed for that blocking tp5 deployment?

alice.

alice nodelman

unread,
May 27, 2011, 1:41:49 PM5/27/11
to jmaher
So, it looks like there are still some developers interested in the
ts/txul results as shown in

https://bugzilla.mozilla.org/show_bug.cgi?id=660124 (disable ts and txul
for branches that have ts_paint and tpaint)

I don't want to block tp5 deployment on getting consensus on disabling
tx/txul. We are only talking about a 2 week window where tp4/tp5 are
running side by side, after that the wait time hit will be gone.

Also, again, the longer that we wait on tp5 the more the pages in the
set age and we start to lose the advantage of having a new set.

For now, let's take ts/txul off the table and simply try and schedule a
reasonable time to start the two week timer.

alice.

Shawn Wilsher

unread,
May 31, 2011, 10:53:07 AM5/31/11
to dev-tree-...@lists.mozilla.org
On 5/27/2011 10:41 AM, alice nodelman wrote:
> So, it looks like there are still some developers interested in the
> ts/txul results as shown in
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=660124 (disable ts and txul
> for branches that have ts_paint and tpaint)
I'm disappointed to see that one developer saying that the "data could
come in handy some day" is enough to block us from freeing up slave time
before consuming more, making our already existing slave problem worse.

/sdwilsh

alice nodelman

unread,
May 31, 2011, 12:59:31 PM5/31/11
to Shawn Wilsher, dev-tree-...@lists.mozilla.org
Yes, there is one comment in the bug. But we also haven't done a wide
question across the newsgroups/email - as we are currently doing for
tp5. We would have to reach the same consensus about disabling ts/txul
as we are currently seeking for enabling tp5.

alice.

Armen Zambrano Gasparnian

unread,
Jun 7, 2011, 11:45:38 AM6/7/11
to alice nodelman, Shawn Wilsher, dev-tree-...@lists.mozilla.org
I would not like this topic to fall off the radar and help anode get
this in.

I propose doing these changes:
* Bug 660124 - disable ts and txul for branches that have ts_paint and
tpaint
* Bug 659328 - merge a11y & scroll into 'chrome'
* there are other suites to merge but I don't want to complicate
things
* wait on Bug 659118 - move w764 talos machines to other silos
* we have repurposed the Win7 64-bit Rev3 machines for other
platforms to increase capacity
* this could bring at most 10% increase of capacity

We could also enable tp5 on mozilla-central only for the first two weeks
and then on all branches + disable tp4.

On another I don't see why it should be a whole two weeks. Could we do
this on just one week?

If we enable it this Thursday/Friday we could be ready by the 16th/17th.
Otherwise we are going to have a hard time to get this enable in between
the 21st of June release and July 5th's merge.

Regards,
Armen

Armen Zambrano Gasparnian

unread,
Jun 7, 2011, 11:45:38 AM6/7/11
to alice nodelman, Shawn Wilsher, dev-tree-...@lists.mozilla.org
I would not like this topic to fall off the radar and help anode get
this in.

I propose doing these changes:
* Bug 660124 - disable ts and txul for branches that have ts_paint and
tpaint
* Bug 659328 - merge a11y & scroll into 'chrome'
* there are other suites to merge but I don't want to complicate
things
* wait on Bug 659118 - move w764 talos machines to other silos
* we have repurposed the Win7 64-bit Rev3 machines for other
platforms to increase capacity
* this could bring at most 10% increase of capacity

We could also enable tp5 on mozilla-central only for the first two weeks
and then on all branches + disable tp4.

On another I don't see why it should be a whole two weeks. Could we do
this on just one week?

If we enable it this Thursday/Friday we could be ready by the 16th/17th.
Otherwise we are going to have a hard time to get this enable in between
the 21st of June release and July 5th's merge.

Regards,
Armen

On 11-05-31 12:59 PM, alice nodelman wrote:

Clint Talbert

unread,
Jun 8, 2011, 4:41:03 PM6/8/11
to Armen Zambrano Gasparnian, Bob Moss, joduinn, alice nodelman, Shawn Wilsher, dev-tree-...@lists.mozilla.org
I agree with you.

Let me restate the goal here: We need to land tp5 into our automation so
that we have a more robust, reliable, and adequate performance test for
our page loading system.

We need to do that at the beginning of the 6 week development cycle.
Every day we do not land this is one more day we are testing our
performance against an out-of-date slice of the web.

The patches are ready to go, have reviews, everything is set to run.
The days are gone when we can talk these issues to death. Now is the
time for action. You have some good points below, but I do not think
any of them block tp5.

On 6/7/2011 8:45 AM, Armen Zambrano Gasparnian wrote:
> I propose doing these changes:
> * Bug 660124 - disable ts and txul for branches that have ts_paint and
> tpaint

It doesn't look like we are anywhere near consensus there, forget it. We
can do this in parallel if consensus emerges.

> * Bug 659328 - merge a11y & scroll into 'chrome'
> * there are other suites to merge but I don't want to complicate things

I'd rather not incur a giant refactoring of the buildbot controllers
that run talos in order to accommodate a temporary wait time increase.
This is a good idea in general, and can be pursued in parallel to the
tp5 deployment.

> * wait on Bug 659118 - move w764 talos machines to other silos
> * we have repurposed the Win7 64-bit Rev3 machines for other platforms
> to increase capacity
> * this could bring at most 10% increase of capacity

This is awesome. But I don't see any reason why this repurposing cannot
happen in parallel to the talos tp5 deployment. We need to move quickly
here, and avoid serializing ourselves unnecessarily.


>
> We could also enable tp5 on mozilla-central only for the first two weeks
> and then on all branches + disable tp4.
>

This just sounds really complex and error-prone, I'm not in favor of it.

> On another I don't see why it should be a whole two weeks. Could we do
> this on just one week?

YES!! If there are no issues, if tp4 and tp5 track each other well then
yes, maybe we only need one week. The two weeks requirement is an "at
most". That's the amount of time we expect to need in the worst case
scenario.


>
> If we enable it this Thursday/Friday we could be ready by the 16th/17th.
> Otherwise we are going to have a hard time to get this enable in between
> the 21st of June release and July 5th's merge.
>

Exactly. So, let's do it tomorrow. Let's activate the tp5 set on m-c as
soon as possible. Let's attack these other items (many of them are
great ideas) in parallel. None of these should really block us moving
forward with making our Tp testing into a more robust and representative
sample of today's web.

Let's get it done.

Clint


Armen Zambrano Gasparnian

unread,
Jun 8, 2011, 4:47:18 PM6/8/11
to Clint Talbert, Bob Moss, joduinn, alice nodelman, Shawn Wilsher, dev-tree-...@lists.mozilla.org
The purpose of any of the things I proposed was not to slow this down
even more but to have things that can help it move forward.
My first though was that while we looked at tracemonkey we could knock
some more improvements on the pool so we have less opposition.

I am about to do beta5 and I have already spoken with anode to try to
see if we can get this in tomorrow (for tracemonkey that is; doing some
staging and local testing right now) and send an email late on Thursday
(depending on what anode says about go/no-go) saying that we want to
enable this everywhere on Monday/Tuesday (subject to adjustments
depending on events/results).

I will tackle the inline comments in another moment.

PS = I am off on Friday

cheers,
Armen


--
Armen Zambrano Gasparnian (armenzg)
Mozilla Corp Release Engineer
~ Jesus Christ is my Lord

Armen Zambrano Gasparnian

unread,
Jun 8, 2011, 4:47:18 PM6/8/11
to Clint Talbert, Shawn Wilsher, Bob Moss, alice nodelman, dev-tree-...@lists.mozilla.org, joduinn
The purpose of any of the things I proposed was not to slow this down
even more but to have things that can help it move forward.
My first though was that while we looked at tracemonkey we could knock
some more improvements on the pool so we have less opposition.

I am about to do beta5 and I have already spoken with anode to try to
see if we can get this in tomorrow (for tracemonkey that is; doing some
staging and local testing right now) and send an email late on Thursday
(depending on what anode says about go/no-go) saying that we want to
enable this everywhere on Monday/Tuesday (subject to adjustments
depending on events/results).

I will tackle the inline comments in another moment.

PS = I am off on Friday

cheers,
Armen

On 11-06-08 4:41 PM, Clint Talbert wrote:

Clint Talbert

unread,
Jun 8, 2011, 8:33:46 PM6/8/11
to Bob Moss, joduinn, Alice Nodelman
On 6/8/2011 1:47 PM, Armen Zambrano Gasparnian wrote:
> The purpose of any of the things I proposed was not to slow this down
> even more but to have things that can help it move forward.
I know. But we need to parallelize more if we're ever going to get
anywhere on this. You had lots of good ideas, but I don't think they
need to be done serially, which is how your post read.

> My first though was that while we looked at tracemonkey we could knock
> some more improvements on the pool so we have less opposition.
>

This is kind of like getting a vaccination. It's going to hurt and be
unpleasant for a little while. It is what we need to do, and the
unpleasantness will be gone in a week or two.

Our window is rapidly shrinking. We need to do it now, or else we are
not going to be testing modern web pages with our performance testing in
aurora and beta, which is not an outcome any of us want.

> I am about to do beta5 and I have already spoken with anode to try to
> see if we can get this in tomorrow (for tracemonkey that is; doing some
> staging and local testing right now) and send an email late on Thursday
> (depending on what anode says about go/no-go) saying that we want to
> enable this everywhere on Monday/Tuesday (subject to adjustments
> depending on events/results).

Sounds like a good way forward. Thanks.

Clint

Armen Zambrano Gasparnian

unread,
Jun 15, 2011, 10:37:16 AM6/15/11
to Clint Talbert, Shawn Wilsher, Bob Moss, alice nodelman, dev-tree-...@lists.mozilla.org, joduinn
This is now enabled on TraceMonkey.
We have enough capacity to enable it on all branches (the extra machines
were added last week). I have placed a patch for it on anode's queue.
There is an intermittent tp5 hang on bug 664371.

Please let us know when you think we are ready to enable tp5 everywhere
and we'll do it right away.

cheers,
Armen

On 11-06-08 4:41 PM, Clint Talbert wrote:

Clint Talbert

unread,
Jun 15, 2011, 1:36:30 PM6/15/11
to Armen Zambrano Gasparnian, Shawn Wilsher, Bob Moss, alice nodelman, dev-tree-...@lists.mozilla.org, joduinn
I think we should go forward with this as soon as possible so that it
doesn't back up against the upcoming m-c -> aurora merge craziness early
next month.

Thanks very much Armen and Alice for working through all these issues!
Clint

Armen Zambrano Gasparnian

unread,
Jun 15, 2011, 1:59:55 PM6/15/11
to Clint Talbert, Shawn Wilsher, Bob Moss, alice nodelman, dev-tree-...@lists.mozilla.org, joduinn
AFAIK there are no more known issues.
I would like to enable this tomorrow and start the 2 weeks clock.

cheers,
Armen

Reply all
Reply to author
Forward
0 new messages