tp5 deployment and what it means for wait times
alice nodelman
deployment:
https://bugzilla.mozilla.org/show_bug.cgi?id=601798 (create tp5 pageset)
tp5 is a replacement for tp4. tp4 runs 100 pages copied from the live
web during February of 2009. tp5 is a set of 100 pages copied from the
live web April 8th, 2011. Other then the newness of the pages in tp5
(which will better reflect the state of web design this year) much
effort was put into tp5 to make the pages selected for the set better
represent real browsing behavior. Instead of the Google home page it
has a page of Google search results, instead of the eBay home page it
has an eBay product page, etc.
To deploy we need to run tp5 side by side with tp4 for 2 weeks. This
will allow us to 1) generate an initial baseline result for tp5 and 2)
prove that tp5 is as effective as tp4 for catching regressions.
The downside here is that this will increase the load on the talos
performance testing pool of machines. tp5 adds an extra 10-15 minutes
of actual test time, plus the overhead of downloading and installing the
new page set. While the inconvenience will be short lived, it could
cause further delays in reporting test results.
I'd like to go ahead with rolling out tp5 as soon as possible to get the
most value out of the new set; the longer that it sits the more aged the
pages in it become.
If there is any reason not to go ahead with roll out please contact me
as soon as possible, either through newsgroups or in the bug.
Thanks,
alice.
Shawn Wilsher
> If there is any reason not to go ahead with roll out please contact me
> as soon as possible, either through newsgroups or in the bug.
server for a time with the try chooser syntax? I'd hate to have to
regenerate any data points for bug 655930 [1] again (where I've already
sent 104 changesets to the try server) because I need to rebase on tp5.
Cheers,
Shawn
alice nodelman
what is possible there.
alice.
Lukas Blakk
http://hg.mozilla.org/build/buildbot-configs/file/14be0062035b/mozilla-tests/config.py#l44
It's able to be asked for specifically by the trychooser syntax.
Cheers,
Lukas
On 11-05-24 2:56 PM, Shawn Wilsher wrote:
> On 5/24/2011 12:05 PM, alice nodelman wrote:
>> If there is any reason not to go ahead with roll out please contact me
>> as soon as possible, either through newsgroups or in the bug.
> Can you please make sure it's possible to continue to run tp4 on the
> try server for a time with the try chooser syntax? I'd hate to have
> to regenerate any data points for bug 655930 [1] again (where I've
> already sent 104 changesets to the try server) because I need to
> rebase on tp5.
>
> Cheers,
>
> Shawn
>
> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=655930
>
>
>
> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management
Shawn Wilsher
> performance testing pool of machines. tp5 adds an extra 10-15 minutes of
> actual test time, plus the overhead of downloading and installing the
> new page set. While the inconvenience will be short lived, it could
> cause further delays in reporting test results.
ts and twinopen/txul because the new tests were better. Could we
possibly disable those older tests (on mozilla-central), so we have more
cycles for this?
Cheers,
Shawn
Dao
What exactly do ts_paint and tpaint measure?
Boris Zbarsky
> What exactly do ts_paint and tpaint measure?
The time it takes to respectively start the browser and open a window
and then paint said browser or window on the screen.
The Ts/Txul tests just measure the time it takes to fire the load event,
which in many cases happens before the window has actually been painted.
So ts_paint and tpaint are closer to the user-perceived startup and
window open times.
-Boris
Dao
What I see on Windows is that the window is rendered with a blank tab,
then the browser freezes for a second or two, and then my tabs appear. I
believe Jimm made it work this way, I frankly find it pretty annoying.
If measuring the first paint would mean not measuring the following
process (e.g. the freeze) anymore, then this seems like a bad idea. So I
guess you were referring to the /content/ window being painted?
Boris Zbarsky
> If measuring the first paint would mean not measuring the following
> process (e.g. the freeze) anymore, then this seems like a bad idea. So I
> guess you were referring to the /content/ window being painted?
I believe this is correct, but I have not verified that.
-Boris
alice nodelman
that tp4 currently is - which should be all actively tested branches.
alice.
On 5/24/11 3:50 PM, Shawn Wilsher wrote:
> (sorry, I keep coming up with more questions)
>
> On 5/24/2011 12:05 PM, alice nodelman wrote:
>> The work in constructing tp5 is now complete, green and ready for
>> deployment:
> Will this be deployed to all trees, or just mozilla-central and project
> branches following it?
>
> Cheers,
>
> Shawn
>
alice nodelman
We need to be sure that we trust the tpaint numbers and have a
discussion about if there is any value in maintaining ts.
alice.
jmaher
When we landed and turned on ts_paint and tpaint, it was only for
specific branches because the support for moz_after_paint is not on
all branches. I suspect we would be fine turning this off for mozilla-
central and and branches from there (the next migration to Aurora).
Turning off ts/txul will save a little bit of time, probably combined
enough to run tp5 side by side.
alice nodelman
branches can we get a bug filed for that blocking tp5 deployment?
alice.
alice nodelman
ts/txul results as shown in
https://bugzilla.mozilla.org/show_bug.cgi?id=660124 (disable ts and txul
for branches that have ts_paint and tpaint)
I don't want to block tp5 deployment on getting consensus on disabling
tx/txul. We are only talking about a 2 week window where tp4/tp5 are
running side by side, after that the wait time hit will be gone.
Also, again, the longer that we wait on tp5 the more the pages in the
set age and we start to lose the advantage of having a new set.
For now, let's take ts/txul off the table and simply try and schedule a
reasonable time to start the two week timer.
alice.
Shawn Wilsher
> So, it looks like there are still some developers interested in the
> ts/txul results as shown in
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=660124 (disable ts and txul
> for branches that have ts_paint and tpaint)
come in handy some day" is enough to block us from freeing up slave time
before consuming more, making our already existing slave problem worse.
/sdwilsh
alice nodelman
question across the newsgroups/email - as we are currently doing for
tp5. We would have to reach the same consensus about disabling ts/txul
as we are currently seeking for enabling tp5.
alice.
Armen Zambrano Gasparnian
this in.
I propose doing these changes:
* Bug 660124 - disable ts and txul for branches that have ts_paint and
tpaint
* Bug 659328 - merge a11y & scroll into 'chrome'
* there are other suites to merge but I don't want to complicate
things
* wait on Bug 659118 - move w764 talos machines to other silos
* we have repurposed the Win7 64-bit Rev3 machines for other
platforms to increase capacity
* this could bring at most 10% increase of capacity
We could also enable tp5 on mozilla-central only for the first two weeks
and then on all branches + disable tp4.
On another I don't see why it should be a whole two weeks. Could we do
this on just one week?
If we enable it this Thursday/Friday we could be ready by the 16th/17th.
Otherwise we are going to have a hard time to get this enable in between
the 21st of June release and July 5th's merge.
Regards,
Armen
Armen Zambrano Gasparnian
this in.
I propose doing these changes:
* Bug 660124 - disable ts and txul for branches that have ts_paint and
tpaint
* Bug 659328 - merge a11y & scroll into 'chrome'
* there are other suites to merge but I don't want to complicate
things
* wait on Bug 659118 - move w764 talos machines to other silos
* we have repurposed the Win7 64-bit Rev3 machines for other
platforms to increase capacity
* this could bring at most 10% increase of capacity
We could also enable tp5 on mozilla-central only for the first two weeks
and then on all branches + disable tp4.
On another I don't see why it should be a whole two weeks. Could we do
this on just one week?
If we enable it this Thursday/Friday we could be ready by the 16th/17th.
Otherwise we are going to have a hard time to get this enable in between
the 21st of June release and July 5th's merge.
Regards,
Armen
On 11-05-31 12:59 PM, alice nodelman wrote:
Clint Talbert
Let me restate the goal here: We need to land tp5 into our automation so
that we have a more robust, reliable, and adequate performance test for
our page loading system.
We need to do that at the beginning of the 6 week development cycle.
Every day we do not land this is one more day we are testing our
performance against an out-of-date slice of the web.
The patches are ready to go, have reviews, everything is set to run.
The days are gone when we can talk these issues to death. Now is the
time for action. You have some good points below, but I do not think
any of them block tp5.
On 6/7/2011 8:45 AM, Armen Zambrano Gasparnian wrote:
> I propose doing these changes:
> * Bug 660124 - disable ts and txul for branches that have ts_paint and
> tpaint
It doesn't look like we are anywhere near consensus there, forget it. We
can do this in parallel if consensus emerges.
> * Bug 659328 - merge a11y & scroll into 'chrome'
> * there are other suites to merge but I don't want to complicate things
I'd rather not incur a giant refactoring of the buildbot controllers
that run talos in order to accommodate a temporary wait time increase.
This is a good idea in general, and can be pursued in parallel to the
tp5 deployment.
> * wait on Bug 659118 - move w764 talos machines to other silos
> * we have repurposed the Win7 64-bit Rev3 machines for other platforms
> to increase capacity
> * this could bring at most 10% increase of capacity
This is awesome. But I don't see any reason why this repurposing cannot
happen in parallel to the talos tp5 deployment. We need to move quickly
here, and avoid serializing ourselves unnecessarily.
>
> We could also enable tp5 on mozilla-central only for the first two weeks
> and then on all branches + disable tp4.
>
This just sounds really complex and error-prone, I'm not in favor of it.
> On another I don't see why it should be a whole two weeks. Could we do
> this on just one week?
YES!! If there are no issues, if tp4 and tp5 track each other well then
yes, maybe we only need one week. The two weeks requirement is an "at
most". That's the amount of time we expect to need in the worst case
scenario.
>
> If we enable it this Thursday/Friday we could be ready by the 16th/17th.
> Otherwise we are going to have a hard time to get this enable in between
> the 21st of June release and July 5th's merge.
>
Exactly. So, let's do it tomorrow. Let's activate the tp5 set on m-c as
soon as possible. Let's attack these other items (many of them are
great ideas) in parallel. None of these should really block us moving
forward with making our Tp testing into a more robust and representative
sample of today's web.
Let's get it done.
Clint
Armen Zambrano Gasparnian
even more but to have things that can help it move forward.
My first though was that while we looked at tracemonkey we could knock
some more improvements on the pool so we have less opposition.
I am about to do beta5 and I have already spoken with anode to try to
see if we can get this in tomorrow (for tracemonkey that is; doing some
staging and local testing right now) and send an email late on Thursday
(depending on what anode says about go/no-go) saying that we want to
enable this everywhere on Monday/Tuesday (subject to adjustments
depending on events/results).
I will tackle the inline comments in another moment.
PS = I am off on Friday
cheers,
Armen
--
Armen Zambrano Gasparnian (armenzg)
Mozilla Corp Release Engineer
~ Jesus Christ is my Lord
Armen Zambrano Gasparnian
even more but to have things that can help it move forward.
My first though was that while we looked at tracemonkey we could knock
some more improvements on the pool so we have less opposition.
I am about to do beta5 and I have already spoken with anode to try to
see if we can get this in tomorrow (for tracemonkey that is; doing some
staging and local testing right now) and send an email late on Thursday
(depending on what anode says about go/no-go) saying that we want to
enable this everywhere on Monday/Tuesday (subject to adjustments
depending on events/results).
I will tackle the inline comments in another moment.
PS = I am off on Friday
cheers,
Armen
On 11-06-08 4:41 PM, Clint Talbert wrote:
Clint Talbert
> The purpose of any of the things I proposed was not to slow this down
> even more but to have things that can help it move forward.
anywhere on this. You had lots of good ideas, but I don't think they
need to be done serially, which is how your post read.
> My first though was that while we looked at tracemonkey we could knock
> some more improvements on the pool so we have less opposition.
>
This is kind of like getting a vaccination. It's going to hurt and be
unpleasant for a little while. It is what we need to do, and the
unpleasantness will be gone in a week or two.
Our window is rapidly shrinking. We need to do it now, or else we are
not going to be testing modern web pages with our performance testing in
aurora and beta, which is not an outcome any of us want.
> I am about to do beta5 and I have already spoken with anode to try to
> see if we can get this in tomorrow (for tracemonkey that is; doing some
> staging and local testing right now) and send an email late on Thursday
> (depending on what anode says about go/no-go) saying that we want to
> enable this everywhere on Monday/Tuesday (subject to adjustments
> depending on events/results).
Sounds like a good way forward. Thanks.
Clint
Armen Zambrano Gasparnian
We have enough capacity to enable it on all branches (the extra machines
were added last week). I have placed a patch for it on anode's queue.
There is an intermittent tp5 hang on bug 664371.
Please let us know when you think we are ready to enable tp5 everywhere
and we'll do it right away.
cheers,
Armen
On 11-06-08 4:41 PM, Clint Talbert wrote:
Clint Talbert
doesn't back up against the upcoming m-c -> aurora merge craziness early
next month.
Thanks very much Armen and Alice for working through all these issues!
Clint
Armen Zambrano Gasparnian
I would like to enable this tomorrow and start the 2 weeks clock.
cheers,
Armen