tp4 & tp5

[armenzg]

Hi all,
Today (a couple of hours ago) we have enabled tp5 everywhere (except
for older release branches).
tp5 and tp4 will be running side by side for some time until we are in
good condition to turn off tp4.

We estimate that 2 weeks should be plenty of time to get a good
baseline and allow people to start using tp5 as reference rather than
tp4.
We will bring this up to the developers' meeting and make sure that is
alright to shut tp4 on June 30th (except older release branches).

This adds ~15 mins for each tp run compared to just running tp4.

Let me know if you have any questions.

Regards,
Armen

-> bug 601798

[armenzg]

We are still on schedule to disable tp4 on Thursday June 30th.

This was brought up at yesterday's developer meeting and no one raised
any questions or concerns.
I will raise this up on next Tuesday meeting to make sure everyone is
aware.

If you have any comments about it or find a bug please let us know on
bug 661010.

cheers,
Armen

On Jun 16, 1:37 pm, armenzg <arme...@gmail.com> wrote:
> I forgot to post this on dev.planning as well.

[Boris Zbarsky]

On 6/22/11 9:21 AM, armenzg wrote:
> We are still on schedule to disable tp4 on Thursday June 30th.

I have a related question, actually. Tp5 doesn't seem to be in
compare-talos. Can we fix that before disabling Tp4 so that performance
comparisons can still be done?

> If you have any comments about it or find a bug please let us know on
> bug 661010.

Is that the right bug number? Seems unrelated to Tp4....

-Boris

[armenzg]

That's right.

Who are the owners of the following?
https://services.forerunnerdesigns.com/compare-talos/
http://perf.snarkfest.net/compare-talos/

Which one is the official one?

Right! bug 664831 that is.

>
> -Boris

[Armen Zambrano Gasparnian]

I posted a patch and will announce it again once is live.

Thanks Boris!

[Daniel Holbert]

forerunnerdesigns.com is owned by sdwilsh, I think.

~Daniel

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management

[Armen Zambrano Gasparnian]

This has now been deployed to perf.snarkfest thanks to mconnor.

I see it working now:
http://perf.snarkfest.net/compare-talos/index.html?oldRevs=02f8dbc08005&newRev=b5dead378605&tests=tp4,tp4_memset,tp4_pbytes,tp4_rss,tp4_shutdown,tp4_xres,tp5,tp5_memset,tp5_pbytes,tp5_rss,tp5_shutdown,tp5_xres&submit=true

[Boris Zbarsky]

On 6/22/11 4:45 PM, Armen Zambrano Gasparnian wrote:
> This has now been deployed to perf.snarkfest thanks to mconnor.
>
> I see it working now:
> http://perf.snarkfest.net/compare-talos/index.html?oldRevs=02f8dbc08005&newRev=b5dead378605&tests=tp4,tp4_memset,tp4_pbytes,tp4_rss,tp4_shutdown,tp4_xres,tp5,tp5_memset,tp5_pbytes,tp5_rss,tp5_shutdown,tp5_xres&submit=true

Awesome! Does tbpl also need updating to set the right URI? The one it
sends right now looks like this:

http://perf.snarkfest.net/compare-talos/index.html?oldRevs=b5dead378605&newRev=db6d8ef5b27c&tests=%20a11y,tdhtml,tdhtml_nochrome,tp4,tp4_memset,tp4_pbytes,tp4_rss,tp4_shutdown,tp4_xres,dromaeo_basics,dromaeo_css,dromaeo_dom,dromaeo_jslib,dromaeo_sunspider,dromaeo_v8,tsspider,tsspider_nochrome,tgfx,tgfx_nochrome,tscroll,tsvg,tsvg_opacity,ts,ts_cold,ts_cold_generated_max,ts_cold_generated_max_shutdown,ts_cold_generated_med,ts_cold_generated_med_shutdown,ts_cold_generated_min,ts_cold_generated_min_shutdown,ts_cold_shutdown,ts_places_generated_max,ts_places_generated_max_shutdown,ts_places_generated_med,ts_places_generated_med_shutdown,ts_places_generated_min,ts_places_generated_min_shutdown,ts_shutdown,twinopen&submit=true

which excludes Tp5.

-Boris

[Armen Zambrano Gasparnian]

Thanks for pointing it out.
I have filed a bug and hopefully will be resolved in the next day or two.

https://bugzilla.mozilla.org/show_bug.cgi?id=666711

cheers,
Armen

[Boris Zbarsky]

On 6/23/11 3:50 PM, Armen Zambrano Gasparnian wrote:
> Thanks for pointing it out.
> I have filed a bug and hopefully will be resolved in the next day or two.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=666711

Also, does Try Chooser (both the hook and the syntax builder) need updating?

-Boris

[Armen Zambrano Gasparnian]

On 11-06-23 11:18 PM, Boris Zbarsky wrote:
> On 6/23/11 3:50 PM, Armen Zambrano Gasparnian wrote:
>> Thanks for pointing it out.
>> I have filed a bug and hopefully will be resolved in the next day or two.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=666711

tbpl has been fixed.

BTW when we disable tp4 I would like to remove it as it will return
empty comparison.

>
> Also, does Try Chooser (both the hook and the syntax builder) need
> updating?

The Try Syntax gets updated automatically.

The Try Chooser web page needs to be updated manually:
http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36

>
> -Boris

Man this is finicky!
Thanks a lot Boris, doing things to the last details is very important.

cheers,
Armen

[Boris Zbarsky]

On 6/24/11 12:22 PM, Armen Zambrano Gasparnian wrote:
> The Try Chooser web page needs to be updated manually:
> http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36

Huh. So it's just called "tp" in the try syntax now and that triggers
both tp4 and tp5 at the moment but will switch to just tp5? Or just
triggers tp5 already?

> Man this is finicky!
> Thanks a lot Boris, doing things to the last details is very important.

No problem. If I think of anything else, I'll bring it up. ;)

Are we getting all this down in a checklist for next time we roll out a
new test suite, by the way?

-Boris

[Armen Zambrano Gasparnian]

On 11-06-24 12:43 PM, Boris Zbarsky wrote:
> On 6/24/11 12:22 PM, Armen Zambrano Gasparnian wrote:
>> The Try Chooser web page needs to be updated manually:
>> http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36
>>
>
> Huh. So it's just called "tp" in the try syntax now and that triggers
> both tp4 and tp5 at the moment but will switch to just tp5? Or just
> triggers tp5 already?
>

Two things happened.
* The job got renamed from "tp4" to "tp".
* tp4 & tp5 are run within the "tp" job
Next week the "tp" job will only run "tp5".

>> Man this is finicky!
>> Thanks a lot Boris, doing things to the last details is very important.
>
> No problem. If I think of anything else, I'll bring it up. ;)

I am glad you do! :)

>
> Are we getting all this down in a checklist for next time we roll out a
> new test suite, by the way?

I added a note on the bug and added a checklist on the a-team's wiki page.
https://bugzilla.mozilla.org/show_bug.cgi?id=664831#c4
https://wiki.mozilla.org/Auto-tools/staging#Newer_suites

>
> -Boris

[Armen Zambrano Gasparnian]

We are still on schedule to disable tp4 this week and leave tp5 running.

The tbpl and compare-talos issues were ironed out.

This is also to be announced on the platform meeting tomorrow.

For more details please read:
http://armenzg.blogspot.com/2011/06/disabling-tp4.html

cheers,
Armen

[Armen Zambrano Gasparnian]

On 11-06-28 11:38 PM, Robert O'Callahan wrote:

> On Thu, Jun 23, 2011 at 1:21 AM, armenzg<arm...@mozilla.com> wrote:
>
>> We are still on schedule to disable tp4 on Thursday June 30th.
>>
>> This was brought up at yesterday's developer meeting and no one raised
>> any questions or concerns.
>> I will raise this up on next Tuesday meeting to make sure everyone is
>> aware.
>>
>> If you have any comments about it or find a bug please let us know on
>> bug 661010.
>>
>

> Just so you know, I am currently working on a regression that was reported
> for Tp4 but not Tp5. See my email(s) to dev.tree-management.
>
> Rob
If we keep on running it on mozilla-inbound and try would it give you
what you need until you figure things out?

cheers,
Armen

[Armen Zambrano Gasparnian]

On 11-06-28 11:38 PM, Robert O'Callahan wrote:
> On Thu, Jun 23, 2011 at 1:21 AM, armenzg<arm...@mozilla.com> wrote:
>

>> We are still on schedule to disable tp4 on Thursday June 30th.
>>
>> This was brought up at yesterday's developer meeting and no one raised
>> any questions or concerns.
>> I will raise this up on next Tuesday meeting to make sure everyone is
>> aware.
>>
>> If you have any comments about it or find a bug please let us know on
>> bug 661010.
>>
>

[Boris Zbarsky]

On 6/29/11 10:22 AM, Armen Zambrano Gasparnian wrote:
> If we keep on running it on mozilla-inbound and try would it give you
> what you need until you figure things out?

Well, an additional interesting question is why Tp5 doesn't show the
regression....

-Boris

[Armen Zambrano Gasparnian]

On another note, I can find a regression that tp5 found but not tp4:
Search for "Talos Regression :( Tp5 (Private Bytes) increase 3.74% on
MacOSX 10.6.2 Firefox"; is this right?

From looking at the email regression notifications I have seen that
every time a regression is found there is around a ~2% difference [1].

Considered that, the regression that roc is looking into is 2.68% for
tp4 and the tp5 difference could be around 0.68% or less (or even more
if my theory fails).

I am CCing other people that could answer this question.

Also note that these are two different sets of pages and perhaps the
newer set of pages can have pages that have intentionally made so they
are harder to run slow no matter what browser runs them. (NOTE:
Hypothesis of a non-expert in the field of browser performance).

For example: [1]
- 6.97%/5.97%
- 5.14%/3.8%
- 145%/131% (14 points difference but we are also talking about 3 digits)