Re: tp4 & tp5
armenzg
On Jun 16, 1:36 pm, armenzg <arme...@gmail.com> wrote:
> Hi all,
> Today (a couple of hours ago) we have enabled tp5 everywhere (except
> for older release branches).
> tp5 and tp4 will be running side by side for some time until we are in
> good condition to turn off tp4.
>
> We estimate that 2 weeks should be plenty of time to get a good
> baseline and allow people to start using tp5 as reference rather than
> tp4.
> We will bring this up to the developers' meeting and make sure that is
> alright to shut tp4 on June 30th (except older release branches).
>
> This adds ~15 mins for each tp run compared to just running tp4.
>
> Let me know if you have any questions.
>
> Regards,
> Armen
>
> -> bug 601798
armenzg
This was brought up at yesterday's developer meeting and no one raised
any questions or concerns.
I will raise this up on next Tuesday meeting to make sure everyone is
aware.
If you have any comments about it or find a bug please let us know on
bug 661010.
cheers,
Armen
Boris Zbarsky
> We are still on schedule to disable tp4 on Thursday June 30th.
I have a related question, actually. Tp5 doesn't seem to be in
compare-talos. Can we fix that before disabling Tp4 so that performance
comparisons can still be done?
> If you have any comments about it or find a bug please let us know on
> bug 661010.
Is that the right bug number? Seems unrelated to Tp4....
-Boris
armenzg
Who are the owners of the following?
https://services.forerunnerdesigns.com/compare-talos/
http://perf.snarkfest.net/compare-talos/
Which one is the official one?
Right! bug 664831 that is.
>
> -Boris
Armen Zambrano Gasparnian
Thanks Boris!
beltzner
cheers,
mike
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Mike Connor
forerunner is sdwilsh's test install, perf.snarkfest is what tbpl points at.
perf.snarkfest.net is now updated with armen's patch to enabled tp5*
-- Mike
On 2011-06-22, at 4:09 PM, beltzner wrote:
> Forerunner is sdwilsh. I think snarkfest is mconnor.
>
> cheers,
> mike
> On 22/06/2011 2:51 PM, "armenzg" <arm...@mozilla.com> wrote:
Armen Zambrano Gasparnian
Boris Zbarsky
> This has now been deployed to perf.snarkfest thanks to mconnor.
>
> I see it working now:
> http://perf.snarkfest.net/compare-talos/index.html?oldRevs=02f8dbc08005&newRev=b5dead378605&tests=tp4,tp4_memset,tp4_pbytes,tp4_rss,tp4_shutdown,tp4_xres,tp5,tp5_memset,tp5_pbytes,tp5_rss,tp5_shutdown,tp5_xres&submit=true
Awesome! Does tbpl also need updating to set the right URI? The one it
sends right now looks like this:
which excludes Tp5.
-Boris
Armen Zambrano Gasparnian
I have filed a bug and hopefully will be resolved in the next day or two.
https://bugzilla.mozilla.org/show_bug.cgi?id=666711
cheers,
Armen
Boris Zbarsky
> Thanks for pointing it out.
> I have filed a bug and hopefully will be resolved in the next day or two.
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=666711
Also, does Try Chooser (both the hook and the syntax builder) need updating?
-Boris
Armen Zambrano Gasparnian
> On 6/23/11 3:50 PM, Armen Zambrano Gasparnian wrote:
>> Thanks for pointing it out.
>> I have filed a bug and hopefully will be resolved in the next day or two.
>>
>> https://bugzilla.mozilla.org/show_bug.cgi?id=666711
BTW when we disable tp4 I would like to remove it as it will return
empty comparison.
>
> Also, does Try Chooser (both the hook and the syntax builder) need
> updating?
The Try Syntax gets updated automatically.
The Try Chooser web page needs to be updated manually:
http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36
>
> -Boris
Man this is finicky!
Thanks a lot Boris, doing things to the last details is very important.
cheers,
Armen
Boris Zbarsky
> The Try Chooser web page needs to be updated manually:
> http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36
Huh. So it's just called "tp" in the try syntax now and that triggers
both tp4 and tp5 at the moment but will switch to just tp5? Or just
triggers tp5 already?
> Man this is finicky!
> Thanks a lot Boris, doing things to the last details is very important.
No problem. If I think of anything else, I'll bring it up. ;)
Are we getting all this down in a checklist for next time we roll out a
new test suite, by the way?
-Boris
Armen Zambrano Gasparnian
> On 6/24/11 12:22 PM, Armen Zambrano Gasparnian wrote:
>> The Try Chooser web page needs to be updated manually:
>> http://hg.mozilla.org/users/lsblakk_mozilla.com/trychooser/rev/639ee77ced36
>>
>
> Huh. So it's just called "tp" in the try syntax now and that triggers
> both tp4 and tp5 at the moment but will switch to just tp5? Or just
> triggers tp5 already?
>
* The job got renamed from "tp4" to "tp".
* tp4 & tp5 are run within the "tp" job
Next week the "tp" job will only run "tp5".
>> Man this is finicky!
>> Thanks a lot Boris, doing things to the last details is very important.
>
> No problem. If I think of anything else, I'll bring it up. ;)
I am glad you do! :)
>
> Are we getting all this down in a checklist for next time we roll out a
> new test suite, by the way?
I added a note on the bug and added a checklist on the a-team's wiki page.
https://bugzilla.mozilla.org/show_bug.cgi?id=664831#c4
https://wiki.mozilla.org/Auto-tools/staging#Newer_suites
>
> -Boris
Armen Zambrano Gasparnian
The tbpl and compare-talos issues were ironed out.
This is also to be announced on the platform meeting tomorrow.
For more details please read:
http://armenzg.blogspot.com/2011/06/disabling-tp4.html
cheers,
Armen
Robert O'Callahan
> We are still on schedule to disable tp4 on Thursday June 30th.
>
> This was brought up at yesterday's developer meeting and no one raised
> any questions or concerns.
> I will raise this up on next Tuesday meeting to make sure everyone is
> aware.
>
> If you have any comments about it or find a bug please let us know on
> bug 661010.
>
Just so you know, I am currently working on a regression that was reported
for Tp4 but not Tp5. See my email(s) to dev.tree-management.
Rob
--
"If we claim to be without sin, we deceive ourselves and the truth is not in
us. If we confess our sins, he is faithful and just and will forgive us our
sins and purify us from all unrighteousness. If we claim we have not sinned,
we make him out to be a liar and his word is not in us." [1 John 1:8-10]
Armen Zambrano Gasparnian
> On Thu, Jun 23, 2011 at 1:21 AM, armenzg<arm...@mozilla.com> wrote:
>
>> We are still on schedule to disable tp4 on Thursday June 30th.
>>
>> This was brought up at yesterday's developer meeting and no one raised
>> any questions or concerns.
>> I will raise this up on next Tuesday meeting to make sure everyone is
>> aware.
>>
>> If you have any comments about it or find a bug please let us know on
>> bug 661010.
>>
>
> Just so you know, I am currently working on a regression that was reported
> for Tp4 but not Tp5. See my email(s) to dev.tree-management.
>
> Rob
what you need until you figure things out?
cheers,
Armen
Armen Zambrano Gasparnian
>
>> We are still on schedule to disable tp4 on Thursday June 30th.
>>
>> This was brought up at yesterday's developer meeting and no one raised
>> any questions or concerns.
>> I will raise this up on next Tuesday meeting to make sure everyone is
>> aware.
>>
>> If you have any comments about it or find a bug please let us know on
>> bug 661010.
>>
>
> Just so you know, I am currently working on a regression that was reported
> for Tp4 but not Tp5. See my email(s) to dev.tree-management.
>
> Rob
Boris Zbarsky
> If we keep on running it on mozilla-inbound and try would it give you
> what you need until you figure things out?
Well, an additional interesting question is why Tp5 doesn't show the
regression....
-Boris
Armen Zambrano Gasparnian
On another note, I can find a regression that tp5 found but not tp4:
Search for "Talos Regression :( Tp5 (Private Bytes) increase 3.74% on
MacOSX 10.6.2 Firefox"; is this right?
From looking at the email regression notifications I have seen that
every time a regression is found there is around a ~2% difference [1].
Considered that, the regression that roc is looking into is 2.68% for
tp4 and the tp5 difference could be around 0.68% or less (or even more
if my theory fails).
I am CCing other people that could answer this question.
Also note that these are two different sets of pages and perhaps the
newer set of pages can have pages that have intentionally made so they
are harder to run slow no matter what browser runs them. (NOTE:
Hypothesis of a non-expert in the field of browser performance).
For example: [1]
- 6.97%/5.97%
- 5.14%/3.8%
- 145%/131% (14 points difference but we are also talking about 3 digits)
Robert O'Callahan
arm...@mozilla.com> wrote:
> If we keep on running it on mozilla-inbound and try would it give you what
> you need until you figure things out?
>
Yes, although it's a concern that Tp5 did not find this regression.
Armen Zambrano Gasparnian
> On Thu, Jun 30, 2011 at 2:22 AM, Armen Zambrano Gasparnian
>
> If we keep on running it on mozilla-inbound and try would it give
> you what you need until you figure things out?
>
>
Tp4 did not.
If we had Tp2 and Tp3 I bet we could find instances that one of them
would regress while the others wouldn't.
>
> Rob
> --
> "If we claim to be without sin, we deceive ourselves and the truth is
> not in us. If we confess our sins, he is faithful and just and will
> forgive us our sins and purify us from all unrighteousness. If we
> claim we have not sinned, we make him out to be a liar and his word is
> not in us." [1 John 1:8-10]
--
Armen Zambrano Gasparnian (armenzg)
Mozilla Corp Release Engineer
~ Jesus Christ is my Lord
John O'Duinn
Can you give details in
https://bugzilla.mozilla.org/show_bug.cgi?id=601798?
Alice and Armen can investigate whether this is:
* a flaw in tp5 (and should halt the tp5 rollout)
* an accurate tp5 and a bug in tp4 (not halt the tp5 rollout).
tc
John.
=====
On 6/28/11 8:38 PM, Robert O'Callahan wrote:
> On Thu, Jun 23, 2011 at 1:21 AM, armenzg <arm...@mozilla.com> wrote:
>
>> We are still on schedule to disable tp4 on Thursday June 30th.
>>
>> This was brought up at yesterday's developer meeting and no one raised
>> any questions or concerns.
>> I will raise this up on next Tuesday meeting to make sure everyone is
>> aware.
>>
>> If you have any comments about it or find a bug please let us know on
>> bug 661010.
>>
>
Robert O'Callahan
> Can you give details in
> https://bugzilla.mozilla.org/show_bug.cgi?id=601798?
>
> Alice and Armen can investigate whether this is:
> * a flaw in tp5 (and should halt the tp5 rollout)
> * an accurate tp5 and a bug in tp4 (not halt the tp5 rollout).
>
Let me figure out what the regression is, then I'll report.
Robert O'Callahan
arm...@mozilla.com> wrote:
> On 11-06-29 5:38 PM, Robert O'Callahan wrote:
>
> On Thu, Jun 30, 2011 at 2:22 AM, Armen Zambrano Gasparnian <
> arm...@mozilla.com> wrote:
>
>> If we keep on running it on mozilla-inbound and try would it give you what
>> you need until you figure things out?
>>
>
> Yes, although it's a concern that Tp5 did not find this regression.
>
> As per my comment, I found an instance of Tp5 finding a regression while
> Tp4 did not.
> If we had Tp2 and Tp3 I bet we could find instances that one of them would
> regress while the others wouldn't.
>
Yes, I'm sure that's true. I'm not saying we shouldn't retire Tp4, but I
want to understand the issue.
John O'Duinn
On 6/29/11 5:08 PM, Robert O'Callahan wrote:
> On Thu, Jun 30, 2011 at 12:02 PM, John O'Duinn <jod...@mozilla.com
> <mailto:jod...@mozilla.com>> wrote:
>
> Can you give details in
> https://bugzilla.mozilla.org/show_bug.cgi?id=601798?
>
> Alice and Armen can investigate whether this is:
> * a flaw in tp5 (and should halt the tp5 rollout)
> * an accurate tp5 and a bug in tp4 (not halt the tp5 rollout).
>
>
> Let me figure out what the regression is, then I'll report.
>
> Rob
Thanks Rob. If I read your other post on thread correctly, you are still
ok with us disconnecting TP4 in production tomorrow?
If you want us to leave TP4 running while you investigate, please
comment in the bug sometime today, and we'll reschedule if it helps with
your investigations...
tc
John.
Robert O'Callahan
If it's still running on try, I'll be OK.
Robert O'Callahan
http://tbpl.mozilla.org/?tree=Try&rev=a291ec51b00e
I can't track down my regression anymore :-(.
Nick Thomas
> Looks like it stopped running on try:
> http://tbpl.mozilla.org/?tree=Try&rev=a291ec51b00e
I think you need to use this:
try: -b o -p macosx64 -u none -t tp4
or if you want both tp4 and tp5
try: -b o -p macosx64 -u none -t tp,tp4
Cheers,
Nick
Robert O'Callahan
Armen Zambrano Gasparnian
> Thanks!
>
> Rob
phew... I am glad it was just the syntax.
Thanks Nick!