Windows Regression Test Platforms
sayrer
regression tests.
What are they?
Mike Beltzner
As I recall, and by current reckoning
P1 platforms:
- Windows XP SP3+
- Windows 2003 Server
- Windows Vista
- Windows 7
P2 platforms:
- Windows 2000
- Windows ME
(the P2 platforms are such because they are no longer nominally supported by Microsoft, and may soon be removed from our supported platforms list, so they shouldn't be given equal weight)
cheers,
mike
Jeff Muizelaar
On 13-Jul-10, at 2:56 PM, Mike Beltzner wrote:
> P2 platforms:
> - Windows 2000
PMAX_INT:
> - Windows ME
-Jeff
Armen Zambrano Gasparnian
Up until now we have been running unit tests on Windows 2003 VMs
(builder machines). We are moving away from them to run unit tests on
the minis: WinXP, Win7 and Win7 x64. We will do the complete switch once
we have all test suites down to zero perma-oranges.
We have switched (for mozilla-central only) in the last two weeks to run
unit tests on the minis:
* Rev3 WINNT 6.1 (Windows 7) (only optimized builds [1])
* Rev3 WINNT 6.1 x64 (Windows 7 x64) (run once a day [2] - all hidden)
* Rev3 WINNT 5.1 (Windows XP) (not running at all - [3])
[1] bug 562459 - can't run unit test for debug - help welcome
* we don't have OPSI to deploy VS to all Win OSes
[2] bug 565402 - add remaining tools before cloning pool of Win64 slaves
[3] bug 563036 - can't run opt unit tests on XP
To see the hidden unit test suites you can visit:
* http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox&noignore=1
Cheers,
Armen
John O'Duinn
On 7/13/10 11:56 AM, Mike Beltzner wrote:
> As I recall, and by current reckoning
>
> P1 platforms:
> - Windows XP SP3+
yes.
> - Windows 2003 Server
The plan of record is that this would change from "build and test" to
"build only", once we have Win7x32 tests run green.
> - Windows Vista
dropped in favour of Win7x32 at the end of 2009.
> - Windows 7
...and added:
- Windows7x64
> P2 platforms:
> - Windows 2000
> - Windows ME
> (the P2 platforms are such because they are no longer nominally supported by Microsoft, and may soon be removed from our supported platforms list, so they shouldn't be given equal weight)
What does P2 means in this context; we have no infrastructure running on
these OS. Should these be formally removed from the "supported OS" list?
> cheers,
> mike
tc
John.
L. David Baron
> > - Windows 2003 Server
> The plan of record is that this would change from "build and test" to
> "build only", once we have Win7x32 tests run green.
Presumably this change should wait until we have both the Win7-32
and the WinXP tests green? (Or maybe the WinXP tests alone would be
sufficient, given the similarity, but it sounds like they'll be up
after the Win7 tests anyway, so the difference doesn't really
matter.)
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
John O'Duinn
On 7/13/10 2:32 PM, L. David Baron wrote:
> On Tuesday 2010-07-13 14:25 -0700, John O'Duinn wrote:
>>> - Windows 2003 Server
>> The plan of record is that this would change from "build and test" to
>> "build only", once we have Win7x32 tests run green.
>
> Presumably this change should wait until we have both the Win7-32
> and the WinXP tests green?
This is not the plan of record. Of course, plans can be changed, but to
be clear, here is the current plan as I described it after today's
platform meeting:
1) Get tests running on Win7x32 was first priority.
2) Disable tests running on win2k3 build machines to improve win32 build
wait times.
3) I believe getting WinXP and Win7x64 tests running, and running green,
are required for shipping FF4.0, but are not required in order for us to
reduce duplicate load on the build machines.
Of course, having all OS is ideal. However, during this transition,
we're swapping out one less-common-user-OS for a commonly-used-user-OS,
so even this seems an improvement on our current situation, imho.
If these priorities are incorrect, please let me know and we can work
through pro-vs-con of alternatives.
tc
john.
=====
sayrer
>
> Of course, having all OS is ideal. However, during this transition,
> we're swapping out one less-common-user-OS for a commonly-used-user-OS,
> so even this seems an improvement on our current situation, imho.
This prioritization is unpleasant. But if none of the parameters here
can be changed, it is the right one.
- Rob
John O'Duinn
On 7/14/10 10:30 AM, Benjamin Smedberg wrote:
> On 7/14/10 12:27 PM, Christian Legnitto wrote:
>
>> I don't know if you saw this last week:
>>
>> http://windowsteamblog.com/windows/b/bloggingwindows/archive/2010/07/08/64-bit-momentum-surges-with-windows-7.aspx
>>
>>
>> 46% of people on Windows 7 are using the 64 bit version. Of course,
>> they can run the 32 bit version of Firefox if need be.
>
> Yeah, it seems mostly irrelevant to our decision here:
>
> * We don't know how much of a performance win we'll get (don't have
> Talos numbers yet).
There are Talos numbers for win64 on graphserver, just like for linux64,
osx10.6x64, and the other 32bit OS.
Note: we only do once-per-day win64 builds, not yet build-on-checkin.
This is because we still only have few machines on win64, while we sort
out compiler versions. Of course, if there is a specific changeset
during the day that you want built on win64, ping us and we can trigger
that for you.
> * We do know it will be a fairly large hunk of work to get 32-on-64 plugins
> * Many of the people who can do that work are busy chasing regressions,
> implementing the new theme, etc.
>
> I admit I am also worried about the effects of adding three new
> platforms to our release and QA matrix in a single major cycle, but I'll
> let releng and QA speak to that more.
RelEng is expecting to ship FF4.0 on win64, as well as on linux64,
10.6x64bit. This was per request from shaver+beltzner earlier this year.
Of course, there are many other factors involved in a ship decision, but
thats the RelEng point-of-view.
tc
John.
=====
> Seems like we should focus on the
> two new platforms we've already committed to.
>
> --BDS
> _______________________________________________
> dev-planning mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-planning
Benjamin Smedberg
>> * We don't know how much of a performance win we'll get (don't have
>> Talos numbers yet).
> There are Talos numbers for win64 on graphserver, just like for linux64,
> osx10.6x64, and the other 32bit OS.
bhearsum noted these too, but they don't actually help make our decision
because we can't compare them against the 32-bit builds running on the same
OS. I'll file a ticket to see if we can get good comparison runs. They also
seem really noisy, but I'm not sure whether that's just because I'm reading
them wrong.
--BDS
John O'Duinn
On 7/14/10 12:58 PM, Benjamin Smedberg wrote:
> On 7/14/10 2:39 PM, John O'Duinn wrote:
>
>>> * We don't know how much of a performance win we'll get (don't have
>>> Talos numbers yet).
>> There are Talos numbers for win64 on graphserver, just like for linux64,
>> osx10.6x64, and the other 32bit OS.
>
> bhearsum noted these too, but they don't actually help make our decision
> because we can't compare them against the 32-bit builds running on the
> same OS.
Oh, I think I follow you now. We currently do:
* talos for 32bit builds on 32bit OS
* talos for 64bit builds on 64bit OS
In addition to this, you want:
* talos for 32bit builds on 64bit OS
Is that correct?
> I'll file a ticket to see if we can get good comparison runs.
For now, comparing the results for 32bit-build-on-32bit-OS with results
for 64bit-build-on-64bit-OS seems useful also?
> They also seem really noisy, but I'm not sure whether that's just
> because I'm reading them wrong.
> --BDS
tc
John.
Benjamin Smedberg
> In addition to this, you want:
> * talos for 32bit builds on 64bit OS
>
> Is that correct?
Yes. Not permanently, but just for the purposes of figuring out what the
performance win of a 64-bit firefox is.
> For now, comparing the results for 32bit-build-on-32bit-OS with results
> for 64bit-build-on-64bit-OS seems useful also?
Probably not. Or at least, I can't see how you'd turn that into actionable data.
--BDS
Ben Hearsum
Hash: SHA1
On 10-07-14 1:47 PM, Mike Shaver wrote:
> On Wed, Jul 14, 2010 at 1:44 PM, Ben Hearsum <bhea...@mozilla.com> wrote:
>> Actually...we do have some numbers. The full set of tests is run once
>> per day:
>> http://tinyurl.com/357worw
>
> That graph may not be sufficient for answering the performance
> questions we have.
>
I did some comparisons of 32-bit build on 64-bit machine and 64-bit
build on 64-bit machine today. Results are here:
https://bugzilla.mozilla.org/show_bug.cgi?id=578767#c2
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
iEYEARECAAYFAkw/bIsACgkQJE25Np0n+Nu0SgCfTn0fxfNTEGJ3YR0kxhFG70Os
g8IAoKG7DjRvPwn+gMuuz2GYME0ug8h8
=WkGC
-----END PGP SIGNATURE-----
Armen Zambrano Gasparnian
* build against MSVC10
* they don't have jemalloc
* they don't have PGO
I am moving now to:
* MSVC9 (where we can have jemalloc)
* enable jemalloc
* hopefully PGO just works (anyone who could help me with this by trying
I would greatly appreciate it)
Cheers,
Armen