Using different hardware for different platforms on Talos
Jeff Muizelaar
-Jeff
Boris Zbarsky
> I've talked with some people from Releng about switching our linux and or windows talos machines from mac minis to different hardware. This will free up more mac minis for OS X talos and let us buy hardware that's more flexible. I'm wondering if anyone has any examples of situations where they have compared results across operating systems, what they wanted to use the results for and how useful they were.
I thought the main reason we were using minis was because they were
simple and not too fast. There's no a priori reason to use same
hardware across OSes, afaik.
-Boris
Mike Hommey
I've found that comparing talos results across platforms can be
interesting information, and that can only be done on the same
hardware. However, while I do believe it's an interesting information to
have occasionally (most notably, when switching toolchains, like the
recent switch to gcc 4.5), I don't think it's needed for every single
check-in.
Mike
Justin Dolske
> I'm wondering if anyone has any examples of situations where they have compared results across operating systems, what they wanted to use the results for and how useful they were.
I can think of a couple useful examples:
1) "OMG Firefox is faster on $OS_1 than $OS_2, why do you hate $OS_2"?
It would be good to have some ability to compare across platforms to see
if there's an issue.
2) Platform-specific regression splits. It's not unheard of for a patch
to be a perf win on one platform, but a regression on another. It's
harder to judge "small regression on X, big win on Y" if the hardware is
different.
I think both of these could be worked around if we had good reason to,
though.
What's the motivation here? Just faster/cheaper hardware for running
tests on Windows and Linux, and keep Mac tests on pricey Apple hardware?
Justin
Mike Shaver
> 1) "OMG Firefox is faster on $OS_1 than $OS_2, why do you hate $OS_2"? It
> would be good to have some ability to compare across platforms to see if
> there's an issue.
IME, even with something as quantitative and steady as SunSpider
numbers, you're already boned here. Number of cores, amount of RAM,
OS version, drivers, cache size, disk speed, etc are going to make it
basically worthless to compare OS-X-on-Mini to Windows-on-Mini (which
is representative of 0.00% of our users).
(When we have seen these cases in the past, people have just run the
app on different platforms, often different people commenting in the
same bug. If it's significant enough to be visible to the human eye,
it'll probably show up that way.)
> 2) Platform-specific regression splits. It's not unheard of for a patch to
> be a perf win on one platform, but a regression on another. It's harder to
> judge "small regression on X, big win on Y" if the hardware is different.
Similarly.
We've seen deltas of different magnitudes on different platforms in
the past, but we really didn't care whether it was the OS that was
different, we just attacked each one until they were all gone or small
enough to live with. You don't need the same hardware for that.
(If you need it for a specific case and you suspect that it might be
OS related, and being OS related would change your decisions somehow,
then bootcamp your MBP while you work on it.)
Mike
armenzg
> On 5/19/11 10:24 AM, Jeff Muizelaar wrote:
>
> > I'm wondering if anyone has any examples of situations where they have compared results across operating systems, what they wanted to use the results for and how useful they were.
>
> I can think of a couple useful examples:
>
> 1) "OMG Firefox is faster on $OS_1 than $OS_2, why do you hate $OS_2"?
> It would be good to have some ability to compare across platforms to see
> if there's an issue.
>
> 2) Platform-specific regression splits. It's not unheard of for a patch
> to be a perf win on one platform, but a regression on another. It's
> harder to judge "small regression on X, big win on Y" if thehardwareis
> different.
>
> I think both of these could be worked around if we had good reason to,
> though.
nice information but not something that is needed for day to day work.
If such experiment would be needed we could always buy a 7-10 machines
to get such information.
>
> What's the motivation here? Just faster/cheaperhardwarefor running
> tests on Windows and Linux, and keep Mac tests on pricey Applehardware?
>
The problem with the minis is that we get locked down with them every
time Apple stops producing a certain version.
Removing this hard requirement would give releng and IT the ability of
not capping our testing capacity so easily.
It's not the same to buy 560 same-hardware-machines than let's say 80
for a given OS.
> Justin