Migrating to Win64 rev2 machines

John Hopkins

unread,

Oct 9, 2013, 5:22:52 PM10/9/13

to dev.platform, sher...@mozilla.com

Release Engineering is migrating the Windows 64-bit _build_ machines to
a new, managed machine image which will decrease the amount of
administrative overhead needed to deploy new instances, make image
changes, etc.

It uses the same build toolchain as the current Windows 64-bit images
(MSVC10) but located at different paths so you'll notice conditional
checks like
http://hg.mozilla.org/mozilla-central/file/a141e39bf6da/build/win64/mozconfig.vs2010#l2
which will be uplifted.

The 'cedar' project branch has been building successfully using the new
image. The plan is to migrate other branches over the next few weeks in
this order:
- remaining project branches
- mozilla-central
- mozilla-inbound, b2g-inbound
- try
- aurora
- beta
- release, mozilla-b2g18*
- esr

Wait times may increase for a short time on days that we cut over to the
new build machines (since we are reimaging existing machines in chunks).
We'll be keeping an eye on those wait times and adjusting the number of
machines cut over if needed. Thunderbird comm-* branches will be cut
over at the same time as their mozilla-* counterpart.

The project branches will be cut over tomorrow and we'll let those build
over the weekend before cutting over mozilla-central. I will send an
email notification on this thread before and after a cutover so everyone
knows the current status.

Bug 781277 tracks this effort. If you have any questions or concerns,
please let me know.

Thanks,
John

Jet Villegas

unread,

Oct 9, 2013, 5:35:51 PM10/9/13

to John Hopkins, sher...@mozilla.com, dev.platform

I'm currently looking for someone to green up the tests on Win64. Can we dedicate one machine instance (I'm assuming this is on AWS?) for the purpose of advancing this greenery project? Also, who is the point-person on Release Engineering that we should pull into the testing discussion?

Thanks,

--Jet

_______________________________________________
dev-platform mailing list
dev-pl...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Ben Hearsum

unread,

Oct 9, 2013, 5:50:52 PM10/9/13

to

Hey Jet,

I'd say that this greenery project is completely aside from the work
John is doing -- build and tests machines are generally separate.

Are you looking for you (or someone else) to have access to a machine to
green up tests on your own, or looking to have tests run as part of the
normal automation (or maybe on Try?)?

- Ben

Jet Villegas

unread,

Oct 9, 2013, 6:29:53 PM10/9/13

to Ben Hearsum, dev-pl...@lists.mozilla.org

I think we could have it happen in this order:

1. a way to green up tests on a single machine hooked up to a build server
2. put this on Try
3. put this on production automation

Of course, it would be awesome if we could skip #1 and go to #2 directly as we green up the tests, as this will open it up to multiple contributors for the test greening.

Ben Hearsum

unread,

Oct 9, 2013, 6:49:26 PM10/9/13

to

On 10/09/13 06:29 PM, Jet Villegas wrote:
> I think we could have it happen in this order:
>
> 1. a way to green up tests on a single machine hooked up to a build server
> 2. put this on Try
> 3. put this on production automation
>
> Of course, it would be awesome if we could skip #1 and go to #2 directly as we green up the tests, as this will open it up to multiple contributors for the test greening.

Single dedicated machines aren't really a thing for us, so #1 isn't
actually doable.

While research what it would take to do #2 I noticed that we already
have 64-bit Windows tests running over on the Date branch! I'm not fully
in the loop on this, but I believe this was revived within the last
couple of months. Results are over here: https://tbpl.mozilla.org/?tree=Date

We could probably make it possible to run them on Try, but I should
defer to Chris AtLee or John Hopkins at this point, as they've got much
more context on this than me.

- Ben

Ben Hearsum

unread,

Oct 9, 2013, 7:15:04 PM10/9/13

to

On 10/09/13 06:49 PM, Ben Hearsum wrote:
> We could probably make it possible to run them on Try, but I should
> defer to Chris AtLee or John Hopkins at this point, as they've got much
> more context on this than me.

OK, one more reply from: These _are_ enabled on Try, but the try syntax
to use them is undocumented. Pushing with "try: -b o -p win64 -u
all[win64_vm] -t none" should get you a full suite of tests. You can
also use the normal subselection by replacing "all" with "reftest" or
similar.

- Ben

Jet Villegas

unread,

Oct 9, 2013, 7:33:16 PM10/9/13

to Ben Hearsum, dev-pl...@lists.mozilla.org

Excellent.

Now to find willing volunteer(s.) Windows 64 represents a key platform for us now that we've opened up Firefox for high-performance graphics via WebGL and Canvas. Freeing up the CPU by going to the GPU exposes the next bottleneck: Memory Bandwidth.

Can you help get the test failures fixed? Please reply to me if you can contribute here.

Thanks,

--Jet

----- Original Message -----
From: "Ben Hearsum" <bhea...@mozilla.com>
To: dev-pl...@lists.mozilla.org
Sent: Wednesday, October 9, 2013 4:15:04 PM
Subject: Re: Migrating to Win64 rev2 machines

Kyle Huey

unread,

Oct 9, 2013, 7:35:45 PM10/9/13

to Jet Villegas, Ben Hearsum, Benjamin Smedberg, dev-platform

Did we start caring about Win64 again recently?

+bsmedberg

- Kyle

On Wed, Oct 9, 2013 at 7:33 PM, Jet Villegas <j...@mozilla.com> wrote:

> Excellent.
>
> Now to find willing volunteer(s.) Windows 64 represents a key platform for
> us now that we've opened up Firefox for high-performance graphics via WebGL
> and Canvas. Freeing up the CPU by going to the GPU exposes the next
> bottleneck: Memory Bandwidth.
>
> Can you help get the test failures fixed? Please reply to me if you can
> contribute here.
>

> Thanks,
>
> --Jet
>
> ----- Original Message -----
> From: "Ben Hearsum" <bhea...@mozilla.com>
> To: dev-pl...@lists.mozilla.org
> Sent: Wednesday, October 9, 2013 4:15:04 PM
> Subject: Re: Migrating to Win64 rev2 machines
>

John Hopkins

unread,

Oct 9, 2013, 10:17:30 PM10/9/13

to Jet Villegas, sher...@mozilla.com, dev.platform

Hi, Jet:

On 13-10-09 5:35 PM, Jet Villegas wrote:
> I'm currently looking for someone to green up the tests on Win64. Can we dedicate one machine instance (I'm assuming this is on AWS?) for the purpose of advancing this greenery project? Also, who is the point-person on Release Engineering that we should pull into the testing discussion?

This switchover is specifically for the build machines, not test
machines. You probably want bug 886640.

Thanks,
John

jmaher

unread,

Oct 10, 2013, 6:26:31 AM10/10/13

to

it takes a lot of work to get green tests on a new platform. I spent the better part of December to March getting tests green on Ubuntu.

If these are in vms, we wouldn't have the graphics cards mentioned above. In fact we might not have the ability to run the webgl tests.

Armen Zambrano G.

unread,

Oct 10, 2013, 9:11:49 AM10/10/13

to Kyle Huey, Jet Villegas, Ben Hearsum, Benjamin Smedberg, dev-platform, Vladimir Vukicevic

Yes, we have.

+vlad

Armen Zambrano G.

unread,

Oct 10, 2013, 9:11:49 AM10/10/13

to Kyle Huey, Jet Villegas, Ben Hearsum, Benjamin Smedberg, dev-platform, Vladimir Vukicevic

Yes, we have.

+vlad

On 2013-10-09 7:35 PM, Kyle Huey wrote:

John Hopkins

unread,

Oct 10, 2013, 4:25:18 PM10/10/13

to dev.platform, sher...@mozilla.com

Project branches are cutting over to the new win64-rev2 build machines.

John

John Hopkins

unread,

Oct 11, 2013, 10:03:53 AM10/11/13

to dev.platform, sher...@mozilla.com

On 13-10-10 4:25 PM, John Hopkins wrote:
>
> Project branches are cutting over to the new win64-rev2 build machines.

Project branch builds have run overnight on the new build machines and
are looking good.

We will continue migrating more branches next week.