Speed test of building Chrome under Windows

1,678 views
Skip to first unread message

Arthur Hsu

unread,
Apr 11, 2011, 5:42:22 PM4/11/11
to chromi...@chromium.org
f you don't compile Chrome under Windows, feel free to ignore this e-mail.

Summary: use a separate disk to host Chrome source and reduce VC concurrent job count to (number of physical cores - 1) from default (number of logical cores) reduces compilation time on my machine from 1:23:15 to 49:30 (40.5% speed up)

Machine configuration:
Dell Precision T3500, Xeon X5...@2.67 GHz, 12GB of RAM, initially 500GB SATA HD, BitLock disabled
Windows 7 Enterprise
Visual Studio 2008 SP1

Observations:
  • By default, VC sets number of logical cores as max number of parallel project builds (Tools->Options->Project and Solutions->Build and Run).  In my case, it's 12 (6 physical cores and hyper thread)
  • The "max number of parallel project builds" is actually number of VCBuildHelper.exe instances, not the number of concurrent cl.exe.  Similar things happened in VC 2010: this number indicates number of MSBuild.exe instances, not cl.exe.  See attached picture.  
    • Also, each link.exe in the picture is committing high amount of memory (~1G each) and thus 12 instance will invite very high page fault counts.
    • mspdbsrv.exe is the service that synchronize and generate pdb usage during compile time.  It's also obvious from the picture that this is the single bottleneck of compilation speed (high I/O bytes, high memory usage (the working set), ridiculous page fault counts).  

Capture-cl.PNG

Countermeasures:
  • To handle mspdbsrv.exe issue, we need to separate the swap from source so that the I/O and page fault pressure can be separated into two different I/O channels (in my case, a different SATA port).  This is solved by adding a new disk to the system and make sure it's not used for swap.  (Control Panel->System and Security->System->Advanced->Performance/Settings->Advanced->Virtual memory/Change)
  • Reduce the pressure of Windows Virtual Memory manager by reducing concurrent jobs.  In my case, 5 (number of physical cores - 1) is taken from rules of thumb.  It may not be optimal but works very well in my case.
Actual results:

Time required to build chrome.sln (DEBUG|Win32) reduced from 1:23:15 to 49:30.



Capture-cl.PNG

Peter Kasting

unread,
Apr 11, 2011, 5:47:09 PM4/11/11
to arth...@chromium.org, chromi...@chromium.org
On Mon, Apr 11, 2011 at 2:42 PM, Arthur Hsu <arth...@chromium.org> wrote:
Summary: use a separate disk to host Chrome source and reduce VC concurrent job count to (number of physical cores - 1) from default (number of logical cores) reduces compilation time on my machine from 1:23:15 to 49:30 (40.5% speed up)

Currently http://dev.chromium.org/developers/how-tos/build-instructions-windows instructs people to reduce the number of parallel builds based on available RAM and suggests 8 builds for 12 GB.  I'm currently using 6 for my Z600 with 12 GB.  Perhaps this number needs more adjusting or perhaps your and my values are slightly too low.

PK

Arthur Hsu

unread,
Apr 11, 2011, 5:56:54 PM4/11/11
to Peter Kasting, chromi...@chromium.org
8 builds for 12GB might be a bit bold.  Worst case is 8 linkers each takes ~1G of RAM, and thus only 4G to spare to the rest of system.  Unfortunately, mspdbsrv is not that well behaved and it might take another 2 to 4G, leaving rest of the system underwater.

The other thing to consider is the number of cl.exe spawned.  In my first test, I've spotted over 100 cl.exe instances being created when the number is set to 10.  The picture is not captured because I'm not that fast at clipping the screenshot and the system was not responsive, either.  Anyway lots of CPU cycles will be wasted for context switching among that many cl.exe instances and thus we shall drive the number to a lower range (e.g. 5 or 6).

-Arthur

Scott Violet

unread,
Apr 11, 2011, 6:01:30 PM4/11/11
to arth...@chromium.org, chromi...@chromium.org
On Mon, Apr 11, 2011 at 2:42 PM, Arthur Hsu <arth...@chromium.org> wrote:
The killer for me is touch a file (just a .cc) and build chrome. Currently it takes around 4 minutes! Compare this to 40 seconds on my linux box... Both machines are the same (z600, two hard drives) but the windows side has Sophos and Bit9 to contend with.

  -Scott

Capture-cl.PNG

Arthur Hsu

unread,
Apr 11, 2011, 6:20:24 PM4/11/11
to Scott Violet, chromi...@chromium.org
How is the touch done?  Invoking cygwin /usr/bin/touch?  That will definitely take a while since it needs to pass cygwin process launch hack, loads all DLLs, run by Sophos scan, and Bits decryption.  This command shall do it much faster

cmd /c "echo. 2>ph.txt"

Replace the ph.txt with your file path.

-Arthur
Capture-cl.PNG

Scott Violet

unread,
Apr 11, 2011, 6:29:42 PM4/11/11
to Arthur Hsu, chromi...@chromium.org
By touch I just meant edit the file then do a build. I wasn't timing the actual touch part of it, just the compile.

  -Scott
Capture-cl.PNG

Ryan Norton

unread,
Apr 12, 2011, 11:01:13 AM4/12/11
to Chromium-dev
On Apr 11, 2:42 pm, Arthur Hsu <arthur...@chromium.org> wrote:
>    - The "max number of parallel project builds" is actually number of
>    VCBuildHelper.exe instances, not the number of concurrent cl.exe.  Similar
>    things happened in VC 2010: this number indicates number of MSBuild.exe
>    instances, not cl.exe.  See attached picture.

The number of cl/link instances is controlled by the /MP switch and
set to use all your effective processors by default if you don't
specify a number - which is what chromium currently does; see
http://msdn.microsoft.com/en-us/library/bb385193.aspx. You can
override it by setting 'msvs_multi_core_compile': 0 and manually
setting it yourself through 'msvs_settings' -> 'VCCLCompilerTool' ->
'AdditionalOptions': ['/MP***']; *** would be the max # of cl.exe etc.
processes you want running per project.

For example, your include.gypi file could be this for limiting it to
12 processes per project:
-------------
{
'variables': {
'msvs_multi_core_compile': 0,
},
'target_defaults': {
'msvs_settings': {
'VCCLCompilerTool': {
'AdditionalOptions': ['/MP12'],
},
},
},
}
---------------

Carlos Pizano

unread,
Apr 12, 2011, 10:29:09 PM4/12/11
to Chromium-dev, Ryan Norton
I report 2.5 minutes with VS2010 for a single .cc touch and debug
build.

BTW, VS2010 SP1 builds chrome just fine. I do have the "ultimate"
edition.



On Apr 12, 8:01 am, Ryan Norton <rnor...@gmail.com> wrote:
> On Apr 11, 2:42 pm, Arthur Hsu <arthur...@chromium.org> wrote:
>
> >    - The "max number of parallel project builds" is actually number of
> >    VCBuildHelper.exe instances, not the number of concurrent cl.exe.  Similar
> >    things happened in VC 2010: this number indicates number of MSBuild.exe
> >    instances, not cl.exe.  See attached picture.
>
> The number of cl/link instances is controlled by the /MP switch and
> set to use all your effective processors by default if you don't
> specify a number - which is what chromium currently does; seehttp://msdn.microsoft.com/en-us/library/bb385193.aspx.  You can

Naoki Takano

unread,
Apr 13, 2011, 3:22:10 PM4/13/11
to Chromium-dev
Hi,

I mainly use Linux dev environment because build time is faster than Windows.

As you know, link time on Windows is extremely slow!!
So I always disabled pdb generation and no opt ref in Release mode manually.

I don't know why the Release setting still enable pdb generation as
default though.
Once disable them, mspdbsrv.exe consumes less memories and link time is shorten.
Of course, it might help for debugger, but I don't use debugger these days.

Thanks,

> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
>    http://groups.google.com/a/chromium.org/group/chromium-dev
>

Sanjeev Radhakrishnan

unread,
Apr 13, 2011, 3:38:05 PM4/13/11
to takano...@gmail.com, chromium-dev
PDB generation is needed for making crash reports have readable stack traces (and for debugging, of course).

Steve VanDeBogart

unread,
Apr 13, 2011, 3:41:55 PM4/13/11
to sanj...@chromium.org, takano...@gmail.com, Chromium-dev
Should we have a Release-NoSymbol target for faster development builds?

--
Steve

Naoki Takano

unread,
Apr 13, 2011, 4:19:30 PM4/13/11
to Chromium-dev
Of course, I know PDB generation purpose.

But I agree with Steve. Of course, we can change manually setting, but
it's convenient if we have.

Thanks,

Brian Salomon

unread,
Apr 13, 2011, 4:44:17 PM4/13/11
to takano...@gmail.com, Chromium-dev
I've been using the Multi-DLL build and used a tool to boost my system file cache to 2GB. If I touch a file in WebKit (GraphicsContext3DChromium.cpp) and build the Chrome target in Debug from VS 2008 it takes 10 seconds or less to complete. This is with a hot cache (I've built several times recently).

Brian

Darin Fisher

unread,
Apr 13, 2011, 4:58:15 PM4/13/11
to bsal...@google.com, takano...@gmail.com, Chromium-dev
yay, multi-dll build!!!  we need to carve up chrome into more dlls to increase these benefits :)

Scott Violet

unread,
Apr 13, 2011, 5:09:49 PM4/13/11
to bsal...@google.com, takano...@gmail.com, Chromium-dev
SWEETNESS!
Please share with the rest of us how to set this up.

-Scott

On Wed, Apr 13, 2011 at 1:44 PM, Brian Salomon <bsal...@google.com> wrote:

Brian Salomon

unread,
Apr 13, 2011, 5:20:23 PM4/13/11
to Scott Violet, takano...@gmail.com, Chromium-dev
Sure, this is all I did:

To boost the file cache I used this tool:
The max allowed seems to be 2GB. My Z600 system has 12GB of RAM.

To enable the Multi-DLL build I put:
{'variables': {'component': 'shared_library'}}
in c:\users\<my username>\.gyp\include.gypi
I had to blow-away my <chrome>\src\build\Debug and <chrome>\src\build\Release dirs before the first builds after switching to Multi-DLL.

I also have my chrome tree on a different drive than my OS.

Brian

Naoki Takano

unread,
Apr 13, 2011, 6:07:53 PM4/13/11
to Chromium-dev
Cool!!

I'll try it. Although my machine only has 4GB, I wish it will help...

Vangelis Kokkevis

unread,
Apr 19, 2011, 4:14:44 PM4/19/11
to Chromium-dev
Just wanted to mention that switching to the multi-dll build in this past couple of weeks has been a tremendous productivity boost.  Not only the code builds faster (the webkit library links in under 10 secs) but also now the VS debugger seems to be able to load and run chrome within seconds.  If windows is your primary development platform, do yourself a favor and give it a try!

Many kudos to Brian for bringing it up. 

Vangelis
PS I also used the utility that Brian mentions to set the file cache size to 2GB.  I don't know how much that plays into the speed improvement.

Mike Reed

unread,
Apr 19, 2011, 4:19:58 PM4/19/11
to vang...@google.com, Chromium-dev
What bot will exercise this? i.e. gcl try foo --bot=?

Lei Zhang

unread,
Apr 19, 2011, 4:21:43 PM4/19/11
to re...@google.com, Chromium-dev
I don't think we have shared configuration try bots for either Windows or Linux.

Vangelis Kokkevis

unread,
Apr 19, 2011, 4:32:51 PM4/19/11
to the...@chromium.org, re...@google.com, Chromium-dev

Vangelis Kokkevis

unread,
Apr 19, 2011, 4:33:48 PM4/19/11
to the...@chromium.org, re...@google.com, Chromium-dev
Never mind, I just saw you were talking about try bots....

Scott Violet

unread,
Apr 19, 2011, 5:00:31 PM4/19/11
to vang...@google.com, Chromium-dev
Agreed, but when the multi-dll build was broke for the better part of
a day last week I lost faith. Can we treat the multi-dll build like
any other compile bots?

-Scott

John Abd-El-Malek

unread,
Apr 19, 2011, 5:03:37 PM4/19/11
to s...@chromium.org, vang...@google.com, Chromium-dev
Does anyone know if there's a way to enable it per checkout?

Mike Reed

unread,
Apr 19, 2011, 5:16:27 PM4/19/11
to s...@chromium.org, vang...@google.com, Chromium-dev
+1 for adding multi-dlls to the try suite

Ricardo Vargas

unread,
Apr 19, 2011, 6:41:36 PM4/19/11
to jabde...@google.com, John Abd-El-Malek, s...@chromium.org, vang...@google.com, Chromium-dev
On Tue, Apr 19, 2011 at 2:03 PM, John Abd-El-Malek <j...@chromium.org> wrote:
Does anyone know if there's a way to enable it per checkout?

Edit build/common.gypi and change the value of component to be
   'component%': 'shared_library'

(and remember not to commit that)

John Abd-El-Malek

unread,
Apr 19, 2011, 7:09:16 PM4/19/11
to Ricardo Vargas, s...@chromium.org, vang...@google.com, Chromium-dev
Thanks!

Darin Fisher

unread,
Apr 19, 2011, 7:15:03 PM4/19/11
to rva...@chromium.org, jabde...@google.com, John Abd-El-Malek, s...@chromium.org, vang...@google.com, Chromium-dev
On Tue, Apr 19, 2011 at 3:41 PM, Ricardo Vargas <rva...@chromium.org> wrote:


On Tue, Apr 19, 2011 at 2:03 PM, John Abd-El-Malek <j...@chromium.org> wrote:
Does anyone know if there's a way to enable it per checkout?

Edit build/common.gypi and change the value of component to be
   'component%': 'shared_library'

(and remember not to commit that)


^^^ trick:  "gcl change do_not_commit", and add that file to it.  now that file will never accidentally appear in another CL :)

John Bauman

unread,
Apr 19, 2011, 7:21:27 PM4/19/11
to da...@google.com, rva...@chromium.org, jabde...@google.com, John Abd-El-Malek, s...@chromium.org, vang...@google.com, Chromium-dev
"git update-index --assume-unchanged build/common.gypi" also helps, though I'm not sure how reliably.

Ricardo Vargas

unread,
Apr 19, 2011, 8:43:09 PM4/19/11
to Darin Fisher, jabde...@google.com, John Abd-El-Malek, s...@chromium.org, vang...@google.com, Chromium-dev
On Tue, Apr 19, 2011 at 4:15 PM, Darin Fisher <da...@chromium.org> wrote:
On Tue, Apr 19, 2011 at 3:41 PM, Ricardo Vargas <rva...@chromium.org> wrote:


On Tue, Apr 19, 2011 at 2:03 PM, John Abd-El-Malek <j...@chromium.org> wrote:
Does anyone know if there's a way to enable it per checkout?

Edit build/common.gypi and change the value of component to be
   'component%': 'shared_library'

(and remember not to commit that)


^^^ trick:  "gcl change do_not_commit", and add that file to it.  now that file will never accidentally appear in another CL :)


I forgot to mention that you have to do the same thing to src/native_client/build

Ricardo Vargas

unread,
Apr 19, 2011, 8:44:30 PM4/19/11
to Darin Fisher, jabde...@google.com, John Abd-El-Malek, s...@chromium.org, vang...@google.com, Chromium-dev
On Tue, Apr 19, 2011 at 5:43 PM, Ricardo Vargas <rva...@chromium.org> wrote:


On Tue, Apr 19, 2011 at 4:15 PM, Darin Fisher <da...@chromium.org> wrote:
On Tue, Apr 19, 2011 at 3:41 PM, Ricardo Vargas <rva...@chromium.org> wrote:


On Tue, Apr 19, 2011 at 2:03 PM, John Abd-El-Malek <j...@chromium.org> wrote:
Does anyone know if there's a way to enable it per checkout?

Edit build/common.gypi and change the value of component to be
   'component%': 'shared_library'

(and remember not to commit that)


^^^ trick:  "gcl change do_not_commit", and add that file to it.  now that file will never accidentally appear in another CL :)


I forgot to mention that you have to do the same thing to src/native_client/build
 
src/native_client/build/common.gypi
Reply all
Reply to author
Forward
0 new messages