Problems compiling Chrome/Blink on Windows

[Yngve Pettersen]

Hello all,

A couple of weeks ago I decided to have a look at the Chromium source. Unfortunately, after spending 7 days on it, I only got as far that I had a functional build (I did manage to get a running, non-functional build after 5 days, the functionality issues were due to the main issue discussed below).

For comparison, compiling a functional version of the most recent Mozilla Firefox beta took a couple of hours (mostly due to a problem with MIDL).

My system: Windows 7 Ultimate 64-bit, 32GB RAM, Visual Studio 2010 Express, primary shell is git-bash.

The principal reason (I had a a number of other problems, too, see below) for how long it took to get a functional build was the "make_derived_sources" project in the Blink module, which did not produce the necessary code for a functional build, when it wasn't crashing and hanging.

What finally solved the problems and resulted in a working build was that I edited the preprocess.pm and make_names.pl::readNames to do more than 100 attempts at producing the correct output. Even with these changes I have to nursemaid this particular project build along during its half to one hour run, handling Windows crash dialogs for perl and killing hanging perl processes the whole time. At least one of the files generated by this project took more than 70 attempts before it was successfully generated.

The cause for the hangs and crashes appears, based on bug reports and other postings I have found, to be due to address mapping conflicts between Cygwin (the version checked out by Chromium) and other applications on my machine.

Examples of posts:

    <http://magpcss.org/ceforum/viewtopic.php?f=6&t=10310>
    <https://groups.google.com/a/chromium.org/d/msg/chromium-dev/cnz9_D_DhS4/Hhty_pDfqt4J>

The suggested solution from those post have been to use the Cygwin command "rebaseall". However, there is no "rebaseall" command in the chromium version of Cygwin, and I have been unable to convince the normal version of Cygwin to do the rebasing (at most it produced a lot of funny characters and error messages).

As rebasing was not practical, I had to make the changes I mentioned above. Those changes are, however, just hacks that are not practical for general usage.

(Aside: it would not surprise me if these problems with Cygwin caused my Norton 360 installation to break, and be in need of reinstallation).

Based on the above I would like to suggest that at least one of the following changes are made in the Windows version:

  - Preferably: get rid of Cygwin, and remove all shell scripting that can't be handled with cmd batch script.
  - If Cygwin cannot be removed, include rebase and rebaseall in the checkout, and preferably run it automatically when the module is checked out. Additionally, perhaps it would be an idea to upgrade to a more recent version?
  - Use the build environment's compiler for preprocessing, that is: cl with Visual Studio. I would think that a global python module for handling such tasks would solve most of the problems with generating the correct command lines.



I also encountered some other problems that was easier to solve:

I wanted to have a local git mirror of the source code, so that I would be able to play around in multiple environments if I wanted to. This proved more problematic than I had thought it would be.

<http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/creating-local-mirrors> describes how to create such a mirror.

Unfortunately, it leaves out a few important points:

  - That the gerrit-source.xml file manifest file is (apparently) the manifest file one need to use. I had to pull that information out of a bug report
  - The manifest left out all modules required by the Windows version, and I had to clone those manually.
  - One of the modules <http://git.chromium.org/gitweb/?p=native_client/src/third_party/mingw-w64/mingw/bin.git;a=summary> is NOT the same module that is on <https://chromium.googlesource.com/native_client/deps/third_party/mingw-w64/mingw/bin/> and it is the last one that is referenced by chrome.

Additionally, the mirror script could not be used in git-bash on Windows. See <http://code.google.com/p/git-repo/issues/detail?id=88> for a description of the problem. Also, once that was fixed, I discovered that the forall.py and project.py scripts are using Linux-only python modules and APIs. Even when I disabled those it was not possible to get the mirroring or repo commands to work properly.

I changed to using my Linux server for the mirror repository, which I had planned to avoid; I would have preferred to have had a local disk repository, at least to start with.

I would suggest that the mirror script clones all modules needed for a given project for all supported platforms, not just a subset.




Then, I encountered problems with getting the code checked out from my mirror. Essentially, both "gclient sync" and "fetch chromium --nosvn=True" started downloading from the central repositories, and "fetch chromium" is hardcoded to use that repository. Even when "fetch chromium" was "fixed", it eventually started downloading from the central repository.

I was eventually able to stop this problem, by manually adding

   u'custom_vars':{
     'git_url':u'ssh://localserver/path-to-repository',
     u'webkit_url':u'ssh://localserver/path-to-repository/chromium/blink.git'
   },

in the .gclient file for the project.

In my opinion these variables should be added automatically by "gclient config" (and perhaps "fetch", too), when a URL different than one of the known ones are used.


Minor issues:

  * Visual studio 2010 Express complains that "Some of the properties associated with the solution could not be read" every time the project is loaded.

  * At least once, Norton 360 detected nptestnetscapeplugin.dll as the threat "Suspicious.Cloud.7.F"

  * Several of the 32-bit projects (that are not included in the chrome dependencies) don't link due to Out of Memory conditions (have not yet chased down these), and I had to manually disable incremental linking for the "chrome_main_dll" project to get it linked.

--
Sincerely,
Yngve N. Pettersen

[Daniel Cheng]

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

Cygwin is optional. You shouldn't need it at all.

Daniel

[Christian Biesinger]

 You should probably tell Norton not to scan the Chromium folder. It will slow down your build a lot. Perhaps it is even responsible for some of your problems.

-christian

[Yngve Pettersen]

On Monday, July 1, 2013 11:46:38 PM UTC+2, Daniel Cheng wrote:

Cygwin is optional. You shouldn't need it at all.

Daniel

Hi,

Using a manually/globally installed Cygwin on Windows is indeed optional, using Cygwin to build Chromium isn't. Chromium solves that by using a special version from <http://git.chromium.org/gitweb/?p=chromium/deps/cygwin.git;a=summary> which is later used by the gclient builder or Visual Studio to perform shell script and perl tasks. That Cygwin version, however, is causing the issues I described.

Sincerely,

Yngve N. Pettersen

[Carlos Pizano]

Unfortunately Windows NT is allergic to forking [1] which cygwin needs and our legacy webkit scripts still use it.  AV products are very good at injecting crap dlls into every process thereby disrupting the forking operation. I've been told that we are rewriting the scripts in python but I don't know when that would be complete.

[1] http://www.cygwin.com/ml/cygwin/2009-05/msg00413.html

[Adam Barth]

On Windows, we should be using native Perl for compiling IDL files rather than Cygwin Perl:

            ['OS=="win"', {

                # Using native perl rather than cygwin perl cuts execution time

                # of idl preprocessing rules by a bit more than 50%.

                'perl_exe': '<(DEPTH)/third_party/perl/perl/bin/perl.exe',

                'gperf_exe': '<(DEPTH)/third_party/gperf/bin/gperf.exe',

                'bison_exe': '<(DEPTH)/third_party/bison/bin/bison.exe',

                # Using cl instead of cygwin gcc cuts the processing time from

                # 1m58s to 0m52s.

                'preprocessor': '--preprocessor "cl.exe -nologo -EP -TP"',

              },

It's possible that other parts of the build process still depend on Cygwin's Perl, however.

@yngve: I'd be happy to review Blink CLs that remove dependencies on Cygwin.

Adam

On Wed, Jul 3, 2013 at 4:04 PM, Carlos Pizano <c...@chromium.org> wrote:

Unfortunately Windows NT is allergic to forking [1] which cygwin needs and our legacy webkit scripts still use it.  AV products are very good at injecting crap dlls into every process thereby disrupting the forking operation. I've been told that we are rewriting the scripts in python but I don't know when that would be complete.

[1] http://www.cygwin.com/ml/cygwin/2009-05/msg00413.html

[Scott Graham]

The main IDL is native, but there's some remaining in third_party/WebKit that would be good to fix: https://code.google.com/p/chromium/codesearch#search/&q=msvs_cygwin_shell.*1&sq=package:chromium&type=cs

Bug here with ongoing (slow) progress: http://crbug.com/123026

[Yngve Pettersen]

Hi,

On Thursday, July 4, 2013 1:04:49 AM UTC+2, Carlos Pizano wrote:

Unfortunately Windows NT is allergic to forking [1] which cygwin needs and our legacy webkit scripts still use it.  AV products are very good at injecting crap dlls into every process thereby disrupting the forking operation. I've been told that we are rewriting the scripts in python but I don't know when that would be complete.

 It seems that it was Norton 360 that was causing the problem.

While disabling AV for the directory is a possibility, both to fix this and, as mentioned above, to increase performance, I don't think that is generally good advice, in particular if the browser in the build download identifiable malware that gets stored in the hierarchy and the user subsequently runs it.

I think this is a further indication that the cygwin dependency needs to be fixed, somehow.

Sincerely,

Yngve N. Pettersen

[Daniel Bratell]

Den 2013-07-05 02:12:19 skrev Yngve Pettersen
<yngve.n....@gmail.com>:

> I think this is a further indication that the cygwin dependency needs to
> be fixed, somehow.

I can only agree. cygwin is magic to Windows developers and it's magic to
Linux-developers so every time something goes wrong related to cygwin it's
very painful because nobody really knows how anything works.

/Daniel

[Nico Weber]

Maybe you want to help with http://crbug.com/123026 ?

 

/Daniel

[wangha...@gmail.com]

Hi man,

 Could you please upload the file named "nptestnetscapeplugin.dll". I'm very interested in it.

Thanks. 

在 2013年7月2日星期二UTC+8上午4时50分01秒，Yngve Pettersen写道：