cannot open `m4sugar/foreach.m4'

[Sergiy Byelozyorov]

Hello, 

I am trying to build Chromium using command line build. The build fails with the following message:

2>------ Build started: Project: webcore_bindings_sources, Configuration: Release Win32 ------

2>bison

2>/usr/bin/m4:stdin:1: include: cannot open `m4sugar/foreach.m4': No such file or directory

2>d:/devenv3/chromium-xml3d/src/third_party/WebKit/WebCore/css/CSSGrammar.y: fatal error: invalid @

in skeleton: @$.

2>                                                             -*- C -*-

2># Yacc compatible skeleton for Bison

...

When build is done from Visual Studio interface I experience no problems. I am running on Windows 7 and build using the following command:

D:\devenv3\chromium-xml3d\src\chrome>devenv.com chrome.sln /build "Release|Win32" /project chrome.vcproj

from the shell script in msys-git. Since build also doesn't fail if started from pure command line (cmd.exe), I believe the problem is that msys-git enviroment somehow conflicts with cygwin. Could you please help me investigate the problem further to solve it?

Sincerely yours,

Byelozyorov Sergiy

[Marc-Antoine Ruel]

Chromium checkout on windows includes a cygwin copy to have bison, yacc and others. User's cygwin normally don't interfere since we use batch file which shells out cmd.exe appropriately but having msys-git hooks into cmd's PATH. So just don't do that or send patches to fix your specific use case.

M-A

Sincerely yours,

Byelozyorov Sergiy

[Evan Martin]

On Sat, Oct 23, 2010 at 12:47 AM, Sergiy Byelozyorov
<byelo...@cs.uni-sb.de> wrote:
> from the shell script in msys-git. Since build also doesn't fail if started
> from pure command line (cmd.exe), I believe the problem is that msys-git
> enviroment somehow conflicts with cygwin. Could you please help me
> investigate the problem further to solve it?

In my experience I had better luck with msys-git if I avoided its
shell and environment settings.
Instead, I just made a "git.cmd" elsewhere in my path that called the
msysgit git.cmd via its full path.

[Sergiy Byelozyorov]

Chromium checkout on windows includes a cygwin copy to have bison, yacc and others. User's cygwin normally don't interfere since we use batch file which shells out cmd.exe appropriately but having msys-git hooks into cmd's PATH. So just don't do that or send patches to fix your specific use case.

I don't fully understand the problem, so I can't send any patches yet. Could you please explain what exactly is wrong with my PATH?

 

In my experience I had better luck with msys-git if I avoided its shell and environment settings. Instead, I just made a "git.cmd" elsewhere in my path that called the msysgit git.cmd via its full path.

Do you suggest to use a batch that will start the build? Or did I misunderstand you?

[Sergiy Byelozyorov]

Hello,

Just throwing in more information. Hopefully this will be useful.

build.log contains full build log and value of the PATH variable can be found in path.log. Also the build process never ends on it's own and I have to kill the devenv.com and processes spawned by it (using End Process Tree from Task Manager). And even after that there are still two processes m4.exe running and taking 100% of one CPU core each (I guess they are single-threaded). I have to kill them every time after build ends.

Sergiy

[Marc-Antoine Ruel]

If you're unsure about what's happening, the quick and dirty answer is to uninstall msysgit.

The longer answer is that you should probably remove msysgit from your PATH.

The longest answer is that you could probably send a patch to have the m4 and other wrapper .bat files to fix the PATH to make sure to call the expected executable or to use relative paths when shelling out bison and friends instead of calling without path, .e.g use 

..\third_party\cygwin\bin\bison foo

 instead of:

set PATH=%PATH%;..\third_party\cygwin\bin\

bison foo

M-A

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Evan Martin]

On Mon, Oct 25, 2010 at 10:25 PM, Sergiy Byelozyorov <rry...@gmail.com> wrote:
>> Chromium checkout on windows includes a cygwin copy to have bison, yacc
>> and others. User's cygwin normally don't interfere since we use batch file
>> which shells out cmd.exe appropriately but having msys-git hooks into cmd's
>> PATH. So just don't do that or send patches to fix your specific use case.
>
> I don't fully understand the problem, so I can't send any patches yet. Could
> you please explain what exactly is wrong with my PATH?

cygwin provides many helper binaries (sed, awk, m4, etc.) and puts
them into your PATH.
msysgit provides the same helper binaries, which behave slightly
differently. If you put them into your PATH things can get confused.

>> In my experience I had better luck with msys-git if I avoided its shell
>> and environment settings. Instead, I just made a "git.cmd" elsewhere in my
>> path that called the msysgit git.cmd via its full path.
>
> Do you suggest to use a batch that will start the build? Or did I
> misunderstand you?

1) uninstall msysgit
2) install msysgit, pick the install option that says "don't add to PATH"
3) make it so you can still run "git" from a command prompt by adding
a forwarding batch file in your path; depends on whether you use cmd
(in which case you want to make git.cmd) or cygwin (in which case you
can symlink from /usr/local/bin I think)

[Sergiy Byelozyorov]

cygwin provides many helper binaries (sed, awk, m4, etc.) and puts

them into your PATH.
msysgit provides the same helper binaries, which behave slightly
differently.  If you put them into your PATH things can get confused.

Thank you for the clarification. After studying build log for webcore_bindings_sources, I have tried to perform actions manually and narrowed the problem down to the fact that build fails whenever

C:\Program Files (x86)\Git\bin

is present in the PATH before running rule_bison.py. Once it is removed, build succeeds. However, I still don't understand why does it fail -- cygwin directories are preceding this entry and therefore files should be resolved to cygwin's executables. Here is the PATH value before running rule_bison.py:

PATH=d:\devenv3\chromium-xml3d\src\third_party\cygwin\\bin;d:\devenv3\chromium-xml3d\src\third_party\cygwin\\..\python_24;d:\devenv3\contrib\bin\x86;d:\devenv3\contrib\new\x86\release\expat_2_0_1;d:\devenv3\contrib\bin\x86;d:\devenv3\contrib\new\x86\release\expat_2_0_1;d:\devenv3\contrib\bin\x86;d:\devenv3\contrib\new\x86\release\expat_2_0_1;d:\devenv3\contrib\bin\x86;d:\devenv3\contrib\new\x86\release\expat_2_0_1;d:\devenv3\install\build-msvc-opt-x86\rtsg2;d:\devenv3\install\build-msvc-opt-x86\rtsg2\bin;d:\devenv3\install\build-msvc-opt-x86\rtsg2\lib;d:\devenv3\install\build-msvc-opt-x86\rtfact\bin;d:\devenv3\install\build-msvc-opt-x86\rtfact\lib;d:\devenv3\xerces-c\x86\xerces-c-3_1_1\bin;d:\devenv3\contrib\bin\x86;d:\devenv3\contrib\new\x86\release\expat_2_0_1;d:\devenv3\resources\build-msvc-x86\boost\bin;d:\devenv3\resources\build-msvc-x86\boost\lib;d:\devenv3;c:\Users\Rryk\bin;.;C:\Program Files (x86)\Git\local\bin;C:\Program Files (x86)\Git\mingw\bin;C:\Program Files (x86)\Git\bin;c:\Windows\system32;c:\Windows;c:\Windows\System32\Wbem;c:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files\ThinkPad\Bluetooth Software\;c:\Program Files\ThinkPad\Bluetooth Software\syswow64;c:\Program Files (x86)\CMake 2.8\bin;c:\Program Files\TortoiseSVN\bin;C:\Program Files (x86)\Git\cmd;c:\Python26;d:\devenv3\depot_tools;c:\Program Files (x86)\Notepad++

 

>> In my experience I had better luck with msys-git if I avoided its shell
>> and environment settings. Instead, I just made a "git.cmd" elsewhere in my
>> path that called the msysgit git.cmd via its full path.
>
> Do you suggest to use a batch that will start the build? Or did I
> misunderstand you?

1) uninstall msysgit
2) install msysgit, pick the install option that says "don't add to PATH"
3) make it so you can still run "git" from a command prompt by adding
a forwarding batch file in your path; depends on whether you use cmd
(in which case you want to make git.cmd) or cygwin (in which case you
can symlink from /usr/local/bin I think)

My msysgit version (1.7.0.2) has this solution as one of the options during the install. I've never installed entire toolkit into the PATH, but only git and gitk that are executed via batch files that are created by the installer and registered in PATH automatically. Nevertheless msysgit tools get into the PATH, becase I am starting build from inside msysgit shell. Unfortunately I can't avoid this as our build system runs in msysgit. I have tried assigning PATH an empty value before running devenv.com, but it still fails for some reason. How can I output PATH environment variable from the Python script? 

The longest answer is that you could probably send a patch to have the m4 and other wrapper .bat files to fix the PATH to make sure to call the expected executable or to use relative paths when shelling out bison and friends instead of calling without path, .e.g use 

..\third_party\cygwin\bin\bison foo

 instead of:

set PATH=%PATH%;..\third_party\cygwin\bin\

bison foo

Where should I place such batch files, so they are called instead of msysgit binaries?

P.S. What does "shell out" means?

Sergiy

[Evan Martin]

On Thu, Oct 28, 2010 at 3:21 AM, Sergiy Byelozyorov <rry...@gmail.com> wrote:
> My msysgit version (1.7.0.2) has this solution as one of the options during
> the install. I've never installed entire toolkit into the PATH, but only git
> and gitk that are executed via batch files that are created by the installer
> and registered in PATH automatically. Nevertheless msysgit tools get into
> the PATH, becase I am starting build from inside msysgit shell.

I am missing something. You wrote:
1) that only git and gitk are installed into the path
2) that you get all msysgit tools in your path because of the msysgit shell

It seems that you already had the tools you needed in PATH alrady, so
I'm not sure why you need the msysgit shell.

> Unfortunately I can't avoid this as our build system runs in msysgit. I have
> tried assigning PATH an empty value before running devenv.com, but it still
> fails for some reason. How can I output PATH environment variable from the
> Python script?

import os
print os.environ
print os.environ['PATH']

> Where should I place such batch files, so they are called instead of msysgit
> binaries?

I dunno, somewhere else in your path. c:\windows I guess. It's been
a long time since I last touched it.

> P.S. What does "shell out" means?

Sorry. http://www.catb.org/jargon/html/S/shell-out.html
I wasn't using it in the right way anyway. I just meant "run".

[Sergiy Byelozyorov]

On Thu, Oct 28, 2010 at 5:52 PM, Evan Martin <ev...@chromium.org> wrote:

On Thu, Oct 28, 2010 at 3:21 AM, Sergiy Byelozyorov <rryk.ua@gmail.com> wrote:
> My msysgit version (1.7.0.2) has this solution as one of the options during
> the install. I've never installed entire toolkit into the PATH, but only git
> and gitk that are executed via batch files that are created by the installer
> and registered in PATH automatically. Nevertheless msysgit tools get into
> the PATH, becase I am starting build from inside msysgit shell.

I am missing something.  You wrote:
1) that only git and gitk are installed into the path
2) that you get all msysgit tools in your path because of the msysgit shell

When running build from Visual Studio 2008 Command Prompt, I have no problems (build succeeds), because msysgit doesn't push it's tools into my path. However, when I am starting the build from withing msysgit bash shell, then all the tools are in path and they conflict with cygwin.

 

It seems that you already had the tools you needed in PATH alrady, so
I'm not sure why you need the msysgit shell.

We are running our own dev system, which requires msysgit bash shell. From this shell a script starts build of chrome, by running the command:

D:\devenv3\chromium-xml3d\src\chrome>devenv.com chrome.sln /build "Release|Win32" /project chrome.vcproj

 

> Unfortunately I can't avoid this as our build system runs in msysgit. I have
> tried assigning PATH an empty value before running devenv.com, but it still
> fails for some reason. How can I output PATH environment variable from the
> Python script?

import os
print os.environ
print os.environ['PATH']

Strange. This is what I get when placing this code into rule_bison.py:

d:\devenv3\chromium-xml3d\src\third_party\cygwin\\bin;d:\devenv3\chromium-xml3d\src\third_party\cygwin\\..\python_24;c:\Program Files (x86)\Microsof

t Visual Studio 9.0\VC\bin;C:\Program Files\Microsoft SDKs\Windows\v6.0A\\bin;C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Tools\bin;C:\

Program Files (x86)\Microsoft Visual Studio 9.0\Common7\tools;C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\ide;C:\Program Files (x86)\HT

ML Help Workshop;C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin;C:\Windows\Microsoft.NET\Framework\v2.0.5072

There is no msysgit PATHS at all. Why does it fail then... i don't understand.

> Where should I place such batch files, so they are called instead of msysgit
> binaries?

I dunno, somewhere else in your path.  c:\windows I guess.  It's been
a long time since I last touched it.

How should I create a patch then? I thought that I should place batch files somewhere in the Chromium repository.

[Sergiy Byelozyorov]

Hello,

I have tried to find the difference between two scenarios (when it fails and when it doesn't). To do that I have replaced m4.exe with my own version which does nothing but prints used command line into debug file. This is the output from scenario when build succeeds (given original m4.exe):

M4 replacement. Command line: d:\devenv3\chromium-xml3d\src\third_party\cygwin\bin\m4.exe /usr/share/bison/m4sugar/m4sugar.m4 - /usr/share/bison/yacc.c 

This is output for the scenario when build fails:

M4 replacement. Command line: d:\devenv3\chromium-xml3d\src\third_party\cygwin\bin\m4.exe d:/devenv3/contrib/share/bison/m4sugar/m4sugar.m4 - d:/devenv3/contrib/share/bison/yacc.c 

Note the command-line arguments for m4.exe. It appears that paths for the second one are wrong and use Windows-style, while paths are correct and are Cygwin-style in the first scenario. Thus something is wrong inside bison.exe, which starts m4.exe. Any ideas how can I debug this? Do I really have to build/debug bison.exe for that?

Sergiy

[Sergiy Byelozyorov]

Hello,

After some more investigation it turned out to be the BISON_PKGDATADIR environment variable, which was set by one of the modules in our development system. This has to be empty before building Chromium, since Cygwin will set another value for it. I thank everyone for helping me. Hope this will be useful for someone.

Sergiy