cmake -G Ninja failes on windows, need help

[claus.klein]

Hi,

the cmake Ninja generater creates build commands like this:

c:\PROGRA~1\MICROS~4\VC\bin\cl.exe /nologo /D_DEBUG /MDd /Zi /Ob0 /
Od /RTC1 /TP /FoCMakeFiles\cmTryCompileExec1840110254.dir
\testCXXCompiler.cxx.obj /FdcmTryCompileExec1840110254.pdb -c
testCXXCompiler.cxx

cd. && C:\Programme\CMake2.8\bin\cmake.exe -E vs_link_exe c:
\PROGRA~1\MICROS~4\VC\bin\cl.exe /nologo CMakeFiles
\cmTryCompileExec1840110254.dir\testCXXCompiler.cxx.obj /D_DEBUG /
MDd /Zi /Ob0 /Od /RTC1 /FecmTryCompileExec1840110254.exe /
FdcmTryCompileExec1840110254.pdb -link /
implib:cmTryCompileExec1840110254.lib /version:0.0 /STACK:10000000 /
machine:X86 /debug /INCREMENTAL:YES /subsystem:console kernel32.lib
user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib
uuid.lib comdlg32.lib advapi32.lib && cd.

cd "C:\Dokumente und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles
\CMakeTmp" && C:\Programme\CMake2.8\bin\cmake.exe -H"C:\Dokumente und
Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles\CMakeTmp" -B"C:\Dokumente
und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles\CMakeTmp"

##########################################

This works at command shell, but not within ninja:

ninja rebuild_cache
rem [1/1] Running CMake to regenerate build system...
rem FAILED: cd "C:\Dokumente und Einstellungen\KLEIN_CL\ninjabuild
\CMakeFiles\CMakeTmp" && C:\Programme\CMake2.8\bin\cmake.exe -H"C:
\Dokumente und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles\CMakeTmp" -
B"C:\Dokumente und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles
\CMakeTmp"
rem CreateProcess failed: The system cannot find the file specified.
rem ninja: build stopped: subcommand failed.

############################################

What goes wrong?

Claus

[claus.klein]

a same problem on mingw:

KLEIN_CL@WD002072 /c/downloads/ninja
$ ninja -n -v graph.png
[1/2] ./ninja -t graph > build\graph.dot
[2/2] dot -Tpng build\graph.dot > graph.png

KLEIN_CL@WD002072 /c/downloads/ninja
$ ninja -v graph.png
[1/2] ./ninja -t graph > build\graph.dot
FAILED: ./ninja -t graph > build\graph.dot
ninja: ERROR: unknown target '>', did you mean 'all'?

ninja: build stopped: subcommand failed.

KLEIN_CL@WD002072 /c/downloads/ninja
$

[claus.klein]

// FIXME: We have to prepend 'cmd /c' on Windows, too if this breaks
command
// lines greater than 8,191 chars! ck

if (!CreateProcessA(NULL, (char *)(string("cmd /c ") +
command).c_str(), NULL, NULL,
/* inherit handles */ TRUE,
CREATE_NEW_PROCESS_GROUP,
NULL, NULL,
&startup_info, &process_info)) {
DWORD error = GetLastError();
if (error == ERROR_FILE_NOT_FOUND) { // file (program) not found
error is treated as a normal build action failure
if (child_pipe)
CloseHandle(child_pipe);
CloseHandle(pipe_);
CloseHandle(nul);
pipe_ = NULL;
// child_ is already NULL;
buf_ = "CreateProcess failed: The system cannot find the file
specified.\n";
return true;
} else {
Win32Fatal("CreateProcess"); // pass all other errors to
Win32Fatal

[Nico Weber]

Hi Claus,

On Wed, May 23, 2012 at 4:06 AM, claus.klein <claus...@arcormail.de> wrote:
> This works at command shell, but not within ninja:
>
> ninja rebuild_cache
> rem [1/1] Running CMake to regenerate build system...
> rem FAILED: cd "C:\Dokumente und Einstellungen\KLEIN_CL\ninjabuild
> \CMakeFiles\CMakeTmp" && C:\Programme\CMake2.8\bin\cmake.exe -H"C:
> \Dokumente und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles\CMakeTmp" -
> B"C:\Dokumente und Einstellungen\KLEIN_CL\ninjabuild\CMakeFiles
> \CMakeTmp"

I believe the ninja support in cmake 2.8.8 was linux-only. While ninja
on Windows does mostly work, there are still a few unresolved issues
(in ninja too, not just in cmake's generator).

Nico

[Bill Hoffman]

The cmake support for windows ninja is tied to this fork of ninja right now:

http://sourceforge.net/projects/cmakescript/files/ninja.exe/download

I believe it adds response file support.

The other really big hole in ninja on windows is file level depends.
What is the status of depend generation with MS CL?

Thanks.

-Bill

[Claus Klein]

Hi Bill,
After changing this Single line it Works!
The current git ninja HEAD is Good,
I may send you a Patch if you like to test it yourself?

Dependencies are Not my Problem I use gcc only ;-))

Claus

Sent from my iPhone

[Bill Hoffman]

On 5/23/2012 2:00 PM, Claus Klein wrote:
> Hi Bill,
> After changing this Single line it Works!
> The current git ninja HEAD is Good,
> I may send you a Patch if you like to test it yourself?
>

Sure, a patch would be good.

Thanks.

> Dependencies are Not my Problem I use gcc only ;-))
>

I have seen some work on this list, I am wondering where that work is?

Thanks.

-Bill

[Scott Graham]

>> Dependencies are Not my Problem I use gcc only ;-))
>>
> I have seen some work on this list, I am wondering where that work is?

I've been using a fork also for the Chromium Windows builds
https://github.com/sgraham/ninja. I'm (slowly) working on the pieces
around dependencies and trying to upstream.

There's another implementation here: https://github.com/martine/ninja/pull/258

>
> Thanks.
>
> -Bill
>

[Claus Klein]

This is my final patch for today

it fix the build problems with Cmake on Windows,
the timestamp will be as 64bit value in 100nsec on OSX, Linux, and
Windows.
Tested with nightly build of Cmake on Windows and OSX (Leopard)
MVS2005, native mingw, cygwin on XP, gcc4.7, mingw cross on Linux and
Mac

I hope it helps.

// Regards
Claus

[Bill Hoffman]

On 5/23/2012 3:54 PM, Scott Graham wrote:

>> I have seen some work on this list, I am wondering where that work is?
>
> I've been using a fork also for the Chromium Windows builds
> https://github.com/sgraham/ninja. I'm (slowly) working on the pieces
> around dependencies and trying to upstream.
>
> There's another implementation here: https://github.com/martine/ninja/pull/258
>

What is the path forward to get one of these pushed into master of
ninja? ninja is not very useful right now with the MSVC compiler
without file level depends. Are there still technical issues to work out?

Thanks.

-Bill

[Qingning Huo]

I am against this change, which will break our build commands, those longer than

8k characters that is.

The other bad thing about this patch is it enforces cmd /c on all ninja build commands

on windows. Whereas currently, the writer of ninja fies can choose whether to prepend

cmd /c to individual commands. So it removes a choice from ninja users.

Regards,

Qingning

[Claus Klein]

You are right!

On Windows ninja breaks it own Build and design rules.

With MVS you have no dependency Files, so for what is it good on Windows?

On posix Ninja use sh an on windoof there is only cmd.

How generats you your dependency files and keep them uptodate with microschrot?

Bad News, for all who waits for cmake with Ninja Generator 

Ck

[Philip Craig]

Hi Claus,

I also don't want the "cmd /c, or not" choice removed from ninja users. The correct place to do the change you want is in cmake's generation capabilities.

On your other topic, you are correct that Microsoft compilers do not emit dependency information in a form parseable by ninja. We wrote a script to extract it (via parsing the output of  the showIncludes option. That script is not part of ninja)

regards,

Philip

[claus.klein]

Hi Philip,

yes, cmake should be changed to solve this problem, so send a patch!

The cmake crew is waiting...

But it is fact, cmake has disabled the ninja generator as long as
problems with ninja on all platforms are NOT soved!


And at the ninja docu, I cant find any hints about this "cmd /c" hack?
And "ninja -v graph.png" fails on native windows or mingw, but is
used by the generated by the ninja configure.py script itself?

Do you kown how subprocess class works on possix:
execl("/bin/sh", "/bin/sh", "-c", command.c_str(), (char *) NULL);

and this is the same exactly the same on windows:

CreateProcessA(NULL, (char *)(string("cmd /c ") +

command).c_str(), ...)


So please change what you want, but do it consistent and document it!

By
Claus

On May 25, 10:37 am, Philip Craig <phi...@pobox.com> wrote:
> Hi Claus,
>
> I also don't want the "cmd /c, or not" choice removed from ninja users. The
> correct place to do the change you want is in cmake's generation
> capabilities.
>
> On your other topic, you are correct that Microsoft compilers do not emit
> dependency information in a form parseable by ninja. We wrote a script to
> extract it (via parsing the output of  the showIncludes option. That script
> is not part of ninja)
>
> regards,
> Philip
>
>
>
>
>
>
>
> On Thursday, 24 May 2012 22:00:46 UTC+1, claus.klein wrote:
>
> > You are right!
> > On Windows ninja breaks it own Build and design rules.
>
> > With MVS you have no dependency Files, so for what is it good on Windows?
>
> > On posix Ninja use sh an on windoof there is only cmd.
>
> > How generats you your dependency files and keep them uptodate with
> > microschrot?
>
> > Bad News, for all who waits for cmake with Ninja Generator
>
> > Ck
>

[Philip Craig]

I don't use cmake, so I am in no position to send a patch, sorry.

The two create process lines you quote are not the same, due to Windows behaving differently. That's not for ninja to fix though. It's not ninja's duty to hide differences between operating systems for programs (such as cmake) that make use of ninja to generate build commands. That fundamental difference has to go all the way through to the client tool, cmake in this case.

[Bill Hoffman]

On 5/25/2012 9:00 AM, Philip Craig wrote:
> I don't use cmake, so I am in no position to send a patch, sorry.
>
> The two create process lines you quote are not the same, due to Windows
> behaving differently. That's not for ninja to fix though. It's not
> ninja's duty to hide differences between operating systems for programs
> (such as cmake) that make use of ninja to generate build commands. That
> fundamental difference has to go all the way through to the client tool,
> cmake in this case.
>

This is a patch for cmake.
diff --git a/Source/cmLocalNinjaGenerator.cxx
b/Source/cmLocalNinjaGenerator.cxx
index 425b219..ed55fae 100644
--- a/Source/cmLocalNinjaGenerator.cxx
+++ b/Source/cmLocalNinjaGenerator.cxx
@@ -285,6 +285,8 @@ std::string cmLocalNinjaGenerator::BuildCommandLine(
li != cmdLines.end(); ++li) {
if (li != cmdLines.begin())
cmd << " && ";
+ else if (cmdLines.size() > 1)
+ cmd << "cmd.exe /c ";
cmd << *li;
}
return cmd.str

I am working on getting it upstream. To get cmake off the fork of ninja
it is using for windows, it needs this patch, plus a change to make it
use response files instead of depending on ninja to create them.

-Bill


-Bill

[Claus Klein]

Hi Bill,

I think the cmake patch should be like this, or not?
Mail me if it is committed, please.

One note about cmake ninja generator:
Why generate empty pre and post link commands?
It would be faster if they are skipped too.

Claus

claus-kleins-macbook-pro:cmake clausklein$ cat
NinjaGeneratorBuildCommandLine.patch
diff --git a/Source/cmLocalNinjaGenerator.cxx b/Source/
cmLocalNinjaGenerator.cxx
index 425b219..d62c0ec 100644
--- a/Source/cmLocalNinjaGenerator.cxx
+++ b/Source/cmLocalNinjaGenerator.cxx
@@ -283,8 +283,13 @@ std::string
cmLocalNinjaGenerator::BuildCommandLine(
std::ostringstream cmd;
for (std::vector<std::string>::const_iterator li = cmdLines.begin();
li != cmdLines.end(); ++li) {
- if (li != cmdLines.begin())
+ if (li != cmdLines.begin()) {
cmd << " && ";
+#ifdef _WIN32

+ } else if (cmdLines.size() > 1) {
+ cmd << "cmd.exe /c ";

+#endif
+ }
cmd << *li;
}
return cmd.str();
claus-kleins-macbook-pro:cmake clausklein$

[Nicolas Desprès]

On Sat, May 26, 2012 at 4:19 PM, Claus Klein <claus...@arcormail.de> wrote:
> Hi Bill,
>
> I think the cmake patch should be like this, or not?
> Mail me if it is committed, please.
>
> One note about cmake ninja generator:
>        Why generate empty pre and post link commands?

Probably to have "generic" Ninja rules.

>        It would be faster if they are skipped too.

Most shell optimizes that.

-Nico

--
Nicolas Desprès

[Claus Klein]

Hi Nico,

yes shell do this, but windows cmd.exe should be avoided ...

To be fast, we should not execute "cmd.exe /c cd. && g++ in.cc out.o
&& :" or something like that.
Direct call the compiler/linker would be faster and without cmdline
length restrictions at least under windows.

My tips:

Be fast and use ninja.
But do it right, use ccache too!

Don't use MSVC and do NOT work on Windows if possible, it is terrible
slow. :-((

By
Claus

[Nicolas Desprès]

Hi Claus,

I agree with you on the mainline. Even on Linux it would be faster to
not fork a shell at all. I have some idea about how to optimize that
but it would long to write such a patch and I don't have the time
right now. The current solution has several advantages:
1/ It already works and some people can use it right now.
2/ It is not that slow, specially compared to Makefiles (even for full build).

There is room for improvement. This is a work in progress.

Cheers,
-Nico

--
Nicolas Desprès

[Peter Kümmel]

Ninja for Windows support in CMake is now in "tests pass" state:

https://github.com/syntheticpp/CMake/commits/ninja-cldeps

Changes are on the way into the next release.

Thanks to Scott's cldeps tool, which was a good starting point,
cmake now also has dependency tracking for C/C++ files
when MS tools are used.

Peter

[Peter Kümmel]

> Ninja for Windows support in CMake is now in "tests pass" state:

With martine/master ninja.

Binaries are here:
https://sourceforge.net/projects/cmakescript/files/

Peter

[Bill Hoffman]

This is all in CMake upstream next, and will be in master soon, and will
be in the next CMake release.

CMake is testing martine/master ninja against CMake next for linux,
windows and Mac. The results can be found here:

http://open.cdash.org/index.php?project=CMake&filtercount=1&showfilters=1&field1=buildname/string&compare1=63&value1=ninja

The OSX failures are because CMake does not yet create application
frameworks and bundles with the ninja generator yet.

However, CMake ninja on windows is now working with full depend
information. The depend wrapper code for cl is in CMake source tree
right now instead of ninja.

-Bill

[Evan Martin]

On Wed, Jun 13, 2012 at 10:54 AM, Bill Hoffman <bill.h...@kitware.com> wrote:
> This is all in CMake upstream next, and will be in master soon, and will be
> in the next CMake release.

Awesome!

> However, CMake ninja on windows is now working with full depend information.
>  The depend wrapper code for cl is in CMake source tree right now instead of
> ninja.

Can you point me at a link to this code?

Longer-term, where would you prefer this code to live? Does CMake
need a way to extract dependency information independently of Ninja?

Relatedly, someone on IRC was asking about how to get dependency
information correct for Windows .rc files, which apparently also go
through the preprocessor but doesn't appear to have a /showIncludes
equivalent flag. Do you know anything about what we can do about
that?

[Peter Kümmel]

On 13.06.2012 20:29, Evan Martin wrote:
> On Wed, Jun 13, 2012 at 10:54 AM, Bill Hoffman<bill.h...@kitware.com> wrote:
>> This is all in CMake upstream next, and will be in master soon, and will be
>> in the next CMake release.
>
> Awesome!
>
>> However, CMake ninja on windows is now working with full depend information.
>> The depend wrapper code for cl is in CMake source tree right now instead of
>> ninja.
>
> Can you point me at a link to this code?

https://github.com/syntheticpp/CMake/blob/ninja-cldeps/Source/cmcldeps.cxx#L676

>
> Longer-term, where would you prefer this code to live? Does CMake
> need a way to extract dependency information independently of Ninja?

AFAIK there is also cmake version for extracting dependencies, but I
don't know how it is used.

>
> Relatedly, someone on IRC was asking about how to get dependency
> information correct for Windows .rc files, which apparently also go
> through the preprocessor but doesn't appear to have a /showIncludes
> equivalent flag. Do you know anything about what we can do about
> that?

It's already solved in the file above (lang == "RC"):
1. feed the re-processor (cl.exe /P) with /showIncludes and the rc file
2. generate .d
3. compile with rc.exe

Peter

[Peter Kümmel]

On 13.06.2012 21:09, Peter K�mmel wrote:
> On 13.06.2012 20:29, Evan Martin wrote:
>> On Wed, Jun 13, 2012 at 10:54 AM, Bill Hoffman<bill.h...@kitware.com> wrote:
>>> This is all in CMake upstream next, and will be in master soon, and will be
>>> in the next CMake release.
>>
>> Awesome!
>>
>>> However, CMake ninja on windows is now working with full depend information.
>>> The depend wrapper code for cl is in CMake source tree right now instead of
>>> ninja.
>>
>> Can you point me at a link to this code?
>
> https://github.com/syntheticpp/CMake/blob/ninja-cldeps/Source/cmcldeps.cxx#L676

This wrapper is only used by the ninja generator (maybe this was your initial question).

[Philip Craig]

Hi Evan,

Separately to the cmake project, on my project we have cl, rc and cs (csharp) wrapper standalone code, designed to be called from within ninja.build files *instead* of the original compilers/parsers. It is not called at the time that ninja.build is generated. The code wraps the original programs, and uses the same tricks to generate the .dep files in addition to emitting the compiler/parser output files. The code is (of course) bound to Visual Studio.

To be honest, I thought you'd never want such code for these standalone utilities in ninja. But are you now offering? The code is in C++ and C#.

[Bill Hoffman]

On 6/13/2012 2:29 PM, Evan Martin wrote:
> Longer-term, where would you prefer this code to live? Does CMake
> need a way to extract dependency information independently of Ninja?

Since the rest of the stuff was answered by Peter, I can take this one.
I don't really care if it lives in CMake or ninja. Seems like it
would make sense for it to be in ninja. CMake does not need this code
as it has another way to compute depend information for its makefiles.
So, it is there for the sole purpose of supporting ninja. If this code
were moved into ninja, the code would be removed from CMake.

As it stands now CMake is the only tool I know of that can generate
ninja files for Windows with file level depends without requiring a fork
of ninja. I would think that ninja would want to keep this code so that
other generators could use it on Windows.

I am just happy that CMake is now fully working with ninja on Windows. :)

Thanks Peter!


-Bill

[Aditya Sood]

Hi,

I saw your patch file. I am new to this. I dont know where to apply and how to apply patch file to eradicate the ninja build error on windows.

Thanks.