Converting a Visual Studio project to a ninja build file
Alexander Zezulinsky
Evan Martin
Do you know if it's possible to convert a Visual Studio Project (*.vcproj) to a ninja build file?I'd like to use ninja to speed up incremental build of a product on Windows.
--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Neil Mitchell
Alexander Zezulinsky
Neil Mitchell
converter as well (although I raised the bug on the Shake tracker), so
it would also be suitable for that route. However, don't let that stop
you doing it first, since I won't have time for a few weeks, and it
does sound fun.
I just read over my convertor, the only interesting things to bear in mind:
* It is all XML, so you will need to grab an XML library (I guess the
Ninja convention would be to go with something in Python).
* The format is slightly different in each VS version, so try and test
it with a range of project versions, although I think at least 2008
and 2010 are quite similar - if you go back too much before that it
changes radically.
* If you want to support F# or C# then you end up with a single
command line to compile an entire project, not a separate command line
per file.
* Multiple "Configuration" tags could match at once, so you can have a
Win32 configuration that is overridden by a Win32|Release
configuration in a few places. However, if you are going from files
written by Visual Studio you might not have to worry.
* The mapping from tag to in XML to command line parameter is best
seen by going in Visual Studio to Configuration Properties, C/C++,
Command Line, as that updates quickly as you toggle options. My
converter only handled the options I cared about, so I don't have an
exhaustive (or even particularly big) mapping that I can give you.
Thanks, Neil
On Fri, Jul 5, 2013 at 11:11 PM, Alexander Zezulinsky
<alexander....@gmail.com> wrote:
> I can also try to add such a functionality to the ninja build system.
>
Evan Martin
Dave Brotherstone
I got some way with this, ignoring VC 2005, and only concentrating on 2008/2010. As they both use MSBuild, I just overrode the compile, link, resource generation and manifest generation steps, in order to use all the other msbuild targets, conditions etc, to then generate a ninja build file with all the relevant switches on each command. The ninja file generation obviously involved an msbuild call, but as that would only need to be done when the actual project changed, it didn't seem like a big issue - the faithful representation in the ninja file made up for that cost.
The aim was to massively speed up building our 400+ projects, with many interdependencies, by generating a collection of ninja files, then letting ninja work out what really needed rebuilding (currently we call msbuild for each project, which when done as a library call instead of an exe, gives us a 3 minute warm no-op build)
In the process, I quickly realised we'd need tracker support in ninja, in order to discover the dependencies. I know that's been discussed before, but the idea didn't get much love, as I remember. I started work on a patch to support it, and got most of the way to a prototype. There were a few caveats, in that the tracker.exe had to be used instead of the DLL, as the api version tracks the current and child processes, with no support for threads (AFAIK), so that couldn't be used with ninja's concurrent build model. The tracker.exe overhead wasn't much, but was certainly measurable. I expect it would need significant rework to support Evan's new dependency files, and also need work to finish it off.
I can make the msbuild tasks available, if anyone is interested - they don't support all the options, but supporting the options you need would be trivial - I genuinely don't know what options all our projects use, so was working through them alphabetically rather than logically.
If there's enough interest I might go back to the tracker support and finish it off.
Cheers,
Dave
Reid Kleckner
I got some way with this, ignoring VC 2005, and only concentrating on 2008/2010. As they both use MSBuild, I just overrode the compile, link, resource generation and manifest generation steps, in order to use all the other msbuild targets, conditions etc, to then generate a ninja build file with all the relevant switches on each command. The ninja file generation obviously involved an msbuild call, but as that would only need to be done when the actual project changed, it didn't seem like a big issue - the faithful representation in the ninja file made up for that cost.
The aim was to massively speed up building our 400+ projects, with many interdependencies, by generating a collection of ninja files, then letting ninja work out what really needed rebuilding (currently we call msbuild for each project, which when done as a library call instead of an exe, gives us a 3 minute warm no-op build)
In the process, I quickly realised we'd need tracker support in ninja, in order to discover the dependencies. I know that's been discussed before, but the idea didn't get much love, as I remember. I started work on a patch to support it, and got most of the way to a prototype. There were a few caveats, in that the tracker.exe had to be used instead of the DLL, as the api version tracks the current and child processes, with no support for threads (AFAIK), so that couldn't be used with ninja's concurrent build model. The tracker.exe overhead wasn't much, but was certainly measurable. I expect it would need significant rework to support Evan's new dependency files, and also need work to finish it off.
I can make the msbuild tasks available, if anyone is interested - they don't support all the options, but supporting the options you need would be trivial - I genuinely don't know what options all our projects use, so was working through them alphabetically rather than logically.
If there's enough interest I might go back to the tracker support and finish it off.
Cheers,
Dave
--
Evan Martin
I got some way with this, ignoring VC 2005, and only concentrating on 2008/2010. As they both use MSBuild, I just overrode the compile, link, resource generation and manifest generation steps, in order to use all the other msbuild targets, conditions etc, to then generate a ninja build file with all the relevant switches on each command. The ninja file generation obviously involved an msbuild call, but as that would only need to be done when the actual project changed, it didn't seem like a big issue - the faithful representation in the ninja file made up for that cost.
The aim was to massively speed up building our 400+ projects, with many interdependencies, by generating a collection of ninja files, then letting ninja work out what really needed rebuilding (currently we call msbuild for each project, which when done as a library call instead of an exe, gives us a 3 minute warm no-op build)
In the process, I quickly realised we'd need tracker support in ninja, in order to discover the dependencies. I know that's been discussed before, but the idea didn't get much love, as I remember. I started work on a patch to support it, and got most of the way to a prototype. There were a few caveats, in that the tracker.exe had to be used instead of the DLL, as the api version tracks the current and child processes, with no support for threads (AFAIK), so that couldn't be used with ninja's concurrent build model. The tracker.exe overhead wasn't much, but was certainly measurable. I expect it would need significant rework to support Evan's new dependency files, and also need work to finish it off.
I can make the msbuild tasks available, if anyone is interested - they don't support all the options, but supporting the options you need would be trivial - I genuinely don't know what options all our projects use, so was working through them alphabetically rather than logically.
Nico Weber
On Sun, Jul 7, 2013 at 6:35 PM, Dave Brotherstone <dav...@pobox.com> wrote:
I got some way with this, ignoring VC 2005, and only concentrating on 2008/2010. As they both use MSBuild, I just overrode the compile, link, resource generation and manifest generation steps, in order to use all the other msbuild targets, conditions etc, to then generate a ninja build file with all the relevant switches on each command. The ninja file generation obviously involved an msbuild call, but as that would only need to be done when the actual project changed, it didn't seem like a big issue - the faithful representation in the ninja file made up for that cost.
The aim was to massively speed up building our 400+ projects, with many interdependencies, by generating a collection of ninja files, then letting ninja work out what really needed rebuilding (currently we call msbuild for each project, which when done as a library call instead of an exe, gives us a 3 minute warm no-op build)
In the process, I quickly realised we'd need tracker support in ninja, in order to discover the dependencies. I know that's been discussed before, but the idea didn't get much love, as I remember. I started work on a patch to support it, and got most of the way to a prototype. There were a few caveats, in that the tracker.exe had to be used instead of the DLL, as the api version tracks the current and child processes, with no support for threads (AFAIK), so that couldn't be used with ninja's concurrent build model. The tracker.exe overhead wasn't much, but was certainly measurable. I expect it would need significant rework to support Evan's new dependency files, and also need work to finish it off.
Could you elaborate on what you mean by "tracker" here? And also what you mean about a DLL vs an executable.Sorry to be so naive, I've never used msbuild!
I can make the msbuild tasks available, if anyone is interested - they don't support all the options, but supporting the options you need would be trivial - I genuinely don't know what options all our projects use, so was working through them alphabetically rather than logically.
I find it's always beneficial to publish such a thing, if it doesn't take too much of your time -- even if nobody ever ends up using it directly, it can serve as a reference for people attempting to recreate what you've done.
Dave Brotherstone
On 8 Jul 2013 09:35, "Evan Martin" <mar...@danga.com> wrote:
> Could you elaborate on what you mean by "tracker" here? And also what you mean about a DLL vs an executable.
Tracker is what Msbuild (and hence visual studio from version 2010 for c++, and I think c# since at least 2005) uses to track dependencies between compilation units and files. It's actually a very poorly documented DLL with a standard C call interface (startTracking, endTracking, writeTrackingLog type calls). I'm away on holiday at the moment so I haven't got the exact calls in front of me. It creates a log of the files accessed during the tracking period, minus any particular system files that are only indirectly accessed. The list includes things the compiler uses, so if you install a hotfix for cl.exe, it will rebuild your source, as foo.dll has been updated. It also includes all the system headers, obviously.
There is a .net exe called Tracker.exe, that makes the tracking calls to the Filetracker.dll, and starts the given command line. As the filetracker.dll tracks everything the current and child process does, it's unsuitable for multi-threaded builds that call out to child processes. That's where the tracker.exe comes in useful, but it obviously adds an overhead over just calling the functions in the DLL.
There's some undocumented magic that allows cl.exe to compile multiple .cpp files, and write a header in the logs for each input file, I've not yet worked out how that works.
You can see the format of the logs if you look in any cl.read.log.1 file (or cl.write.log.1 file, for what is written to during compilation, used for cleaning), or similar. VS uses tracker by default, and although you can probably turn it off, I don't know how, and that would mean full rebuilds every time (unless there's more magic I don't know about).
The reason you only find problems with it when googling is, I believe, that normally it just sits in the background and works, it's enabled by default, and means incremental builds "just work". Obviously when it goes wrong, people start digging around, and find the issue in the tracker logs - I've had cases where the dependencies for some input files were just never written to the logs, and if an input file isn't listed in the log, it gets rebuilt.
The .net interface to filetracker.dll is here - http://msdn.microsoft.com/en-us/library/microsoft.build.utilities.filetracker.aspx. That's actually better documented than the raw C interface. I'm sorry I've not got the link to the DLL documentation with me - not that it's particularly helpful.
Apologies for the length of that description, that's what I've found out so far about tracker, and most of it comes from experimenting, so some/all of it may be inaccurate or just plain wrong.
> I find it's always beneficial to publish such a thing, if it doesn't take too much of your time -- even if nobody ever ends up using it directly, it can serve as a reference for people attempting to recreate what you've done.
I'll see what I can do when I get back in a week.
Cheers,
Dave
Dave Brotherstone
On 8 Jul 2013 05:40, "Reid Kleckner" <r...@google.com> wrote:
> Coincidentally, I'm also trying to figure out how to override the toolchain used by MSBuild. Do you have any doc links on how to do this? I haven't been able to find anything good. So far I've just been placing my tools in the cwd of MSBuild and naming them cl.exe etc so they come first on PATH search.
There's a more reliable and more efficient way. You can override (redefine) any MSBuild task by compiling a .net assembly, and adding the definition of the task to msbuild. That way, your C# code gets called at the last "real" step when the ClCompile task gets called (so your ClCompile task gets called, instead of the official one that starts cl.exe). At that point, ms build has processed all the conditionals, calculated all the flags in use for the given input (or inputs), and all you have to do is map the provided list of property values to the relevant command line switches - this is time consuming but trivial.
I've not got all the information with me on where everything must go, but I'll try and publish something next week so you can see how it's done.
Cheers,
Dave.
Scott Graham
On 8 Jul 2013 09:35, "Evan Martin" <mar...@danga.com> wrote:
> Could you elaborate on what you mean by "tracker" here? And also what you mean about a DLL vs an executable.Tracker is what Msbuild (and hence visual studio from version 2010 for c++, and I think c# since at least 2005) uses to track dependencies between compilation units and files. It's actually a very poorly documented DLL with a standard C call interface (startTracking, endTracking, writeTrackingLog type calls). I'm away on holiday at the moment so I haven't got the exact calls in front of me. It creates a log of the files accessed during the tracking period, minus any particular system files that are only indirectly accessed. The list includes things the compiler uses, so if you install a hotfix for cl.exe, it will rebuild your source, as foo.dll has been updated. It also includes all the system headers, obviously.
There is a .net exe called Tracker.exe, that makes the tracking calls to the Filetracker.dll, and starts the given command line. As the filetracker.dll tracks everything the current and child process does, it's unsuitable for multi-threaded builds that call out to child processes. That's where the tracker.exe comes in useful, but it obviously adds an overhead over just calling the functions in the DLL.
There's some undocumented magic that allows cl.exe to compile multiple .cpp files, and write a header in the logs for each input file, I've not yet worked out how that works.
You can see the format of the logs if you look in any cl.read.log.1 file (or cl.write.log.1 file, for what is written to during compilation, used for cleaning), or similar. VS uses tracker by default, and although you can probably turn it off, I don't know how, and that would mean full rebuilds every time (unless there's more magic I don't know about).
The reason you only find problems with it when googling is, I believe, that normally it just sits in the background and works, it's enabled by default, and means incremental builds "just work". Obviously when it goes wrong, people start digging around, and find the issue in the tracker logs - I've had cases where the dependencies for some input files were just never written to the logs, and if an input file isn't listed in the log, it gets rebuilt.
The .net interface to filetracker.dll is here - http://msdn.microsoft.com/en-us/library/microsoft.build.utilities.filetracker.aspx. That's actually better documented than the raw C interface. I'm sorry I've not got the link to the DLL documentation with me - not that it's particularly helpful.
Apologies for the length of that description, that's what I've found out so far about tracker, and most of it comes from experimenting, so some/all of it may be inaccurate or just plain wrong.
> I find it's always beneficial to publish such a thing, if it doesn't take too much of your time -- even if nobody ever ends up using it directly, it can serve as a reference for people attempting to recreate what you've done.
I'll see what I can do when I get back in a week.
Cheers,
Dave
--
Nico Weber
On 8 Jul 2013 05:40, "Reid Kleckner" <r...@google.com> wrote:> Coincidentally, I'm also trying to figure out how to override the toolchain used by MSBuild. Do you have any doc links on how to do this? I haven't been able to find anything good. So far I've just been placing my tools in the cwd of MSBuild and naming them cl.exe etc so they come first on PATH search.
There's a more reliable and more efficient way. You can override (redefine) any MSBuild task by compiling a .net assembly, and adding the definition of the task to msbuild. That way, your C# code gets called at the last "real" step when the ClCompile task gets called (so your ClCompile task gets called, instead of the official one that starts cl.exe). At that point, ms build has processed all the conditionals, calculated all the flags in use for the given input (or inputs), and all you have to do is map the provided list of property values to the relevant command line switches - this is time consuming but trivial.
I've not got all the information with me on where everything must go, but I'll try and publish something next week so you can see how it's done.
Cheers,
Dave.
--