Export Bazel build to MSBuild

[Sammi Song]

Hi Guys,

    In order to use VisualStudio to write and debug C++ code, MSBuild seems the best choice so far.  But I really love Bazel to manage all my assets, so I'm wondering how could we just build everything by bazel, but also support Visual Studio?

  A roughly idea from mind:

1. Using bazel built-in gRPC client, send build events to gRPC server;

2. gRPC server parse the build events and build MSBuild project;

Why I'm thinking this way?

-> bazel built-in gRPC buffer is maintained by bazel project and so it will be evolving automatically with future version;

-> MSBuild will be used and be maintained by Visual Studio;

-> The gRPC server will be decoupled with Bazel and MSBuild codebase and so could be flexible to handle any special custom logic.

Before I kick off the project, I'd like ask folks here:

If there any existing project to handle similar use cases?

If my approach make sense?

Thanks,

Sammi

[Konstantin]

Hey Sammi,

I think I understand your idea, but the build events seems semantically too far from the C++ specific Bazel rules initiating it and it would be very hard for gRPC server you proposed to assemble it back to VS compatible Msbuild projects. There is alternative idea - use Bazel aspect facility to "record" each c++ target execution attributes and then in a separate pass gather those recordings and generate VS projects from the data collected. You may want to look at the Lavender project which implements this idea.

Unfortunately Lavender was not updated for years, has many bugs and by now is badly outdated and incompatible with any recent version of Bazel. But still you can grasp the idea from it.

For the needs of the company I work for I took Lavender and updated it substantially, making it practically usable. Unfortunately as this work was mostly done on my very limited spare time (it was not my official project) I took a number of shortcuts in order to get it working sooner and as the result some of our build specifics got leaked into the code.It is probably still Ok to be shared (I need to check), but it is not ready for public consumption without some cleanup and my time is scarce. 

Let me know if you are interested to put some effort into it and I will see what can be done.

Konstantin

[Sammi Song]

Hey  Konstantin,

   I think using aspects to gather all information needed to generate MSbuild projects should be doable.  I checked out lavender project, it builds a python script to run bazel build with starlark code to collect aspect output data, and then generate MSBuild project/solution.

  Since the msbuild project meta data is pretty complicated, I still think managing it in a different process(gRPC server) would be easier to maintain and even develop, such as generate XML from domain classes. How about building a bazel rule which collects data by aspects, and also sending message streams to MSBuild server?

-> Borrow the code and idea from lavender to build the bazel rules to collect target execution details data;

-> Define data model which are required to generate MSBuild project by proto buffer;

-> Send message stream by gRPC client in bazel rules right after collection aspect data;

I do everything in my spare time as well,  just want to create a bridge for two islands:  bazel <<--->> Visual Studio. 

Thanks,

Sammi

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/2bf4500c-8d2f-44f0-9b50-9c0548cc5a98n%40googlegroups.com.

[Konstantin]

Sammi, I keep thinking about it. The Lavender approach which I understand and support is to have two relatively independent stages of the process:

1. Run aspected Bazel build which produces JSON files with the extension .msbuild - one file per C++ target. Those files contain the recordings of the corresponding target attributes when it was run by the build.

2. After the build is done Stage 2 enumerates JSON files generated at Stage 1 and for each produces .vsxproj file.

As far as I understand your idea instead of doing it sequentially from the same process (Python script in Lavender case) you propose to do it in parallel - Stage 1 in the Bazel build process aspect code and Stage 2 in a separate gRPC server process. One advantage of the parallel approach is that it could be faster than sequential. On the other hand, I know from experience that Stage 2 takes negligible time comparing to Stage 1, and running them in parallel would not generate much better performance even for the huge projects. What other advantages do you see in having a separate gRPC server process which would justify increased complexity? 

There are two design points here which can be discussed independently - one is how to gather the data and send them to the project generating code and another is how to implement that project generated code when necessary data are already collected and delivered one way or another. I agree that .vsxproj generating code can be done much better than simple text substitutions in the template files, but so far I don't see the need to do it in a separate process and involve gRPC. 

[Sammi Song]

Hi Konstantin,

   I agree with you to separate the process as two parts, collect aspect data + generate msbuild files. The advantage by gRPC is not about performance improvement, it seems it could decouple the dependencies, so the interface between bazel and visualstudio could be defined by protobuf contracts.

   I guess most tricky logic should be handled by generating msbuild files, such as ClCompile, Midl, Link, and even MakeAppx for msix manifest files.The contract could be designed for msbuild use cases, so we can start from real world use cases, such as CppWinRT UWP project, Windows Dll library etc.

   Another part is even if we use aspects to collect all necessary data by just running "bazel build --aspect=gen-msbuild.bzl", where should we put the gRPC client code? It's much more complicated than just building a python script, I'm guessing if we could build starlark code which can collect data via aspects and also sending data via gRPC client(it could be done by building a bazle package to generate runfiles?).

   If we go this direction, we could make the whole project extendable and also just using bazel build command to generate MSBuild project files, no need to run a special python script and using its own command options. If we want to support Xcode, we could just change the gRPC server, gRPC client, and bazel aspect customized rules int one repo, and keep using the same bazel command with some different option like "bazel build --aspect=gen-xcode.bzl".

- Sammi

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/fc6b556b-0cc0-4546-8bd4-565215fd6b3en%40googlegroups.com.

[Konstantin]

-- The advantage by gRPC is not about performance improvement, it seems it could decouple the dependencies, so the interface between bazel and visualstudio could be defined by protobuf contracts.

I understand this point, but wonder if it worth implementation complexity.

-- generating msbuild files, such as ClCompile, Midl, Link, and even MakeAppx for msix manifest files. The contract could be designed for msbuild use cases, so we can start from real world use cases, such as CppWinRT UWP project, Windows Dll library etc.

Wow, that sounds bold, but where you are going to get all that data on the Bazel side? Built-in Bazel C++ rules have no way to specify those Windows specific details. There is a reason why for the Apple ecosystem there is completely separate set of Bazel C++ rules which take into account Apple specifics. There is no such Windows specific set of rules in Bazel and even if it would exist I still prefer to stick to the platform agnostic rules.

-- Another part is even if we use aspects to collect all necessary data by just running "bazel build --aspect=gen-msbuild.bzl", where should we put the gRPC client code? It's much more complicated than just building a python script, I'm guessing if we could build starlark code which can collect data via aspects and also sending data via gRPC client

I don't think it is possible to write gRPC Client in Starlark. Custom rule could take care of it, but I don't know if it is possible to feed the output of the command line invoked aspect to the rule in the same Bazel invocation. 

[Sammi Song]

On Mon, Jan 4, 2021 at 8:57 AM Konstantin <kon...@ermank.com> wrote:

-- The advantage by gRPC is not about performance improvement, it seems it could decouple the dependencies, so the interface between bazel and visualstudio could be defined by protobuf contracts.

I understand this point, but wonder if it worth implementation complexity.

-- generating msbuild files, such as ClCompile, Midl, Link, and even MakeAppx for msix manifest files. The contract could be designed for msbuild use cases, so we can start from real world use cases, such as CppWinRT UWP project, Windows Dll library etc.

Wow, that sounds bold, but where you are going to get all that data on the Bazel side? Built-in Bazel C++ rules have no way to specify those Windows specific details. There is a reason why for the Apple ecosystem there is completely separate set of Bazel C++ rules which take into account Apple specifics. There is no such Windows specific set of rules in Bazel and even if it would exist I still prefer to stick to the platform agnostic rules.

Yes, it's true, standard cc_libary, cc_binaray, or cc_test cannot differ from Windows specific data. It may not be a bad thing, but a good thing, we do need build rules to support collecting msbuild oriented data. For example, I spent a lot of time building a midl generate code rule, which supports generating header files from a midl file. We may not need to build totally different rules to support windows msbuild, I'd like to say build rules on top of standard cc rules to support windows.  In other words, we only convert bazel projects to msbuild projects when all bazel rules(standard cc rules + windows specific rules) are applicable.

 

-- Another part is even if we use aspects to collect all necessary data by just running "bazel build --aspect=gen-msbuild.bzl", where should we put the gRPC client code? It's much more complicated than just building a python script, I'm guessing if we could build starlark code which can collect data via aspects and also sending data via gRPC client

I don't think it is possible to write gRPC Client in Starlark. Custom rule could take care of it, but I don't know if it is possible to feed the output of the command line invoked aspect to the rule in the same Bazel invocation. 

That would be a challenge, we may look into the bazel code, borrow the idea how they send the build event stream to a remote server, it is already working, and it is live for buildbuddy.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/fa0d11cb-2ae5-4b01-9c08-ed1be1f64c09n%40googlegroups.com.

[Konstantin]

--  how they send the build event stream to a remote server

It works because it is implemented in Bazel using Java. What you want is to do the same with Starlark and I doubt it is possible.

[Sammi Song]

Sent from my iPhone

On Jan 4, 2021, at 11:47, Konstantin <kon...@ermank.com> wrote:

--  how they send the build event stream to a remote server

It works because it is implemented in Bazel using Java. What you want is to do the same with Starlark and I doubt it is possible.

I’m thinking do it by Java as well, and manage it in msbuild repo, the same place for generating msbuild project code.

The Java code should be able to create an executable Runfile and starlark rule depend on the Runfile to send gRPC stream.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/0400985d-7a82-4e8a-a4f3-4abee9e61051n%40googlegroups.com.

[Sammi Song]

After more discussion with Cristian Hancila  in the bazel general channel, I've got an experiment plan to implement the msbuild project:

1. Create a standalone repo named msbuild in github;

2. Define the msbuild project needed proto in msbuild project, which will include data proto and also rpc proto;

3. Implement a gRPC server module in msbuild by Java, which receives gRPC message stream and generates different project msbuild project type;

4. Create a Java application, which executes the command line  "bazel query //... --output=proto", parses the result; sends gRPC stream to the server, and saves the response as msbuild project xml file;

5. In bazel project, it needs introduce msbuild as its external dependency in WORKSPACE,  using define a option in .bazelrc, such as --msbuild-generateor=https://localhost/v1/msvc

In the end, if everything are ready, the user could generate the msbuild project like:

-> bazel run @msbuild:server  to startup the gRPC server locally(optional), if the server is up on the cloud, user can skip this step;

-> Generate msbuild project, for example:

    bazel run @msbuild:uwp to generate UWP project/solution files;

    bazel run @msbuild:winui to generate WinUI project/solution files;

    bazel run @msbuild:install to package windows store application;

Notes:

• No need manually maintain a target to identify the deps for msbuild project, it will get all deps by run bazel query command in msbuild Java application;
• The command are generic, if we make it right, it could generate msbuild projects by the same command for any msbuild ready project;

Thanks a lot for the community to give me inspiration and guide.

- Sammi

[Konstantin]

-- Create a standalone repo named msbuild in github;

Oh-h, please don't call it just "msbuild" - it would be so misleading! Name it "bazel_msbuild_bridge" or something like that - more specific than just "msbuild".

Before you spend too much time implementing the server, the protocol and the project generator I'd recommend to start with the Bazel client side as it is much less clear at the moment. For instance it is not clear to me that the query "bazel query //... --output=proto" would give you the output sufficient to generate vsxproj out of it.

Keep us posted - I am curious to see even the very minimal prototype.

Konstantin

[Sammi Song]

On Wed, Jan 6, 2021 at 10:58 AM Konstantin <kon...@ermank.com> wrote:

-- Create a standalone repo named msbuild in github;

Oh-h, please don't call it just "msbuild" - it would be so misleading! Name it "bazel_msbuild_bridge" or something like that - more specific than just "msbuild".

How about bazel-to-msbuild?

 

Before you spend too much time implementing the server, the protocol and the project generator I'd recommend to start with the Bazel client side as it is much less clear at the moment. For instance it is not clear to me that the query "bazel query //... --output=proto" would give you the output sufficient to generate vsxproj out of it.

Sure, it makes sense. The real client/driver should be from msbuild project requirement, I had collected and analysis  one UWP project sample, before write gRPC server or bazel Client code, I'd like to start to define the contract, msbuild proto files first, which will include message proto and rpc proto.

 

Keep us posted - I am curious to see even the very minimal prototype.

Sure, I'd love to. 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/537209f4-e5a9-4020-a749-5ba731b5043an%40googlegroups.com.

[Sammi Song]

Hi There,

  I'm happy to share the first  bazel-to-msbuild release  I've built to convert cpp bazel console app project to msbuild solution.

-> It is using ```bazel query ... output=proto" to gather all cc_librarr, cc_binary, and cc_test packages;

-> It generates msbuild project for each package;

-> It generates one solution for one bazel workspace;

-> It detect package dependencies and manage it as project references in msbuild project as well;

Please check it out and let me know what you think.

Regards,

Sammi

[Sammi Song]

Fixed the broken zip file by release 0.1.1.

https://github.com/sammi/bazel-to-msbuild/releases/tag/0.1.1

- Sammi