[v2] merged generator tools

[Josh Petrie]

I merged parts of the generator toolchain from v2/SlimDX2/... to v2/... I made a few changes:

- I renamed the projects to have consistent names that are more reflective of our style: we now have Parser.csproj and Generator.csproj as well as the XIDL.csproj support library. I expect to eventually combine most of the functionality here, except perhaps the XIDL utility library itself, into a single Generator.csproj. Note that I did not rename the actual output files yet.

- I removed the MSDN documentation generation. I don't think we can actually use this as-is, because it constitutes creating a derivative work, and I don't believe we have the license to do so.

- I changed wher e the source is actually generated, since the relative locations changed.

I noticed a few things that I'm a little concerned about while doing this, and they make me think we need to reign in the forward progress we're making with respect to supporting new APIs via the generator, and instead focus on hardening the code we've already got.

I am particularly concerned about the shape of the Program*.cs files in Generator.csproj, which contain what looks like a lot of unnecessarily /unsafe code and a lot of hard-coded type references. I'm also concerned about the fact that the mapping order is as brittle as it seems -- I wasn't able to generate only a subset of API easily, and I don't believe it should be necessary to require a particular order or indeed have most of this transformative information in code at all (as o pposed to data).

Additionally, I am confused about the role of DynamicInterop.cs and of the SlimDX.Interop.dll (which I don't see source for), and I'm wondering if there is a better solution available to us than use of the text template engine, because I'm uneasy about the "magic" references that need to be added to the projects to make them function.

-- jmp

[Alexandre Mutel]

 

 

>>>>>>>>> I merged parts of the generator toolchain from v2/SlimDX2/... to v2/... I made a few changes:

>>>>>>>>> - I renamed the projects to have consistent names that are more reflective of our style: we now have Parser.csproj and Generator.csproj as well as the XIDL.csproj support library. I expect to eventually combine most of the functionality here, except perhaps the XIDL utility library itself, into a single Generator.csproj. Note that I did not rename the actual output files yet.

 

Ok, but as I said, I don’t want to have Generator & Parser to be merged yet. This would be really annoying for the development right now.  Keep in mind that HeaderToXIDL is not compiled with Any Target because It relies on an external Boost.Wave.Dll which is a C++/CLI API (meaning that merging would imply to be running with this limitation).

 

Also, I have sometimes HeaderToXIDL crashing for no reasons (although It was running well while debugging). I have not been able to make a clear diagnosis of this bug... not sure boost::wave is always reliable, also I had to compile boost:wave in release mode otherwise the process is much slower, and the debugging boost assembly/pdb is > 60Mo!

 

Also, at some point, I would like to have the Parser code externalized in another project when It will be stabilized, so merging is probably not a good option... I mean not merging the code, but we could of course imagine at some point that the generator will use the Parser as a library.

 

>>>>>>>>> - I removed the MSDN documentation generation. I don't think we can actually use this as-is, because it constitutes creating a derivative work, and I don't believe we have the license to do so.

 

Wow... Ok, I have found a solution that is probably more “compliant”, by using the CHM documentation from inside the Direct SDX folders. I don’t think there is any issue of using it.. I can’t imagine delivering a generated code without the imported documentation.

 

>>>>>>>>> - I changed wher e the source is actually generated, since the relative locations changed.

 

Ok. When I will be able to develop on the root directory? Do you need to perform new things or I can start to work on it? I’m going probably to check-in my code (for parser&generator&generated files) still in the sub directory SlimDX2 if it’s not yet usable.

 

 

>>>>>>>>> I noticed a few things that I'm a little concerned about while doing this, and they make me think we need to reign in the forward progress we're making with respect to supporting new APIs via the generator, and instead focus on hardening the code we've already got.

>>>>>>>>>

>>>>>>>>> I am particularly concerned about the shape of the Program*.cs files in Generator.csproj, which contain what looks like a lot of unnecessarily /unsafe code and a lot of hard-coded type references. I'm also concerned about the fact that the mapping order is as brittle as it seems -- I wasn't able to generate only a subset of API easily, and I don't believe it should be necessary to require a particular order or indeed have most of this transformative information in code at all (as o pposed to data).

 

unnecessarily / unsafe? could you elaborate?

 

What do you mean by the mapping order? The calls in Program to “MapWin32”, “MapDXGI”, “Map...etc”?  I need to check, but the only thing that is necessary is to instantiate global types that are used by other MapXXX. There is also still some cross dependencies between some Map function (not too much though). This could be fix easily.

 

Keep in mind that to generate a subset of the API, you need to have all dependency in the generating process. Meaning that you cannot generate D3D10 if you have not included also DXGI (as it should be for compiling the generated assemblies). This is because the tools is using structure declaration, enum dependency...etc.

 

The minimum requirement for generating an API is MapWin32 which is more a MapGlobalInitializer which should be run before mapping any other things.

 

Also, inside a MapXXX file, order is used but not so important. Things are protected when using type tag like TagName<CppEnum> (or TagFunction which is implicitly working only on CppFunction). But there are inside each sections the need for an order. For example you need sometimes to generate a generic name mapping for all the types (regex with .* ) and then refining for some of them. (explicit naming).

 

For the rest of the API, I have finished parsing XACT3, X3DAudio, XInput, DirectInput. I had still to make some bugfix on the parser, and I have made things much “cleaner” in the #define process in the Parser.

 

Also, putting the mapping code in a text files is something I have in mind BUT, It should be done when the mapping API will be somewhat not in the dev process like this. I making lots of refactoring during the development process in order to make things easier to use. The current mapping code is the results from lots of refactoring and didn’t look like this (It probably changed also quite a bit from the version you have and the version I’m working on it right now).

 

So, at some point, yes, It could be a “nice to have” mapping code inside some external text files, but right now, this would be a bad decision. Also, there are still some cases where I need to directly do custom mapping queries in the code : making a request with Find<XXX> and performing some process on the results. When all use cases will be clearly identified (and It’s on the way, as almost the whole DirectX SDK is parsed and is being mapped), It will be possible to have those custom queries inside a “normalized” one... and later, making those mapping things inside some external text files... They will be probably shorter to code, lighter and easier to maintain.

 

>>>>>>>>> Additionally, I am confused about the role of DynamicInterop.cs and of the SlimDX.Interop.dll (which I don't see source for), and I'm wondering if there is a better solution available to us than use of the text template engine, because I'm uneasy about the "magic" references that need to be added to the projects to make them function.

 

DynamicInterop is used both by the Generator and at runtime by the SlimDX2 assembly. It’s in charge off generating the interop code that cannot be generated in C# (but is valid in CLI and is used by C++/CLI), while still being able to have a “AnyTarget” cpu on it. In order to compile the SlimDX2 assemblies, they need to be compiled against a SlimDX2.Interop.dll. But when running a SlimDX2 program, the interop assembly is generated at runtime, accordingly to the running platform/architecture (x86 or x64).

 

Also, for the text templating, T4 engine is probably the best and most integrated templating engine with C#, developed by Microsoft, and used by VS itself for all generated code. It is available from the Visual Studio 2010 SDK and should appear in the “Extensions” assemblies when you add a reference to them (I don’t understand the “magic” here? You don’t see those assemblies in your “Extensions” list?).

Of course, you could develop your own templating engine, but I really don’t have time to waste my time on such a project when there are existing “standardized” and supported solutions, really.

 

I agree that It will take some time if you need to go to change things in the *.tt files. They are not easy to manipulate, but after using them, you get used to them, and It’s not that annoying.

 

In any case, It will take some time to get also into the whole current code, as some parts are messy, and as it is also pretty large and sometimes “complex” to handle things correctly.

[Alexandre Mutel]

>>>>>>>>> I am particularly concerned about the shape of the Program*.cs files in Generator.csproj, which contain what looks like a lot of unnecessarily /unsafe code and a lot of hard-coded type references.

 

Oh, ok I see now. Well, for unsafe, It’s just a bad copy-paste, as I need at least unsafe in ProgramWin32 (sizeof on struct), but shouldn’t be necessary for other mapping.

 

Also, for “lot of hard-coded type references”, I you mean things like  [

Global.Name + ".Direct2D1"] , I agree and this will be fixed asap!

 

[Alexandre Mutel]

>>>>>>>>> I am particularly concerned about the shape of the Program*.cs files in Generator.csproj, which contain what looks like a lot of unnecessarily /unsafe code and a lot of hard-coded type references. I'm also concerned about the fact that the mapping order is as brittle as it seems -- I wasn't able to generate only a subset of API easily, and I don't believe it should be necessary to require a particular order or indeed have most of this transformative information in code at all (as o pposed to data).

 

Oh, I see know concerning the issue “I wasn't able to generate only a subset of API easily”, I found a kind of feature missing and patch the code. The current generator code was trying to parse all Cpp includes, no matter there was an affected mapping for the include. Meaning that the generator was trying to map all includes declared in the directx.xidl. So if you tried for example to generate only DXGI, It was not possible!

 

This is now fixed, as it should process only the includes that where explicitly mapped to an assembly/namespace.

[Alexandre Mutel]

 

>>>>>>>>> I am particularly concerned about the shape of the Program*.cs files in Generator.csproj, which contain what looks like a lot of unnecessarily /unsafe code and a lot of hard-coded type references. I'm also concerned about the fact that the mapping order is as brittle as it seems -- I wasn't able to generate only a subset of API easily, and I don't believe it should be necessary to require a particular order or indeed have most of this transformative information in code at all (as o pposed to data).

 

Concerning the order problem, It is now fixed (in the sub SlimDX2 folder). I had to make a whole review of the mapping, as lots of MapXXX where working globally on all includes, one reason that order was important.

 

I have cleanup things now and It should be ok ( remove unsafe, settings assembly/namespace just once). I have also extended the usage of FindContext method that force all Tag.* functions to restrict their work on the include context passed to FindContext (It was not the case in the past, meaning that a D3D9 mapping could override a D3DCommon mapping, which was really annoying).

Meaning that when you perform this :

 

group.FindContext.Add(“d3d9”);

group.TagName<CppEnum>(“DIRECT3D9_ENUM...etc.”, “NEW_NAME”);

group.FindContext.Clear();

 

The TagName method is only working on the d3d9 include.

 

There should be almost no more dependency, apart between D3DCompiler and D3DCommon, as the CreateBlob defined in D3DCompiler is moved to D3DCommon (and there is the usage of a functionGroup).

 

The new code updated on subdir SlimDX2 is also able to generate code for DirectInput, XInput, XACT3, X3DAudio, though I didn’t perform any mapping (apart from the basics one) and I didn’t setup any project in the SlimDX2 solution.

 

As soon as things are settled in the svn root-v2 directory, I‘ll write a more complete TODO file with every corners of what is done/not done/to be done for the tools and each generated API.

 

[Alexandre Mutel]

About the doc, there is a much better solution at using the windows_graphics.hxs file in the documentation folder, as it is a plain zip! I’m going to update the code to use it instead of MSDN. That’s great also because I don’t need anymore to keep a cache of the files, as they are all already in the windows_graphics.hxs archive, great! Everything will work smoothly from data extracted from the DirectX SDK folders, pretty cool.

[Josh Petrie]

Ok, but as I said, I don’t want to have Generator & Parser to be merged yet. This would be really annoying for the development right now.  Keep in mind that HeaderToXIDL is not compiled with Any Target because It relies on an external Boost.Wave.Dll which is a C++/CLI API (meaning that merging would imply to be running with this limitation).

This is less of a concern right now, although I don't believe your worries about having a single generator tool are founded -- I had it working a few weeks ago, and it's much faster. With a small amount of command-line options, such as allowing the XIDL to be cached to a file and re-running from the cache, a single tool can be configured to run exactly the same as the two tools we have now. Compiling the generator as a 32 bit application isn't a concern. Your point about externalizing the parser code is a good one, but only relevant to this issue if you intend to do it sooner rather than later.

I expect you could work on the root of the branch starting Tuesday. I need to incorporate the latest changes you've made (or you could, if you like) and set up the .csproj files slightly differently -- we want to isolate hand-written and generated code into different subdirectories within each project directory, because we do not want to check in generated code. But all of that should be done within the next two days.

As for the documentation, we can't use windows_graphics.hxs either for exactly the same reasons -- we can't just copy/paste and tweak their work without permission from them. The documentation generation should not be something we focus on at this stage of development anyhow, the code is more important.

Regarding T4: that means we have a dependency on the Visual Studio SDK, which is too big. It "seems" to mostly work now (I do not have the assemblies anywhere in the VS UI, but I am able to hand-edit the .csproj to add them), but that's an intractable dependency long-term. I agree that developing text template systems is a pain in the ass, though, so we can continue to use it for now and I'll investigate a replacement later on. 

I think an ideal plan of attack at this point would be to stabilize commits to the v2/SlimDX2 directory by Tuesday -- then we can merge things down once more and nuke the subdirectory. We should also enter all the proposed work items into the Google Code bug tracker (as opposed to a TODO.txt type of file) tagged "v2" so we have a clearer idea of what needs to be done. I also want to reiterate that I feel we need to focus on cleaning up and hardening the parser/generator code itself and addressing issues such as any problems that prevent us from externalizing the mapping definitions and all that now, before we finish trying to expand its scope to cover the whole of our API set. We don't need to generate everything, remember. We should focus on Direct3D 10 and/or 11, DXGI, and whatever common support items we need for those.

-- jmp

[Michael Popoloski]

I agree. We've been charging ahead too far, too fast. We need to get something stable so that the rest of us have something to work on, instead of just sitting back and watching you add features to the generator. The generator is going to be a core tenant of a large part of the library, so we need to make sure it's absolutely rock solid before trying to build anything on top of it. Additionally, the build process needs to be simplified as much as possible, so that we can simply hit a button and have the whole process wind up and spit out the assemblies we need.

--
You received this message because you are subscribed to the Google Groups "slimdx-devel" group.
To post to this group, send email to slimdx...@googlegroups.com.
To unsubscribe from this group, send email to slimdx-devel...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/slimdx-devel?hl=en.

[@lx]

> This is less of a concern right now, although I don't believe your worries
> about having a single generator tool are founded -- I had it working a few
> weeks ago, and it's much faster. With a small amount of command-line
> options, such as allowing the XIDL to be cached to a file and re-running
> from the cache, a single tool can be configured to run exactly the same as
> the two tools we have now. Compiling the generator as a 32 bit application
> isn't a concern. Your point about externalizing the parser code is a good
> one, but only relevant to this issue if you intend to do it sooner rather
> than later.

Ok, for the caching, and also, I just want keep them them in separate
assemblies.

> I expect you could work on the root of the branch starting Tuesday. I need
> to incorporate the latest changes you've made (or you could, if you like)
> and set up the .csproj files slightly differently -- we want to isolate
> hand-written and generated code into different subdirectories within each
> project directory, because we do not want to check in generated code. But
> all of that should be done within the next two days.

You now exactly how to organise things regarding the build process and
SlimDX behavior with that, so I'll let you do the migration.

> As for the documentation, we can't use windows_graphics.hxs either for
> exactly the same reasons -- we can't just copy/paste and tweak their work
> without permission from them. The documentation generation should not be
> something we focus on at this stage of development anyhow, the code is more
> important.

Wow, ok. This is a really really bad news. I will keep the generator
on my side here in order to have the documentation.
I'm using intensively the documentation for mapping things, checking
parameters, optionnal parameters. So the documentation is not
something like "just nice" but has valuable information for the
mapping process.

Is there any chance to get anykind of authorization?... I mean, this
information is part of the SDK, if you had to write the documentation
yourself, you would almost copy/paste the whole doc and change few
things...

I know that this is an early stage, but It has been extremely helpfull
to have the documentation so far, even for coding... so...

Btw, I have been able tody to plug the documentation from the hxs
files... just in case... :)

> Regarding T4: that means we have a dependency on the Visual Studio SDK,
> which is too big. It "seems" to mostly work now (I do not have the
> assemblies anywhere in the VS UI, but I am able to hand-edit the .csproj to
> add them), but that's an intractable dependency long-term. I agree that
> developing text template systems is a pain in the ass, though, so we can
> continue to use it for now and I'll investigate a replacement later on.

Well, normally, starting with VS2010, the T4 engine is freely
redistribuable without having VS2010 installed so...
I don't see the dependency to VS SDK big... but ok... will see later
this...

> I also want to reiterate that I feel we need to focus on cleaning up and hardening
> the parser/generator code itself and addressing issues such as any problems
> that prevent us from externalizing the mapping definitions and all that now,
> before we finish trying to expand its scope to cover the whole of our API
> set. We don't need to generate everything, remember. We should focus on
> Direct3D 10 and/or 11, DXGI, and whatever common support items we need for
> those.

Working on other APIs was required in order to make the code more
robust and to support unexpected things (like the need to custom
interop for DWrite, or custom marshalling for all Audios
interfaces...etc.)
So far, the whole API is parsable and generatable, so that's already a
good news.

But I agree that the code is still far from robust:
- Both the parser and the generator have a very nasty error handling,
almost requiring to debug when a mapping is wrong. So this need to be
much more improved.
- The generator code is still a bit messy
- There are couple of things I'm implementing right now, like a
"struct field/size verifier" in order to check that the fields
alignment and structure size are matching the native's one...

Also, as soon as the v2 branch is clean, I will definetely need more
feedback on the generated API... I would like to move on at some point
and use the generated library on my own, and don't want to wait too
much! ;)

[Alexandre Mutel]

About the usage of T4 template and after looking a bit more about the code, It seems that It wouldn’t cost too much to switch to a custom template (without almost any external text files).

 

In fact, when I’m looking at current *.tt template files, they are much more composed of C# code (with logic to determine what to output) than text template, which means that I can almost copy-paste them in a C# file with very minor fixes to make them work with a custom template engine (a very thin infrastructure in order to write easily templatized code in C#). We would benefit from C# intellisense on the template code, which would be more helpful for maintenance.

 

As soon as things are a bit more cleaner in the generator, I would probably switch to this solution.