[v2] New templating prototype

[Josh Petrie]

I was taking a stab at resolving issue 731, related to factoring the generated code to conform to our coding standards, and it occurred to me it would be much easier if I didn't have to cope with the T4 files - Visual Studio insists on doing silly things with them, like trying to execute them when I save. So I've prototyped a replacement template system; a proof of concept is attached to this email as a patch file.

Currently, the generator has a series of types that essentially constitute an object model for C# code. The idea behind the template implementation is to homogenize the access to that object model, treating it like a tree. The CodeObject interface -- which is quite poorly named, and suggestions for a better one are welcome -- does this. The appropriate CSharp* classes would implement CodeObject, which mandates a Name, Namespace and Content property. The first two should be self-explanatory. The last one is a bit unique. Content is a System.Object, which at runtime can be a scalar value or an IEnumerable<CodeModel>. More on this later.

The TemplateProcessor object is used to actually evaluate template replacements. It has a single processing method which is called with a CodeObject input. It understands the association between a concrete CodeObject type and the template via because the concrete type is decorated with a TemplateAttribute object that specifies the file name. TemplateProcessor does simple regular expression replacements for the <{name}> and <{namespace}> fields, but checks the actual type of the content field to see if it should be replaced as a simple scalar, or recursively as a list of children. In the recursive case, care is taken to preserve the text prefixing the <{content}> item and insert newlines as appropriate in order to generate clean code. Double-evaluation is also prevented.

In the prototype, I've implemented this pattern for enumerations. If you run the generator in the debugger, it will produce a bunch of source files for each enumeration under the appropriate Generated directory for each API.

Keep in mind this is only about ten minutes of work, so there are issues. I'm already a bit concerned about the strangeness of the Content property and the special-case handling. This system is very similar to the view object model we use for our tools at work -- all game content objects can be transformed, either via a default transformation or a custom one, into a view object (which is more akin to an interpretation of the content data than an actual UI view), which consists essentially of a value and some children. Content, in the template model I've built here, rolls value and children into one, and I'm not sure I'm happy with that.

-- jmp

[Alexandre Mutel]

Having something to navigate into the object model is good, althgouh looks like CodeObject is somewhat related CSharpContainer... but probably not exactly as you intend to do...

 

I’m not sure about your design, if it’s the object itself (like CSharpInterface), that is going to implement it’s templatization or a TemplateInterface or a generic Templatizer? I would not be confident with a generic templatizer, but with some subclass of the templatizer.

 

Apart from T4 files, I’m also not really happy about the template code, which is spawned between the T4 files and the code. The most complex case is handling interfaces, method declarations and method body. If you look at CSharpMethod and interfaces.tt, you will find that part of the template is done in the CSharpMethod, because It would have been difficult to write this inside a tt files, and this is far from being maintainable as it is.

 

Btw, did you install Tangible T4 Editor free edition in VS2010? It will help you at least to have a colorized view of the templates and identify what is part of C# code (yellow background) and what is text template (white background). If you are going to migrate all the templates to a new design, It will help you a lot to avoid any big headaches while extracting things from tt files...

 

Also, in your design, don’t forget that you need sometimes to recursively call the template. This is used in struct that can have inner structs, as well as for interface that can have inner interfaces (the one that are used for moving methods stage-specialized from ID3D11DeviceContext to an inner interface).

 

[Josh Petrie]

Having something to navigate into the object model is good, althgouh looks like CodeObject is somewhat related CSharpContainer... but probably not exactly  as you intend to do...

Mike pointed out the container class as well when I ran the idea by him in IRC. It's probably usable for this purpose.

 I’m not sure about your design, if it’s the object itself (like CSharpInterface), that is going to implement it’s templatization or a TemplateInterface or a generic Templatizer? I would not be confident with a generic templatizer, but with some subclass of the templatizer.

I don't know what you mean. The TemplateProcessor class is what is responsible for replacing tokens in a template file with values from the input object. Since all input objects are homogenized by the CodeObject interface (or something similar), there is no need for TemplateProcessor to be subclassed per type. The association between a type like CSharpInterface and its template is currently established by decorating CSharpInterface with an attribute specifying which template to use. This mapping could also be manually established via methods on the processor itself.

 

 Apart from T4 files, I’m also not really happy about the template code, which is spawned between the T4 files and the code. The most complex case is handling interfaces, method declarations and method body. If you look at CSharpMethod and interfaces.tt, you will find that part of the template is done in the CSharpMethod, because It would have been difficult to write this inside a tt files, and this is far from being maintainable as it is.

 I will try and extend my prototype to cover interfaces this evening and see how well that can be mapped into the current system. 

 

Btw, did you install Tangible T4 Editor free edition in VS2010? It will help you at least to have a colorized view of the templates and identify what is part of C# code (yellow background) and what is text template (white background). If you are going to migrate all the templates to a new design, It will help you a lot to avoid any big headaches while extracting things from tt files...

That's interesting, although I can read them just fine; it's not their syntax that annoys me (it's actually not too bad), just their half-baked integration into Visual Studio and the weight of the dependency required to actually process them.

 

 Also, in your design, don’t forget that you need sometimes to recursively call the template. This is used in struct that can have inner structs, as well as for interface that can have inner interfaces (the one that are used for moving methods stage-specialized from ID3D11DeviceContext to an inner interface).

The templates are already evaluated recursively. They should be able to handle nested types, although we've never needed a nested type in v1, because they're ugly, and I don't see why we'd need one now -- in fact, I've updated the coding standards page on the wiki to account for this.

-- jmp

[@lx]

> The templates are already evaluated recursively. They should be able to
> handle nested types, although we've never needed a nested type in v1,
> because they're ugly, and I don't see why we'd need one now -- in fact, I've
> updated the coding standards page on the wiki to account for this.

They are mostly used in structure inside another structure, but they
are used only once.
I'm using them for a structure like "DepthStencilViewDescription".
Have a look at the SlimDX version 1 and 2. I have never found the
SlimDX version 1 to be easy to read and to map to the original one.
Because I'm loosing the information of what I need to feel (you don't
know which field correspond to which type of texture, this is really
annoying). So the SlimDX 2 version is stricly the same than the
native one, leading to a more comprehensive one.

Those inner structure are not used outside referencing the inner
fields, so they are not even poluting the global namespace.

[Josh Petrie]

I see what you're trying to do in terms of what the description structure would look like, although I'm not sure it's the best way to solve that problem. Regardless of what the API ends up looking like, though, there's no reason that we need nested types to implement that technique.

-- jmp

--
You received this message because you are subscribed to the Google Groups "slimdx-devel" group.
To post to this group, send email to slimdx...@googlegroups.com.
To unsubscribe from this group, send email to slimdx-devel...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/slimdx-devel?hl=en.

[Alexandre Mutel]

Why not creating a less restrictive rules like “Do not create nested types when they are referenced outside the container”

 

I mean, I’m not against rules, but I don’t really understand where does this rule is coming from, why It is more important than correctness and practical use of a library?

[Josh Petrie]

Coding standards are, by and large, stylistic. There are no objective reasons why we've opted to do most of the things we've codified into our standard. Nor are we personally agreed on all of them -- for example, Washu and I both prefer K&R brace style. What is important is that we have a standard that our code should adhere to.

If there are compelling objective, technical arguments against a particular stylistic choice, then we can debate them. For example, we do not currently call out whether or not "using" directives should be outside or inside the namespace in files, and there are technical differences between the two, so that would be a debatable issue. But if something is just subjective, we aren't going to debate it.

In the example you've provided where nested types are used, it is possible to implement the same API without the nested type. Consider

public struct T {
  public struct U {

  }

  public U U { get; set; } 
}

Given an instance of T called t, one accesses the property U as "t.U" and this is trivially transformable to 

public struct U {

}

public struct T {

  public U U { get; set; } 
}

which preserves the same API. Both structs must be public in order to access the property, and in both cases the reference to the inner type is effectively the same (practically speaking names are more involved than T and U, but in the worst case the difference between the full type name is a matter of a single dot). There's no technical reason to employ an inner structure here, so it remains a subjective issue. We have not encountered a scenario where we needed a nested type, even a private one.

As for the API itself, as I've noted I've never seen any other users take that particular issue with the implementation of the description, and although I do agree it could be done clearer, I'm not sure the faux-union style properties hanging off the description object are the most ideal way either. That, however, is probably a discussion for later on.

-- jmp

--

[Alexandre Mutel]

As I told you, the technical argument is that the U doesn’t need to be exposed outside T because It is not intended to be used outside the usage : t.U.field_Of_U. A nested struct in this case makes sense because you don’t pollute the global namespaces. That’s what namespaces are for, and T is a namespace in itself.

 

Just one thing is that you took the decision of inputing in the coding standard wiki the “do not create nested types” before even debate about it. Things are settled before discussion and as you said, working in a team has some rules too, and I don’t feel that this kind of behavior makes that members of team are working well together.

 

 

From: Josh Petrie

Sent: Wednesday, November 17, 2010 6:57 PM

To: slimdx...@googlegroups.com

Subject: Re: New templating prototype

 

[Josh Petrie]

As I told you, the technical argument is that the U doesn’t need to be exposed outside T because It is not intended to be used outside the usage : t.U.field_Of_U. A nested struct in this case makes sense because you don’t pollute the global namespaces. That’s what namespaces are for, and T is a namespace in itself.

But U is public in both cases, and just as easy to use in isolation in both cases, even if that is not the intended usage.

Just one thing is that you took the decision of inputing in the coding standard wiki the “do not create nested types” before even debate about it. Things are settled before discussion and as you said, working in a team has some rules too, and I don’t feel that this kind of behavior makes that members of team are working well together.

Our project is not a democracy; decisions concerning the direction and vision of the API and its development process -- such as coding standards -- are ultimately up to the principle three developers, generally in terms of seniority. It's just like working at a company -- while any individual developer may offer commentary on the company's coding standards, it's generally the lead engineer who has the final say.

-- jmp 

On Wed, Nov 17, 2010 at 12:38 PM, Alexandre Mutel <alexand...@yahoo.fr> wrote:

 

 

 

From: Josh Petrie

Sent: Wednesday, November 17, 2010 6:57 PM

To: slimdx...@googlegroups.com

Subject: Re: New templating prototype

 

Coding standards are, by and large, stylistic. There are no objective reasons why we've opted to do most of the things we've codified into our standard. Nor are we personally agreed on all of them -- for example, Washu and I both prefer K&R brace style. What is important is that we have a standard that our code should adhere to.

 

If there are compelling objective, technical arguments against a particular stylistic choice, then we can debate them. For example, we do not currently call out whether or not "using" directives should be outside or inside the namespace in files, and there are technical differences between the two, so that would be a debatable issue. But if something is just subjective, we aren't going to debate it.

 

In the example you've provided where nested types are used, it is possible to implement the same API without the nested type. Consider

 

public struct T {
  public struct U {

  }

  public U U { get; set; }
}

 

Given an instance of T called t, one accesses the property U as "t.U" and this is trivially transformable to

 

public struct U {

}

public struct T {

  public U U { get; set; }
}

 

which preserves the same API. Both structs must be public in order to access the property, and in both cases the reference to the inner type is effectively the same (practically speaking names are more involved than T and U, but in the worst case the difference between the full type name is a matter of a single dot). There's no technical reason to employ an inner structure here, so it remains a subjective issue. We have not encountered a scenario where we needed a nested type, even a private one.

 

As for the API itself, as I've noted I've never seen any other users take that particular issue with the implementation of the description, and although I do agree it could be done clearer, I'm not sure the faux-union style properties hanging off the description object are the most ideal way either. That, however, is probably a discussion for later on.

 

-- jmp

--