Replacing preprocessor macros in the language

[stackm...@hotmail.com]

Hi there!

I've had this idea in my head for a long time and I'm wondering if it is worth writing up a proposal. In the following post I am going to try to explain the basics without going into too much technical details.

The basic idea is to introduce the notion of a macro into the language. A macro is a constexpr function that - instead of returning something - emits or injects code at the call site. Let me start out with a simple example:

macro times(unsigned n)
{
    inject "for(unsigned i = 0; i != %, ++i)", n;
}

The inject statement is kind of like a printf statement, the special syntax allows you to insert the value of an expression into code.

This macro can then be used like this:

$times(10)
    std::cout << "Hello, World!\n";

I propose that the call syntax be different from regular function calls to make it more clear what's going on. In this case I used the $-sign.
The keywords are of course subject of bikeshedding, I like 'macro' or 'codegen' and 'emit' or 'inject'.

This feature becomes especially powerful with C++14's mutable compiletime state and upcoming comiletime reflection features. One could easily imagine iterating over the members of a class to generate comparison operators or serialization routines using this feature:

The standard library could provide some predefined macros for this.

Is this a feature people would like to have?

[stackm...@hotmail.com]

Whoops! Somehow my last piece of example code got lost. Here it is:

template <typename T>
macro defineEq()
{
    inject "bool operator == (%0 lhs, %0 rhs) { return true", std::meta::full_name_of<T>();

    for(auto mem : std::meta::members_of<T>())
        inject " && lhs.%0 == rhs.%0", mem.name;

    inject "; }";
}

[Klaim - Joël Lamotte]

On Tue, Oct 29, 2013 at 10:12 AM, <stackm...@hotmail.com> wrote:

Is this a feature people would like to have?

I think you need to clarify how it would differ from the pre-processor macro and how it would be beneficial compared to, say, templates.

[Ville Voutilainen]

Certainly. Do explain why it wouldn't be  something like

inline template<unsigned int N> constexpr void times()

// or inline template<> constexpr void times(unsigned int N)

{

    for (unsigned int i = 0; i != N, ++i)

}

?

Now, this is practically a static if for which the condition always evaluates to true.

And like a static if, it should probably be valid syntactically (which the example above
isn't), and it should establish a new scope. Otherwise the implementation problems
of static if will arise again.

[David Krauss]

On 10/29/13 5:12 PM, stackm...@hotmail.com wrote:
> This feature becomes especially powerful with C++14's mutable compiletime
> state and upcoming comiletime reflection features. One could easily imagine
> iterating over the members of a class to generate comparison operators or
> serialization routines using this feature:
> The standard library could provide some predefined macros for this.
>
> Is this a feature people would like to have?
>

This has merit, but it would probably be too difficult to add to
existing implementations. Assuming you intend for flow control and
computation to go into the macro besides the inject statement, the
compiler has to fully interpret all definitions up to the current state
of the lexer. It's not theoretically impossible, and the march of
technology perhaps makes it inevitable, but nobody is in the ballpark yet.

By the way, the current standard requires evaluation of constexpr
user-defined literals in #if conditions. I noticed this soon after
setting about writing my own preprocessor, and felt like I'd found a
wonderful loophole. Alas, it is not the "design intent." Nevertheless,
as long as nobody proposes a solution, it remains a requirement for
strict compliance.

Thanks for the idea� got me thinking about modules�

[George Makrydakis]

On Tuesday, October 29, 2013 12:40:57 PM UTC+2, David Krauss wrote:

On 10/29/13 5:12 PM, stackm...@hotmail.com wrote:
> This feature becomes especially powerful with C++14's mutable compiletime
> state and upcoming comiletime reflection features. One could easily imagine
> iterating over the members of a class to generate comparison operators or
> serialization routines using this feature:
> The standard library could provide some predefined macros for this.
>
> Is this a feature people would like to have?
>

[ snip ]

Thanks for the idea� got me thinking about modules�

It just so happens that every time somebody comments on preprocessor deficiencies, a thousand different posts pop up around with alternatives for the sake of good argument. His idea just replaces the current preprocessor, with another preprocessor. Extending one of the type safe metaprogramming features C++ already has would be a more favourable and permanent approach to the problem as a whole. If you start just upgrading the preprocessor alone to do more "type safe stuff" and the like, you will end up with re - implementing the template system to a point. Why not just upgrade the template system itself then and spare yourself the trouble?

I have said myself before in the __VA_NARGS__ thread, once modules kick in, it will be a lot clearer what those extensions to the template / constexpr metaprogramming facilities should be all about. After all, module proposals would  eventually have to deal with partial specializations in the contained instantiations of a given module and so on which themselves would require dealing with metaprogramming - wise. This kind of shows the way for upgrading the semantics of the current templates system for modules to actually work unambiguously.

So, perhaps this is in the end something that should be considered at some point when dealing with modules while showing the preprocessor the way out and proposals dealing with either of these issues should explicitely analyze the theoretical implications of said intervention. His idea does not.

[Vittorio Romeo]

This is a feature I definitely want to have. 
Writing mixins/complex compile-time fuctions/avoiding code repetition can be painful at times, and adding complex macros does not help readability. 
Eventually, I believe C++ needs some language feature like this one.

[Evgeny Panasyuk]

Several thoughts:

1. D language has Mixins, which has very similar use-cases and minimalistic design: http://dlang.org/mixin.html - it takes compile time string as a parameter and emits code in-place.
2. This feature is very very powerful. But still - string manipulation is just the weakest form of metaprogramming.
3. I am sure, with help of mixins it is possible to generate very fast code in some special cases.
4. Both examples you showed are just not convincing.
a) "times" - such use cases overlaps greatly with lambdas. If you think that lambdas syntax is noise, then at first we should try to improve it, like terse notation:

transform(first, last, first, [](x) x*x );
// or
times(10) * [](x) cout << x; // using operator*

b) At defineEq you used compile-time reflection. If we would have it - we don't need any macros to generate operators/hash functions/serialization - we could use just something like boost::fusion::fold, which looks much more prettier than strings injection: http://coliru.stacked-crooked.com/a/e4dc167785088c81

struct Hasher
{
    template<typename T>
    size_t operator()(size_t hash, T x)
    {
        boost::hash_combine(hash, x);
        return hash;
    }
};

template<typename Struct>
size_t hash_value(const Struct &x)
{
    return boost::fusion::fold(x, size_t(0), Hasher{});
}

5. Every time you want to use macro meta-programming - most likely there is lack of some good feature in language.

[Sean Middleditch]

On Tuesday, October 29, 2013 9:51:55 AM UTC-7, Evgeny Panasyuk wrote:

Several thoughts:

1. D language has Mixins, which has very similar use-cases and minimalistic design: http://dlang.org/mixin.html - it takes compile time string as a parameter and emits code in-place.

Following up on the rest of your points... these are by far my least favorite feature of D due to the near impossibility of ever being able to write proper tools for them.  C++ is bad enough here thanks to the preprocessor and pre-concepts templates.  Existing external code-generation tools are a nightmare when it comes to debugging typically.  Maintainability and "toolability" matter more than minor speed gains to most industries (even performance-oriented ones like games) _way_ more than gaining a few percent in runtime speed (outside of the most inner of loops, at least, which typically don't need fancy code generation).

Meta-programming can also be a pain because you can only debug the resulting code, not the generation of the code (there's no way to step through template expansions or inspect templates as they're being evaluated).  This is a problem to look towards fixing, not compounding.  Exasperating the problem even further by having a glorified M4 macro code generator in the language forever and ever is a giant leap in the wrong direction.

I don't think anybody has proposed Mixin-like functionality for C++, but figured I'd do my best to nip it before it goes anywhere.

[Evgeny Panasyuk]

I don't think anybody has proposed Mixin-like functionality for C++, but figured I'd do my best to nip it before it goes anywhere.

Well, while it is nice to have feature - I agree it is very controversial, and has very low priority.
But I don't think that it will make situation worse. I think it is the opposite, it will help to simplify code which is already encrypted, and will help to reduce compile times.
And I think that speed gains will be much bigger than few percents, and that will enable new EDSLs which were impractical before.
Just think what kind of code libraries like Eigen, Boost.Spirit, Boost.Xpressive could generate - currently they rely on optimizer and aggressive inlining, but it has its limits. With help of direct code injection they can generate the fastest possible code.
Yes, certainly that feature will be abused in wrong and unmaintainable ways - but if somebody will do that, he will deserve what he will get.

[Sean Middleditch]

On Wed, Oct 30, 2013 at 12:32 AM, Evgeny Panasyuk
<evgeny....@gmail.com> wrote:
>> I don't think anybody has proposed Mixin-like functionality for C++, but
>> figured I'd do my best to nip it before it goes anywhere.
>
>
> Well, while it is nice to have feature - I agree it is very controversial,
> and has very low priority.
> But I don't think that it will make situation worse. I think it is the
> opposite, it will help to simplify code which is already encrypted, and will
> help to reduce compile times.

You do not appear to be talking about what I'm talking about here. :)

I am specifically referring to the D-like mixin system, not a
high-level macro system. Clearly the latter is a good thing to have
compared to what we have today. D's approach is simply not the right
way to do it. Rust is a better role model here at a semantics level
(not necessarily the syntax level):
https://github.com/mozilla/rust/blob/master/doc/tutorial-macros.md

[Evgeny Panasyuk]

You do not appear to be talking about what I'm talking about here.  :)

I am talking about macro system embedded into language, not external one like current preprocessor.
 

I am specifically referring to the D-like mixin system, not a
high-level macro system.  Clearly the latter is a good thing to have
compared to what we have today.  D's approach is simply not the right
way to do it.  Rust is a better role model here at a semantics level
(not necessarily the syntax level):
https://github.com/mozilla/rust/blob/master/doc/tutorial-macros.md

I think D's mixin is bare minimum which will be feature-enabler.
I am talking about something like:

mixin( constexpr_function_returning_string(arg1, arg2, ag3)  ) // injects code in-place
//or
mixin( some_struct<arg1, arg2, arg3>::code_string )  // injects code in-place

Of course higher level macro system will be better, but I guess it will take much more time to standardize.

[George Makrydakis]

On Wed, Oct 30, 2013 at 10:05 AM, Evgeny Panasyuk <evgeny....@gmail.com> wrote:

You do not appear to be talking about what I'm talking about here.  :)

I am talking about macro system embedded into language, not external one like current preprocessor.

If we are talking about an expression  - tree friendly solution, these would be far from being just able plain text substitution macros (which are the problem that causes all others actually). There is a design trilemma here and it should not be taken lightly if we are to adroitly resolve systematic code generation issues (not mere text substitution) :

1. In the eventuality we totally eliminate the preprocessor for when it comes to "macros" and we start implementing high level expression tree friendly macros with term rewriting features (similar to what Nimrod does for example) or even use a more Scala - like approach that is equally powerful and type - safe (http://scalamacros.org) that means that we go towards a "preprocessor" that starts doing compile - time evaluation within macro definitions themselves (and not just in the conditional directives). This would mean that we introduce yet another metaprogramming language within C++ side to templates and constexpr metaprogramming. It is not entirely convincing that we need yet another turing complete (or almost turing complete) metalanguage for such a thing since a lot of the features it would have to have, in the end would overlap with template metaprogramming, first and foremost.
2. We do not introduce another preprocessor in order to deprecate the current one (or extend the current one for that matter) but we do extend / modify either or both the template and constexpr metaprogramming facilities that are already evolving in the language itself in such a way as to allow for expansion of constructs that are specific to syntactically valid and repetitive code constructs commonly used within the language prior to any kind of compile time evaluation. In a sense the semantics of parameter unpacking especially when multiple parameter packs are involved (http://en.cppreference.com/w/cpp/language/parameter_pack offers a decent overview) herald the way of such a thing being not only possible with minor syntactic intrusion of new identifiers, but eventually inevitable. And type safety in this cases is a given.
3. Convert the current preprocessor into a turing tarpit by adding small features that allow for seriously problematic "solutions" to common problems for no other reason than providing evidence that certain things are done for the sake of proving cognitive superiority over feeble - minded languages like shell scripting oriented ones.

I tend to believe that solution 2 is the more future - proof way of dealing with such a problem. Template metaprogramming can be characterized as a pure, non - strict, untyped functional programming language with pattern matching and I think we have little reason not to expand its semantics to cover code generation for more complicated constructs. Such "constructs" already available in valid C++11 where multiple parameter packs are unpacked in various combinations in a single blow.

That behaviour is code generation itself introduced because variadic templates were introduced. Extensions to this way of thinking could eventually solve some of the problems involved with partial specializations in modules if they were allowed to expand or modify to type safe constructs according to input describing module features and facilities (provided they were within the template system but with semantics and syntax that would isolate them properly from the rest of the instantiation cascades and their consequences). This could in the end solve some of the design problems afflicting modules and the template instantiations they include.

 

I am specifically referring to the D-like mixin system, not a
high-level macro system.  Clearly the latter is a good thing to have
compared to what we have today.  D's approach is simply not the right
way to do it.  Rust is a better role model here at a semantics level
(not necessarily the syntax level):
https://github.com/mozilla/rust/blob/master/doc/tutorial-macros.md

I think D's mixin is bare minimum which will be feature-enabler.
I am talking about something like:

mixin( constexpr_function_returning_string(arg1, arg2, ag3)  ) // injects code in-place
//or
mixin( some_struct<arg1, arg2, arg3>::code_string )  // injects code in-place

Of course higher level macro system will be better, but I guess it will take much more time to standardize.

It would take some deeper analysis on why I believe that D's mixin solution does not actually help the cause we are after and I am not alone in this (I think). As for Rust, I reserve my personal opinion until it reaches 1.0 but if you look at it as it is right now, competent C++ programmers who deploy well - orchestrated preprocessor, template and constexpr metaprogramming styles - despite the disproportionately tenuous cognitive overhead for non - experts - end up having superior results.

[Stack Machine]

"How does this differ from preprocessor macros?"
It differs from preprocessor macros in that you can have control flow withing your macro. This feature is supposed to replace preprocessor macros completely. It can do anything preprocessor macros can do, except at the language level, better readable and it is more powerful. Not only that, I'd also like to get rid of external code generation tools. The fact that people use these tools indicates a lack of language features.

I agree that strings are not ideal if used directly. The standard could provide utilities for generating these strings.

template <typename T>
macro generateEq()
{
    std::meta::expression ret = true;

    for(auto mem : std::meta::members_of<T>())

        ret = ret && fb[0].mem == fb[1].mem; // overloaded operators to build expressions
   
    std::meta::function_builder<bool(T, T)> fb("operator ==");
    fb.append(std::meta::return_statement(ret));
    inject fb.str();
}

These classes can be built on top of the basic macro system that uses strings.

Maybe in this specific use case boost::fusion::fold is indeed better. (I wasn't aware of the existence of it.) But there are a lot of cases where external code generators are used and I'd like to get rid of the need for them.

[Vicente J. Botet Escriba]

Le 01/11/13 13:24, Stack Machine a écrit :

Hi,

I have taken a similar approach with what I've called frames. While the example is not enough motivating this big feature, it is simple enough to understand the power of the feature. Instead of macro I have used frame and instead of inject I have used a new operator [[[ ]]].
As your approach, the advantage is the simplicity of the user interface. It allows to mix direct code and transformations using reflection. IMO this interface is closer to the user expectation, as using template meta-programming (in the general case, not this simple case) would be quite complex and request the compiler a lot, well this is what I believe, but I'm not a compiler writer.

To generate the equality operator we can use a frame template DefineOperatorEqual that will do what we expect.

struct X {
  int a;
  string b;
  [[[ DefineOperatorEqual<X> ]]] //or better syntax
};

where

template <typename C>
frame DefineOperatorEqual
(
    meta::Id_t ... Member = meta::data_members_name_of<C>()
) -> meta::function_declaration_t
[[[
  bool operator == (C const& lhs, C const& rhs) {
     return true
     [[[ && lhs.Member == rhs.Member ]]]...
     ;
  }
]]]

The use of [[[ and ]]] is deliberated to disambiguate the parser, as a frame transformation can have any C++ code. You ca see it as { }.

We can see a frame as a program transformation, that takes nodes of C++ BNF grammar as parameters and return a new node.
The parameter of the preceding frame are Id (as C++ identifiers) that has a default value that depend on the template parameter C.
These Id can be used as any other identifier and the [[[ ]]] transformation will replace them by its respective symbols. This works similar to a macro.

When we want to repeat something on variadic parameters we can use the operator... on the sub-frame we want to repeat enclosed by [[[ ]]].

What do you think?

Best,
Vicente

P.S. I'm working on other cases that would be much more complex using template meta-programming than using this frame concept.
P.P.S. My original frame design was used to also extend the C++ grammar, but this is off topic.

[Vicente J. Botet Escriba]

Le 01/11/13 20:13, Vicente J. Botet Escriba a écrit :

Hi, here it is

the example is related to the use case of calling a specific function before delegating.
Given

#include <delegate>
class C {
public:
int i;
void f();
vector<int> g(double);
private:
void h(int);
};

The generic delegate_before_call<C,F,B> should be identical to the manual specialization below

template<class F, class B>
class delegate_before_call<C, F, B> : public B
{
  using this_type = delegate_before<C, F, B>;
  friend F;
public:
  delegate_before_call(C &c, F f) : wrapped(c), fct(f) {}
  void f() {
    fct( ); // (1.a)
    wrapped.f();
  }
  vector<int> g(double d) {
    fct( ); // [2.a]
    return wrapped.g(d);
  }
};

Using the frame the solution almost as short as if we made the specialization by hand

template<class T>
frame delegate_members_before_call[
meta::member_function ...Member = meta::public_function_members_of<T> >
] -> DeclarationList
[[[
  [[[
  Member::signature {
    fct( );
    return wrapped.Member::forward_call() ;
  }
  ]]]...
]]]

template<class T, class F, class B>
class delegate_before_call : public B
{
  using this_type = delegate_before_call<T, F, B>;
  friend F;
public:
  delegate_before(C &c, F f) : wrapped(c), fct(f) {}
 [[[ delegate_members_before_call<T>[] ]]]
};

This is simple for the user. Maybe complex for the compiler.

I have found a lot of difficulties to find a good design for solving this problem without this new frame feature.
How to generate the definition of the new function? 

  vector<int> g(double d) {
    fct( );
    return wrapped.g(d);
  }

Using lambdas?

How to include these function definitions in the new class? Using an inheritance hierarchy has some troubles with the wrapped overloaded functions, as show by Boost.TypeErasure implementation?

IMO, mixing code and [[[reification]]] via frames is the way to go so solve these kind of problems.

I would like to see what other can propose using know template meta-programming techniques.

Best,
Vicente

[George Makrydakis]

Please clarify whether "frame" and your proposed "operator [[[ ]]]" work as cues for expanding code constructs that are respecting type safety / language grammar and when should such operations take place.

It seems that you are you just adroitly attempting to move function - like macro definition and invocation in a form of expansion within the definition scope of class / function templates, while giving some constant expression evaluation characteristics in said operation.

Should that hold, you would just be forcing tightly coupled evaluations to occur prior to template instantiation and even definition. Have you considered where such a way would clash with templates / constexpr ?

I am just saying that by wanting to remove the need for preprocessor metaprogramming with the current facilities, in the end the route is to augment constexpr / template metaprogramming, not introduce another preprocessor.

I would thus like to see where this unambiguously extends templates and constexpr and how it deals with the simple degenerate case of plain text substitution and concatenation, even when the strings in question are totally random (just like what happens with the preprocessor now).

As for people who rely on external tools for code generation, I think they are simply trying to avoid current preprocessor metaprogramming techniques for some subjective disadvantage. Examples like boost's own use demonstrates otherwise.

[Vicente J. Botet Escriba]

Le 03/11/13 01:45, George Makrydakis a écrit :

Please clarify whether "frame" and your proposed "operator [[[ ]]]" work as cues for expanding code constructs that are respecting type safety / language grammar and

Yes, a frame has nodes of the language grammar as parameters and return a specific grammar node.

when should such operations take place.

Good question. I have not a response now and I have no responses to some of your good questions. I have not worked the approach enough to make a concrete proposal.

It seems that you are you just adroitly attempting to move function - like macro definition and invocation in a form of expansion within the definition scope of class / function templates, while giving some constant expression evaluation characteristics in said operation.

Right.

Should that hold, you would just be forcing tightly coupled evaluations to occur prior to template instantiation and even definition.

Why do you think this?

Have you considered where such a way would clash with templates / constexpr ?

No, not yet. Which one do you see?.

I am just saying that by wanting to remove the need for preprocessor metaprogramming with the current facilities, in the end the route is to augment constexpr / template metaprogramming, not introduce another preprocessor.

I don't see a frame as a preprocessor but as a term rewriting system. I want to be able to

* build grammar node instances by reflection
* transform grammar nodes instances on grammar nodes instances, and
* expand the code represented by a grammar node instance by reification.

The operator [[[]]] taking as parameter some code results in a grammar node.
Frames transform grammar nodes on grammar nodes.
The operator [[[]]] taking as parameter a grammar node results in the associated code.
Which syntax is more appropriated to support these requirements needs refinement.

A grammar node instance is represented as a type on the std::meta namespace.
The compiler should provide some intrinsic meta-functions to obtain grammar node instances from C++ types.

My approach doesn't takes the augment constexpr / template metaprogramming route, and maybe the route I'm taking is a dead-end.

 

I would thus like to see where this unambiguously extends templates and constexpr and how it deals with the simple degenerate case of plain text substitution and concatenation, even when the strings in question are totally random (just like what happens with the preprocessor now).

As I see frames, they are orthogonal to templates and constexpr. The examples I gave are template frames, but we could have frames that are not templates. A frame would be implicitly a constexpr as if the parameters are know at compile time, the result is obtained at compile time. If wewere  able to build grammar nodes at run-time, the frames could be evaluated at run-time.

While at a first glance it could seem that frames are nothing more than macros, their parameters are language grammar nodes. This makes a big difference and IMHO it opens to more elaborated transformation techniques.

As for people who rely on external tools for code generation, I think they are simply trying to avoid current preprocessor metaprogramming techniques for some subjective disadvantage. Examples like boost's own use demonstrates otherwise.

I have nothing against external tools, but I think this is off topic on this ML as far as the generation is external.

I have used the preprocessor meta-programming techniques in TBoost.Enums and  TBoost.Opaque and while this allows to experiment I would like to see something more appropriated introducing reflection in the language.

Ville has suggested in [1] an approach that is more in line with the current language. operator member_name when member_name is a compile time string would define a new function. meta::invoke_member allows to call to a function identified by a compile time string.

template <Compile_time_string member_name, class ...Args>
  decltype(auto) operator member_name(Args&&... args)
{

   fct();

   decltype(auto) ret = meta::invoke_member<T>(this, member_name, forward<Args>(args)...);

   return ret;

}

Is like this approach as it requires less changes to the standard.

Best,
Vicente

[1] https://groups.google.com/a/isocpp.org/forum/#!topic/reflection/HsjY_3IznHc

[George Makrydakis]

On Wed, Nov 6, 2013 at 1:56 PM, Vicente J. Botet Escriba <vicent...@wanadoo.fr> wrote:

Le 03/11/13 01:45, George Makrydakis a écrit :

Please clarify whether "frame" and your proposed "operator [[[ ]]]" work as cues for expanding code constructs that are respecting type safety / language grammar and

Yes, a frame has nodes of the language grammar as parameters and return a specific grammar node.

when should such operations take place.

Good question. I have not a response now and I have no responses to some of your good questions. I have not worked the approach enough to make a concrete proposal.

First of all, you will have to excuse my delayed reply, it was due to my own limited availability. Make a note of this reply of yours here because I will be using it later on.

It seems that you are you just adroitly attempting to move function - like macro definition and invocation in a form of expansion within the definition scope of class / function templates, while giving some constant expression evaluation characteristics in said operation.

Right.

Should that hold, you would just be forcing tightly coupled evaluations to occur prior to template instantiation and even definition.

Why do you think this?

Imagine wanting to repetitively create a series of repetitive constructs within a given function / class template definition, where each item is derived from a series of numbers within the [0,N] range where N is any positive integer. Notice that in order for such a template definition to occur, all the syntactically valid constructs within definition scope would have to be present. Since we are talking about systematically generating these at the level of a "preprocessor substitute" (as in your concept), we would have to have a way to control such expansion, some sort of flow control. Such control means having expression evaluation implemented in some form. A prime example of hijacking concatenation into creating such "expression evaluation" mechanics are the internals of boost libraries, where boost.preprocessor is used. Take note that this is very well defined as to where and when everything takes place: this is "text" generation in the end, prior to any action on behalf of the compiler.

I think that the need for such a solution could be done away with if the templates / constexpr system were extended properly. I do not have myself a complete solution on this, but as everyone on the grounds of this mailing list I just have a few ideas about the "dont's" and some few pretty concrete directions as to the "do's". It would take me quite a while to explain the reasons about them and eventually I will. One aspect is actually modifying parameter expansion pack semantics. More on that later on.

Have you considered where such a way would clash with templates / constexpr ?

No, not yet. Which one do you see?.

The one I described above. It is introducing another system for what templates / constexpr should be allowed to do (with a few expansions to their semantics). In a way, through parameter unpacking, templates have already acquired a way to expand type - dependent (and safe) constructs for any given number of type / non type / template template type parameters used. This is the consequence of introducing variadicity in template parameters. In the C++11 standard, 14.5.3 has several examples for your enjoyment. You can also watch Andrei Alexandrescu's excellent talk about variadic templates being "funadic" at http://channel9.msdn.com/Events/GoingNative/GoingNative-2012/Variadic-Templates-are-Funadic. There are several ways to do different things with parameter pack expansion, even more than several when it comes to multiple parameter packs. I am just citing one of Alexandrescu's examples:

template <class... Ts> void fun( Ts... vs ) {
    gun( A<Ts...>::hun(vs)...);
    gun( A<Ts...>::hun(vs...));
    gun( A<Ts>::hun(vs)...);
}

This is not the most complicated thing you will see in parameter pack expansion but it kind of foreshadows how one can deploy extensions to unpacking patterns to cover for a few more scenarios where more complicated constructs are required (no, you can't do it with unpacking as it is with C++11, C++14). I reserve the right to explain this further in a future post all for just that (hopefully). It is not an easy sight and it would require some additional syntax to make things unambiguous because (...) is not enough.

I am just saying that by wanting to remove the need for preprocessor metaprogramming with the current facilities, in the end the route is to augment constexpr / template metaprogramming, not introduce another preprocessor.

I don't see a frame as a preprocessor but as a term rewriting system. I want to be able to

* build grammar node instances by reflection
* transform grammar nodes instances on grammar nodes instances, and
* expand the code represented by a grammar node instance by reification.

The operator [[[]]] taking as parameter some code results in a grammar node.
Frames transform grammar nodes on grammar nodes.
The operator [[[]]] taking as parameter a grammar node results in the associated code.
Which syntax is more appropriated to support these requirements needs refinement.

A grammar node instance is represented as a type on the std::meta namespace.
The compiler should provide some intrinsic meta-functions to obtain grammar node instances from C++ types.

My approach doesn't takes the augment constexpr / template metaprogramming route, and maybe the route I'm taking is a dead-end.

As I see frames, they are orthogonal to templates and constexpr. The examples I gave are template frames, but we could have frames that are not templates. A frame would be implicitly a constexpr as if the parameters are know at compile time, the result is obtained at compile time. If wewere  able to build grammar nodes at run-time, the frames could be evaluated at run-time.

While at a first glance it could seem that frames are nothing more than macros, their parameters are language grammar nodes. This makes a big difference and IMHO it opens to more elaborated transformation techniques.

I would not say that it is a dead - end, but I think it needs more work when it comes to clarifying semantics and where operations should take place in relation i.e. to template definition, declaration and instantiation. But you are not at the same problem domain as the preprocessor right now, so you can't substitute it as you wish. Frames (as you are proposing them at least) are not exactly analogous to AST term rewriting macros (see Nimrod's http://nimrod-code.org/) either. Because of the (for the time being!) ambiguous semantics you propose, they clash with established systems as templates and constexpr. Think again about why I asked as to when and where frame operations should take place. This is more important than it seems.

 

As for people who rely on external tools for code generation, I think they are simply trying to avoid current preprocessor metaprogramming techniques for some subjective disadvantage. Examples like boost's own use demonstrates otherwise.

I have nothing against external tools, but I think this is off topic on this ML as far as the generation is external.

I did not understand your reply here. As I said, the preprocessor is fairly adequate if used properly; however it does require some inner knowledge of the techniques involved when doing metaprogramming with it and awareness of the necessary compromises in designing solutions with it. People not particularly eager to gain that knowledge, turn to external tools. Would I like to have something better? Of course I would, but if the preprocessor is all I am given within C++, then I must be ready to make the best use of it. And I have no problems with that either.

 

I have used the preprocessor meta-programming techniques in TBoost.Enums and  TBoost.Opaque and while this allows to experiment I would like to see something more appropriated introducing reflection in the language.

Ville has suggested in [1] an approach that is more in line with the current language. operator member_name when member_name is a compile time string would define a new function. meta::invoke_member allows to call to a function identified by a compile time string.

I am aware of Ville Voutilainen's approaches to problems like this and I am quite a fan of his solutions when it comes to them. But we are going to reflection now, which would derail the entire thread into something different. Systematic generation of valid C++ constructs for reducing hand - written constructs is not something closely related to reflection but as a "workaround" for the lack of said feature at the language level. It is obviously that in the end, you are actually trying to solve a compile / run - time barrier problem, hence your interest in reflection. Try thinking of the "frames" you are proposing as something that should not however just exist to help reflection purposes.

 

[1] https://groups.google.com/a/isocpp.org/forum/#!topic/reflection/HsjY_3IznHc

Really enjoyed your posts of course, sorry for the delayed reply!

George

[Vicente J. Botet Escriba]

Le 11/11/13 22:08, George Makrydakis a écrit :

On Wed, Nov 6, 2013 at 1:56 PM, Vicente J. Botet Escriba <vicent...@wanadoo.fr> wrote:

Le 03/11/13 01:45, George Makrydakis a écrit :

Please clarify whether "frame" and your proposed "operator [[[ ]]]" work as cues for expanding code constructs that are respecting type safety / language grammar and

Yes, a frame has nodes of the language grammar as parameters and return a specific grammar node.

when should such operations take place.

Good question. I have not a response now and I have no responses to some of your good questions. I have not worked the approach enough to make a concrete proposal.

First of all, you will have to excuse my delayed reply, it was due to my own limited availability. Make a note of this reply of yours here because I will be using it later on.

It seems that you are you just adroitly attempting to move function - like macro definition and invocation in a form of expansion within the definition scope of class / function templates, while giving some constant expression evaluation characteristics in said operation.

Right.

Should that hold, you would just be forcing tightly coupled evaluations to occur prior to template instantiation and even definition.

Why do you think this?

Imagine wanting to repetitively create a series of repetitive constructs within a given function / class template definition, where each item is derived from a series of numbers within the [0,N] range where N is any positive integer. Notice that in order for such a template definition to occur, all the syntactically valid constructs within definition scope would have to be present. Since we are talking about systematically generating these at the level of a "preprocessor substitute" (as in your concept), we would have to have a way to control such expansion, some sort of flow control. Such control means having expression evaluation implemented in some form. A prime example of hijacking concatenation into creating such "expression evaluation" mechanics are the internals of boost libraries, where boost.preprocessor is used. Take note that this is very well defined as to where and when everything takes place: this is "text" generation in the end, prior to any action on behalf of the compiler.

The single way to control expansions I have identified so far and proposed in this post is the operator [[[]]]...
Maybe it is not enough. I need to admit that this is just an idea and that there are a lot of undefined things.

I think that the need for such a solution could be done away with if the templates / constexpr system were extended properly. I do not have myself a complete solution on this, but as everyone on the grounds of this mailing list I just have a few ideas about the "dont's" and some few pretty concrete directions as to the "do's". It would take me quite a while to explain the reasons about them and eventually I will. One aspect is actually modifying parameter expansion pack semantics. More on that later on.

My guess is that a *well defined* frame concept could help a lot. Of course, if we can solve a problem with the current standard or doing some alternative modification I don't insists in this frame concept, as I have not analyzed all the implications.

Have you considered where such a way would clash with templates / constexpr ?

No, not yet. Which one do you see?.

The one I described above. It is introducing another system for what templates / constexpr should be allowed to do (with a few expansions to their semantics). In a way, through parameter unpacking, templates have already acquired a way to expand type - dependent (and safe) constructs for any given number of type / non type / template template type parameters used. This is the consequence of introducing variadicity in template parameters.

Templates accepts only types and builtin types as template parameters. I would like to be able to have any part of the language as template parameters. As you see this is just a wish. 

In the C++11 standard, 14.5.3 has several examples for your enjoyment. You can also watch Andrei Alexandrescu's excellent talk about variadic templates being "funadic" at http://channel9.msdn.com/Events/GoingNative/GoingNative-2012/Variadic-Templates-are-Funadic. There are several ways to do different things with parameter pack expansion, even more than several when it comes to multiple parameter packs. I am just citing one of Alexandrescu's examples:

template <class... Ts> void fun( Ts... vs ) {
    gun( A<Ts...>::hun(vs)...);
    gun( A<Ts...>::hun(vs...));
    gun( A<Ts>::hun(vs)...);
}

I watched it some months ago.  It is a good think you have pointed to this link. I have watched it again and I understand now the complexity/ambiguity of my idea.

This is not the most complicated thing you will see in parameter pack expansion but it kind of foreshadows how one can deploy extensions to unpacking patterns to cover for a few more scenarios where more complicated constructs are required (no, you can't do it with unpacking as it is with C++11, C++14). I reserve the right to explain this further in a future post all for just that (hopefully). It is not an easy sight and it would require some additional syntax to make things unambiguous because (...) is not enough.

I agree with you that this is not simple, even the opposite.

I am just saying that by wanting to remove the need for preprocessor metaprogramming with the current facilities, in the end the route is to augment constexpr / template metaprogramming, not introduce another preprocessor.

I don't see a frame as a preprocessor but as a term rewriting system. I want to be able to

* build grammar node instances by reflection
* transform grammar nodes instances on grammar nodes instances, and
* expand the code represented by a grammar node instance by reification.

The operator [[[]]] taking as parameter some code results in a grammar node.
Frames transform grammar nodes on grammar nodes.
The operator [[[]]] taking as parameter a grammar node results in the associated code.
Which syntax is more appropriated to support these requirements needs refinement.

A grammar node instance is represented as a type on the std::meta namespace.
The compiler should provide some intrinsic meta-functions to obtain grammar node instances from C++ types.

My approach doesn't takes the augment constexpr / template metaprogramming route, and maybe the route I'm taking is a dead-end.

As I see frames, they are orthogonal to templates and constexpr. The examples I gave are template frames, but we could have frames that are not templates. A frame would be implicitly a constexpr as if the parameters are know at compile time, the result is obtained at compile time. If wewere  able to build grammar nodes at run-time, the frames could be evaluated at run-time.

While at a first glance it could seem that frames are nothing more than macros, their parameters are language grammar nodes. This makes a big difference and IMHO it opens to more elaborated transformation techniques.

I would not say that it is a dead - end, but I think it needs more work when it comes to clarifying semantics and where operations should take place in relation i.e. to template definition, declaration and instantiation.

Agreed.

But you are not at the same problem domain as the preprocessor right now, so you can't substitute it as you wish. Frames (as you are proposing them at least) are not exactly analogous to AST term rewriting macros (see Nimrod's http://nimrod-code.org/) either.

This is quite close to what I'm looking for,even if the syntax is completely different and IIUC it is limited to expressions and statements. I would to be able to define any transformation inside the language.
Thanks for the pointer, if you have other like this one please send me them.

Because of the (for the time being!) ambiguous semantics you propose, they clash with established systems as templates and constexpr. Think again about why I asked as to when and where frame operations should take place. This is more important than it seems.

I understand your concern better now and unfortunately I have not taken enough time to be able do a deeper analysis to give you some answers :(

 

As for people who rely on external tools for code generation, I think they are simply trying to avoid current preprocessor metaprogramming techniques for some subjective disadvantage. Examples like boost's own use demonstrates otherwise.

I have nothing against external tools, but I think this is off topic on this ML as far as the generation is external.

I did not understand your reply here.

What I meant is that I didn't wanted to discuss anything related to external tools.

As I said, the preprocessor is fairly adequate if used properly; however it does require some inner knowledge of the techniques involved when doing metaprogramming with it and awareness of the necessary compromises in designing solutions with it. People not particularly eager to gain that knowledge, turn to external tools. Would I like to have something better? Of course I would, but if the preprocessor is all I am given within C++, then I must be ready to make the best use of it. And I have no problems with that either.

Neither me.

 

I have used the preprocessor meta-programming techniques in TBoost.Enums and  TBoost.Opaque and while this allows to experiment I would like to see something more appropriated introducing reflection in the language.

Ville has suggested in [1] an approach that is more in line with the current language. operator member_name when member_name is a compile time string would define a new function. meta::invoke_member allows to call to a function identified by a compile time string.

I am aware of Ville Voutilainen's approaches to problems like this and I am quite a fan of his solutions when it comes to them. But we are going to reflection now, which would derail the entire thread into something different. Systematic generation of valid C++ constructs for reducing hand - written constructs is not something closely related to reflection but as a "workaround" for the lack of said feature at the language level. It is obviously that in the end, you are actually trying to solve a compile / run - time barrier problem, hence your interest in reflection. Try thinking of the "frames" you are proposing as something that should not however just exist to help reflection purposes.

I'm  just trying to explore the use cases as described in N3814 Call for Compile-Time Reflection Proposals and
N3492 - Use Cases for Compile-Time Reflection (Rev. 2).

I'll stop the frame concept thread here until I have something more concrete. Thanks for showing me that ideas can be good, but concrete proposal are much better.

Apologies if I made you lost your time,
Vicente

[David Krauss]

On 11/13/13 7:59 AM, Vicente J. Botet Escriba wrote:
> The single way to control expansions I have identified so far and
> proposed in this post is the operator [[[]]]...

Somewhat off-topic, but if multiple brackets are to be used as
punctuators, it might be a good idea to make them such as part of the
token syntax. As with attributes, the intent isn't really to allow
whitespace between the constituent ['s.

[Michał Dominiak]

Vicente,

Have you made any progress towards completing the design of the idea you have presented in this thread?

[Vicente J. Botet Escriba]

Le 14/06/14 18:35, Michał Dominiak a écrit :

> Vicente,
>
> Have you made any progress towards completing the design of the idea
> you have presented in this thread?
>

Unfortunately no. I have not had (taken :() time for this.

Vicente