"Reflection Lite" proposal

[Niall Douglas]

Dear SG7,

Now this reflector has opened, it's time for me to do the thing I always do and ask the stupid question: is adding reflection to the language a bad idea compared to pushing the problem onto libraries and letting them evolve a superior solution which we can later standardise? I did a similar thing on SG5 Transactional Memory by arguing in favour of dropping language support in favour of exporting just enough for library support (the thread starts at https://groups.google.com/a/isocpp.org/d/msg/tm/zfRyY_XfxSU/47esUzUcHX0J) - I didn't get anywhere, but I still think the question is very useful to ask.

I should also caveat that before I begin, I pitched this idea to Chandler at C++ Now for inclusion into clang, and he was definitely not warm to the idea.

My suggestion for implementing compile-time reflection is very simple, and is the absolute minimum possible: the compiler makes available a magic namespace, let's call it namespace __compiler_internals. In this namespace you get a set of pseudo template definitions which happen to make available the present parsing state of the compiler in this compiland. As I originally pitched this to Chandler regarding clang, I'll stick to a libclang example, but I should emphasise that by this I mean that each compiler makes available its internal state in whatever format it so chooses. It is then up to one or more third party libraries - probably at least one in Boost - to convert compiler-specific namespace __compiler_internals into a unified template API in order to achieve cross-compatibility. I entirely expect such thunk libraries to get repeatedly thrown away and refactored until something emerges which is pretty good for most use cases.

Ok, so here's an example based on clang where one might export libclang's API as namespace __compiler_internals { namespace libclang { ... } }. Here is a mock up of using this facility to enumerate a type's member variables into a Boost MPL typelist:

// Type to be found in the AST and whose members are to be enumerated

struct Foo

{

  int a;

  double b;

  Foo *c;

};

...

// Adds each node to the output type list

template<class node, class parent, class output> struct add_children

{

  typedef typename boost::mpl::push_back<output, typename node::spelling>::type type;

  static constexpr int return_value=2; // continue searching recursively

};

template<class node, class parent, class output, bool is_reference, class type> struct add_if_Foo

{

  // Pass through output for non-matching nodes

  typedef output value;

  static constexpr int return_value=2; // continue searching recursively

};

template<class node, class parent, class output> struct add_if_Foo<node, parent, output, false, Foo>

{

  // Add node's children to output typelist

  typedef typename __compiler_internals::libclang::cindex::Cursor_visit_callback<add_children, output>::type type;

  // Lack of return_value means it defaults to zero, i.e. stop searching

};

template<class node, class parent, class output> struct visit_filter

{

  // Break out subproperties of node being visited

  typedef typename add_if_Foo<node, parent, output, node::kind::is_reference, typename node::spelling>::type type;

};

// Outputs a type list of member variables of Foo

typedef __compiler_internals::libclang::cindex::Cursor_visit_callback<visit_filter, boost::mpl::vector<void>>::type output_type_list;

I do stress that this is a mock up, and therefore is not tested for correctness. Note though how the libclang API is exposed verbatim in the __compiler_internals::libclang namespace, it ought to be very straightforward for clang to expose this as a new set of libclang API bindings, albeit en vivo to the compiland being compiled.

I also see no reason why this API cannot be used to load and parse external translation units i.e. your template metaprogramming can inspect other source files.

And finally, I only depicted libclang as a first stage example. libclang is of course AST read only. Down the line (not now), one could allow template metaprogramming to rewrite the AST en vivo, so if - for example - some third party library likes to pollute the global namespace, you could add a metaprogrammed transformation which rewrites the AST to unpollute the global namespace. Such a facility, I would imagine, would also open lots of fun security holes :), but it would enable C++ to reprogram itself with huge power and flexibility.

Anyway, back to you SG7 for consideration. I know I've simply defined the reflection problem into something else, but hey the above ought to work, and it's real easy to do for clang at least as a minimum. The question becomes: is the above solution acceptable for SG7 to recommend for C++17? The trivial answer is probably not, so let me ask another question: if clang implemented the above soon, I can guarantee you that it would be used by Boost libraries within six months and pressure would be brought to bear on the other compiler vendors to do something similar. It is then entirely possible that we'd have a reasonable reflection library in Boost by 2015.

Niall

[Ville Voutilainen]

On 22 October 2013 21:49, Niall Douglas <nialldo...@gmail.com> wrote:

Dear SG7,

Now this reflector has opened, it's time for me to do the thing I always do and ask the stupid question: is adding reflection to the language a bad idea compared to pushing the problem onto libraries and letting them evolve a superior solution which we can later standardise? I did a similar thing on SG5 Transactional Memory by arguing in favour of dropping language support in favour of exporting just enough for library support (the thread starts at https://groups.google.com/a/isocpp.org/d/msg/tm/zfRyY_XfxSU/47esUzUcHX0J) - I didn't get anywhere, but I still think the question is very useful to ask.

Exposing a minimal set of compiler-supported functionality and writing the rest as libraries
is in general always worth careful consideration.
 

I should also caveat that before I begin, I pitched this idea to Chandler at C++ Now for inclusion into clang, and he was definitely not warm to the idea.

My suggestion for implementing compile-time reflection is very simple, and is the absolute minimum possible: the compiler makes available a magic namespace, let's call it namespace __compiler_internals. In this namespace you get a set of pseudo template definitions which happen to make available the present parsing state of the compiler in this compiland. As I originally pitched this

I'm not a compiler vendor, nor a compiler expert. I have done some minor feature implementations
for the c++ front-end of gcc. Now, with that disclaimer out of the way, I predict implementation
vendors will absolutely balk at this idea. A couple of random remarks:

1) _parsing state_? That's not going to help much, unless you think that "parsing state" somehow

includes instantiated templates, and understands the results of non-dependent lookups, and

understands scoping in general.

2) parsing state until what? The occurrence of the use of such a pseudo template? Implementations

have certain liberties with the points of instantiation of templates in a translation unit.

If you ask me, this duck doesn't fly - it's dead.

 

to Chandler regarding clang, I'll stick to a libclang example, but I should emphasise that by this I mean that each compiler makes available its internal state in whatever format it so chooses. It is then up to one or more third party libraries - probably at least one in Boost - to convert compiler-specific namespace __compiler_internals into a unified template API in order to achieve cross-compatibility. I entirely expect such thunk libraries to get repeatedly thrown away and refactored until something emerges which is pretty good for most use cases.

Why would we standardize the way to get implementation-specific internals and wait until
an additional library will wrap it into a reasonable portable API? Why wouldn't we just standardize
the portable API?

 

And finally, I only depicted libclang as a first stage example. libclang is of course AST read only. Down the line (not now), one could allow template metaprogramming to rewrite the AST en vivo, so if - for

Rewriting code via reflection is certainly in scope for SG7, but probably not a first-step target.
 

Anyway, back to you SG7 for consideration. I know I've simply defined the reflection problem into something else, but hey the above ought to work, and it's real easy to do for clang at least as a minimum. The question becomes: is the above solution acceptable for SG7 to recommend for C++17? The trivial answer is probably not, so let me ask another question: if clang implemented the above soon, I can guarantee you that it would be used by Boost libraries within six months and pressure would be brought to bear on the other compiler vendors to do something similar. It is then entirely possible that we'd have a reasonable reflection library in Boost by 2015.

I would love to know what makes you think you can guarantee that. With the understanding

that I have about boost, I can almost guarantee the opposite.

[Niall Douglas]

My suggestion for implementing compile-time reflection is very simple, and is the absolute minimum possible: the compiler makes available a magic namespace, let's call it namespace __compiler_internals. In this namespace you get a set of pseudo template definitions which happen to make available the present parsing state of the compiler in this compiland. As I originally pitched this

I'm not a compiler vendor, nor a compiler expert. I have done some minor feature implementations
for the c++ front-end of gcc. Now, with that disclaimer out of the way, I predict implementation
vendors will absolutely balk at this idea. A couple of random remarks:

1) _parsing state_? That's not going to help much, unless you think that "parsing state" somehow

includes instantiated templates, and understands the results of non-dependent lookups, and

understands scoping in general.

2) parsing state until what? The occurrence of the use of such a pseudo template? Implementations

have certain liberties with the points of instantiation of templates in a translation unit.

If you ask me, this duck doesn't fly - it's dead.

Maybe. I am probably the most familiar with clang out of any of the compiler's internals, but that's like 0.1 on the familiarity out of 10 scale.

I think it's very doable if you don't mind treacle like performance loss every time someone touches the magic namespace - after all, one could dump out a temporary copy of whatever you currently have of the present compiland state and fire it into libclang for parsing. I also explicitly accept that misoperation, incompleteness and other compiler-specific weirdnesses are to be expected with this proposal - which is exactly why a third party library encodes workarounds etc. for such depending on compiler revision.

Which may sound brittle and expensive. But is this more brittle and expensive than the compiler vendor implementing machinery for reflection, especially when such implementation may force behaviours and designs unhelpful to the vendor? For example, if clang and GCC which are AST based come up with one Reflection solution, MSVC which is not AST based (or so I am told) may simply not implement it. That makes standardisation here rather difficult, because standardisation isn't hugely useful when a major vendor doesn't implement it (C99 I'm talking to you).

These are questions, not answers. I don't know the answers to doing Reflection well, let alone usefully. I'm explicitly proposing it's too hard to design now without more experience.

  

to Chandler regarding clang, I'll stick to a libclang example, but I should emphasise that by this I mean that each compiler makes available its internal state in whatever format it so chooses. It is then up to one or more third party libraries - probably at least one in Boost - to convert compiler-specific namespace __compiler_internals into a unified template API in order to achieve cross-compatibility. I entirely expect such thunk libraries to get repeatedly thrown away and refactored until something emerges which is pretty good for most use cases.

Why would we standardize the way to get implementation-specific internals and wait until
an additional library will wrap it into a reasonable portable API? Why wouldn't we just standardize
the portable API?

I am proposing that we standardise acceptance that this is too hard to get right through a design by committee, and we instead push the problem onto library evolution to figure out. I proposed the same for TM, but they admittedly have more than a decade of implementation experience behind them (I still think TM as currently proposed with C++ is currently a poor fit. Too many corner cases in my opinion).

   

Anyway, back to you SG7 for consideration. I know I've simply defined the reflection problem into something else, but hey the above ought to work, and it's real easy to do for clang at least as a minimum. The question becomes: is the above solution acceptable for SG7 to recommend for C++17? The trivial answer is probably not, so let me ask another question: if clang implemented the above soon, I can guarantee you that it would be used by Boost libraries within six months and pressure would be brought to bear on the other compiler vendors to do something similar. It is then entirely possible that we'd have a reasonable reflection library in Boost by 2015.

I would love to know what makes you think you can guarantee that. With the understanding

that I have about boost, I can almost guarantee the opposite. 

Boost.Mirror, Boost.Puddle et al are good examples of how non-emulated reflection would be a huge help in a huge range of scenarios. Boost.Python could be vastly simplified with reflection. The original reflection proposals to WG21 originated from the never ending interop hassles C++ forces on anyone wanting C++ code to work with any other code (including C++ using a different STL).

I just realised there is a potential for some confusion here where I guaranteed that Boost libraries would use such a facility - I should have said "Boost using" or "Boost like" libraries which is what I actually meant. My former wording would quite correctly be interpreted as those having passed peer review, which I didn't mean. Sorry.

Niall

[Ville Voutilainen]

On 22 October 2013 22:43, Niall Douglas <nialldo...@gmail.com> wrote:

I'm not a compiler vendor, nor a compiler expert. I have done some minor feature implementations
for the c++ front-end of gcc. Now, with that disclaimer out of the way, I predict implementation
vendors will absolutely balk at this idea. A couple of random remarks:

1) _parsing state_? That's not going to help much, unless you think that "parsing state" somehow

includes instantiated templates, and understands the results of non-dependent lookups, and

understands scoping in general.

2) parsing state until what? The occurrence of the use of such a pseudo template? Implementations

have certain liberties with the points of instantiation of templates in a translation unit.

If you ask me, this duck doesn't fly - it's dead.

Maybe. I am probably the most familiar with clang out of any of the compiler's internals, but that's like 0.1 on the familiarity out of 10 scale.

I think it's very doable if you don't mind treacle like performance loss every time someone touches the

You're welcome to try it. I don't believe it will work, for the reasons I stated above, and likely

more reasons that implementation vendors will likely point out if need be.

Which may sound brittle and expensive. But is this more brittle and expensive than the compiler vendor implementing machinery for reflection, especially when such implementation may force

You're asking for a wild guess. Mine is YES, it's more brittle and more expensive.

 

behaviours and designs unhelpful to the vendor? For example, if clang and GCC which are AST based come up with one Reflection solution, MSVC which is not AST based (or so I am told) may simply not implement it. That makes standardisation here rather difficult, because standardisation isn't hugely useful when a major vendor doesn't implement it (C99 I'm talking to you).

Thus far I haven't seen a reflection proposal that necessarily requires AST. 

These are questions, not answers. I don't know the answers to doing Reflection well, let alone usefully. I'm explicitly proposing it's too hard to design now without more experience.

One of the purposes of this study group is to gather such experience, and talk to the implementation
vendors about what is and is not feasible.

 

Why would we standardize the way to get implementation-specific internals and wait until
an additional library will wrap it into a reasonable portable API? Why wouldn't we just standardize
the portable API?

I am proposing that we standardise acceptance that this is too hard to get right through a design by committee, and we instead push the problem onto library evolution to figure out. I proposed the same

And library evolution will do what? Accept that it's too hard to get right through a "design by
committee" (which is, btw, not necessarily what we're doing here) and give up, like we would do?

 

Boost.Mirror, Boost.Puddle et al are good examples of how non-emulated reflection would be a huge help in a huge range of scenarios. Boost.Python could be vastly simplified with reflection. The original

Oh, I have no trouble believing that. There are many more libraries that would benefit from standardized
reflection facilities, that's not a point I would contest.

[Andrew Tomazos]

On Tuesday, October 22, 2013 8:49:52 PM UTC+2, Niall Douglas wrote:

Dear SG7,

Now this reflector has opened, it's time for me to do the thing I always do and ask the stupid question: is adding reflection to the language a bad idea compared to pushing the problem onto libraries and letting them evolve a superior solution which we can later standardise? I did a similar thing on SG5 Transactional Memory by arguing in favour of dropping language support in favour of exporting just enough for library support (the thread starts at https://groups.google.com/a/isocpp.org/d/msg/tm/zfRyY_XfxSU/47esUzUcHX0J) - I didn't get anywhere, but I still think the question is very useful to ask.

Have you seen N3815?  http://isocpp.org/blog/2013/10/n3815

It's how we are proposing solving introspection of enumeration types.  We arrived at the approach you mention.  Standardize a complete but minimal low-level direct interface.  This allows library authors to build higher-level facilities without having to modify the compiler.

There is a link to my co-authors reflection extensions for clang at the bottom of the proposal.  It contains more than just the three queries proposed for standardization.

[Niall Douglas]

I think it's very doable if you don't mind treacle like performance loss every time someone touches the

You're welcome to try it. I don't believe it will work, for the reasons I stated above, and likely

more reasons that implementation vendors will likely point out if need be.

Sure, we'll see if a compiler implementation vendor chips in. They could probably just say if it's viable or not off the top of their head. If it's not viable, and I assume Chandler thought it not when I pitched it to him, then it was worth thinking about and end of discussion.

 

Which may sound brittle and expensive. But is this more brittle and expensive than the compiler vendor implementing machinery for reflection, especially when such implementation may force

You're asking for a wild guess. Mine is YES, it's more brittle and more expensive.

Maybe. I think clang and gcc are very different beasts to msvc. I also think that reflection is far, far harder than most realise. I think you need to look into exported templates and ODR violation resolution handling to get a sense of just how hard reflection actually is when pushed to its end. All this feeds into an eventual C++ Modules implementation of course, but Reflection eventually leads there too, and early miscalculations during design can easily lead to stuff having to be deprecated later.

My proposal - if viable - has the huge advantage of letting third party libraries evolve the right way of handling reflection extending to its fullest possible extent. It also includes refactoring designs to cope with C++ Modules being implemented, and so on. If I ever get my Component Objects for C++ implementation off the ground, then Reflection will need changing there again too.

Before anyone chips in to say this SG is only looking at compile-time reflection for now, then sure, the previous two paragraphs are off-topic. Note though that my proposal - if it allows the loading of external translation units - lets compile time metaprogramming generate API bindings to remote code at compile-time. That is an enormous win for Component Objects for C++ for example, because it saves me or someone having to write a custom link stage tool to grok all the compilands and generate bind shims for different compilers, STLs, C only code etc when then get stored into a graph database from which the new C++ component object dynamic loader generates object load graphs. Instead of me writing a large program full of complexity, I can push all that implementation into the compiler itself, which saves me or someone many months of work.

Hence my very deep interest in this SG :)

 

I am proposing that we standardise acceptance that this is too hard to get right through a design by committee, and we instead push the problem onto library evolution to figure out. I proposed the same

And library evolution will do what? Accept that it's too hard to get right through a "design by
committee" (which is, btw, not necessarily what we're doing here) and give up, like we would do?

My concern is that the committee has a lot of language experts who naturally tend towards a mostly language based solution for some feature. Even when a mostly library based solution makes a lot more sense, especially as mostly language based solutions are very hard to interop with other languages and runtimes, while mostly library based solutions tend to be far more tractable (not always, but tend to).

I should absolutely stress that language shortcuts for library features to save typing are not what I mean here. No one would readily give up on range for loops for example. What I mean is the divergence between providing a single way of telling the compiler to do something (cathedral) versus opening up the innards for manipulation into unknown outcomes (bazaar).

Niall

[Ville Voutilainen]

On 23 October 2013 00:00, Niall Douglas <nialldo...@gmail.com> wrote:

 

Which may sound brittle and expensive. But is this more brittle and expensive than the compiler vendor implementing machinery for reflection, especially when such implementation may force

You're asking for a wild guess. Mine is YES, it's more brittle and more expensive.

Maybe. I think clang and gcc are very different beasts to msvc. I also think that reflection is far, far

Perhaps so, but we don't yet know whether that will matter.
 

harder than most realise. I think you need to look into exported templates and ODR violation resolution handling to get a sense of just how hard reflection actually is when pushed to its end. All this feeds into an eventual C++ Modules implementation of course, but Reflection eventually leads there too, and early miscalculations during design can easily lead to stuff having to be deprecated later.

That's why we have a study group, we're not putting a complete reflection solution into a standard

before, well, studying the problem domain. :)
 

My proposal - if viable - has the huge advantage of letting third party libraries evolve the right way of handling reflection extending to its fullest possible extent. It also includes refactoring designs to cope

I'd like to dispute that "the right way" there.
 

Before anyone chips in to say this SG is only looking at compile-time reflection for now, then sure, the previous two paragraphs are off-topic. Note though that my proposal - if it allows the loading of external translation units - lets compile time metaprogramming generate API bindings to remote code at compile-time. That is an enormous win for Component Objects for C++ for example, because it

I think I can do the same thing with a standard compile-time reflection API, so I don't think that's

a pro that only your proposal has.

committee" (which is, btw, not necessarily what we're doing here) and give up, like we would do?

My concern is that the committee has a lot of language experts who naturally tend towards a mostly language based solution for some feature. Even when a mostly library based solution makes a lot

Fear not, we have enough people who are naturally averse to language extensions, especially
for cases where a library approach will do.
 

[Niall Douglas]

Have you seen N3815?  http://isocpp.org/blog/2013/10/n3815

It's how we are proposing solving introspection of enumeration types.  We arrived at the approach you mention.  Standardize a complete but minimal low-level direct interface.  This allows library authors to build higher-level facilities without having to modify the compiler.

There is a link to my co-authors reflection extensions for clang at the bottom of the proposal.  It contains more than just the three queries proposed for standardization.

I had seen it, but to be honest I had instantly filtered it out because it seemed to me to apply to enum's only (which didn't seem hugely useful to my needs).

Now you've helped me look at it more deeply, I personally speaking don't think your interface is low level enough :) I would personally prefer an interface so low level it is completely compiler version dependent i.e. it can, and does, change arbitrarily.

Note I have absolutely no objection to a low level interface like the one you propose then being abstracted on top. Rather, my point is I think we should have access to the raw implementation details where available.

Of course, if this isn't viable according to compiler vendor opinion, I happily default to your proposal of a low level API. And BTW kudos to you guys for going ahead with a clang fork. If only I had the time (or a job for that matter)!

Niall

[Andrew Tomazos]

On Tuesday, October 22, 2013 11:10:36 PM UTC+2, Niall Douglas wrote:

Have you seen N3815?  http://isocpp.org/blog/2013/10/n3815

It's how we are proposing solving introspection of enumeration types.  We arrived at the approach you mention.  Standardize a complete but minimal low-level direct interface.  This allows library authors to build higher-level facilities without having to modify the compiler.

There is a link to my co-authors reflection extensions for clang at the bottom of the proposal.  It contains more than just the three queries proposed for standardization.

I had seen it, but to be honest I had instantly filtered it out because it seemed to me to apply to enum's only (which didn't seem hugely useful to my needs).

Now you've helped me look at it more deeply, I personally speaking don't think your interface is low level enough :) I would personally prefer an interface so low level it is completely compiler version dependent i.e. it can, and does, change arbitrarily.

If we isolate our discussion to enumeration types for the moment, it's hard for me to visualize exactly what you mean by a lower-level interface.  The three property queries give pretty much direct access to the internal enum-specifier AST.  What information do you need about the enum-specifier AST that you can't obtain from our proposed queries, but could obtain from an even lower-level interface?

[Niall Douglas]

Now you've helped me look at it more deeply, I personally speaking don't think your interface is low level enough :) I would personally prefer an interface so low level it is completely compiler version dependent i.e. it can, and does, change arbitrarily.

If we isolate our discussion to enumeration types for the moment, it's hard for me to visualize exactly what you mean by a lower-level interface.  The three property queries give pretty much direct access to the internal enum-specifier AST.  What information do you need about the enum-specifier AST that you can't obtain from our proposed queries, but could obtain from an even lower-level interface?

I think the biggest low level thing missing from your intrinsic API is access to the compiler's node bind reference machinery, so for example:

class enum Foo : size_t { a=1, b=8 };

Here enum type Foo gets storage of size_t with contents of 'a' bound to constant size_t=1 and 'b' bound to constant size_t=2. Internally, a naive compiler might keep a shared reference to static instances of size_t=1 and size_t=2 which are shared with anything else which also uses a constant bind to those integral constants of that type. Or, indeed, it might recognise that storing such a shared reference costs more than simply storing 1 or 8 and hence store those directly, but for bound references to more complex integral values it might indeed use a shared reference.

Why would you need such detail? It's not hugely useful for reflecting enums alone absolutely. But it does start to become very useful when you're reflecting stuff which uses enums. Let me give a hypothetical example: let us assume some future C++ compiler allows a class enum to have any trivial type, including struct instances, so now one could do:

struct FooType { int a, b, c; };

class enum Foo : FooType { a={1, 2, 3}, b={3, 2, 1} };

Your API can cope with this just fine. But if I am reflecting some function call which passes this class enum as a parameter, and I can see that Foo::b is being passed as a variable, it would be super-useful if I had access to the the compiler's node reference machinery so I can use a nice void * unique value instead of having to push all comparisons through the compiler's partial template specialisation engine.

Let me state this another way: I am peer review managing proposed Boost.TypeIndex right now. That library lets you avoid RTTI in many use cases by converting the compiler's type into a cross-compiland opaque comparable for order and identity, and therefore enables guaranteed execution complexities as well as turning on much more Boost use on embedded systems where RTTI is disabled. Boost.TypeIndex is effectively a workaround for the fact we don't have access to the compiler's AST node reference machinery, so TypeIndex metaprograms the compiler to simulate one.

In other words, having access to opaque type unique identifiers which work across compilands and can be stored to disc etc is really useful. And I'd love to see that in any Reflection proposal, because it really helps my Component Objects for C++ in particular as nice void * unique identifiers are easily hashable.

Niall

[christia...@googlemail.com]

As the second author of N3815, I am not sure I can follow your explanation. All this talk about a hypothetical enumeration standard/feature set, AST node binding/referencing and void pointer comparisons does not give me an idea how your preferred interface would look like, and to the best of my knowledge all of this is not very relevant to any actual enum implementation in current C++ compilers. This seems like highly dubious "future proofing" at best.

To me, your previously expressed wish to *standardize* a "completely compiler version dependent" e.g. arbitrary(?) interface seems rather paradoxical. You could lobby the individual compiler maintainers for such an interface, but I don't see any role for ISO or SG7 in that effort.

And I'm quite sceptical that this in any way would get us closer to standardized C++ reflection. Anyone serious about creating a reflection interface can simply implement your proposed extensions themselves in a Clang or GCC branch in exactly the form they need for their higher level library API.

But finding good, intuitive interface mechanisms for the typical use cases seems like the biggest problem to me. I am not deeply familiar with the Mirror library, but its heavy reliance on boost::MPL for most non-trivial uses might not be ideal answer for a standardized solution.
A few meta-programming improvements (like Concepts Lite and some proposed support features like better parameter packs etc) to me seem like very important requirements for really good interfaces. I'm afraid (existing) TMP library capabilities are just not good enough for more advanced reflection areas (but enumerations should be doable, as we propose).

[Niall Douglas]

As the second author of N3815, I am not sure I can follow your explanation. All this talk about a hypothetical enumeration standard/feature set, AST node binding/referencing and void pointer comparisons does not give me an idea how your preferred interface would look like, and to the best of my knowledge all of this is not very relevant to any actual enum implementation in current C++ compilers. This seems like highly dubious "future proofing" at best.

My original mock up enumerated members of a struct type, and it gave me access to node reference machinery and AST cursors which I feel is desirable. The future proofing is not the point, rather it's where you decide to put the future proofing: into new language extensions, or into third party abstraction libraries. I was proposing the latter despite that this SG appears to be biased towards the former.

 

To me, your previously expressed wish to *standardize* a "completely compiler version dependent" e.g. arbitrary(?) interface seems rather paradoxical. You could lobby the individual compiler maintainers for such an interface, but I don't see any role for ISO or SG7 in that effort.

Not at all. There are plenty of areas in C++ where the standard explicitly standardises intentionally no standard. A perfectly valid outcome from SG7 deliberations is to recommend to the committee to do nothing, or more likely to do some particularly useful variant of doing nothing.

 

And I'm quite sceptical that this in any way would get us closer to standardized C++ reflection. Anyone serious about creating a reflection interface can simply implement your proposed extensions themselves in a Clang or GCC branch in exactly the form they need for their higher level library API.

Exactly what I'm proposing this SG recommends to the committee as what the C++17 standard should state. Note I refer to the 2017 or 1x standard only here - I would expect a later standard to incorporate what we have learned.

 

But finding good, intuitive interface mechanisms for the typical use cases seems like the biggest problem to me. I am not deeply familiar with the Mirror library, but its heavy reliance on boost::MPL for most non-trivial uses might not be ideal answer for a standardized solution.

Mirror's API isn't bad, it's more that it needs a post compilation step to add in the reflection metadata which is annoying. And actually the MPL is probably one of the most core tools anyone using Boost uses - I don't think I've written much code in the past five years which didn't use the MPL. Once Boost has figured out how/whether to refactor MPL for C++11, I'm very sure MPL will enter the standard as it's incredibly useful.

 

A few meta-programming improvements (like Concepts Lite and some proposed support features like better parameter packs etc) to me seem like very important requirements for really good interfaces. I'm afraid (existing) TMP library capabilities are just not good enough for more advanced reflection areas (but enumerations should be doable, as we propose).

On this we agree. I'm also asserting that we simply don't have sufficient experience to design good language extensions right now, and we ought to demur to a library evolution approach.

Note to anyone concerned: I'll be interviewing all day with Microsoft tomorrow, then flying back to Canada the following day. Expect no replies till the weekend.

Niall 

[Ville Voutilainen]

On 23 October 2013 05:00, Niall Douglas <nialldo...@gmail.com> wrote:

As the second author of N3815, I am not sure I can follow your explanation. All this talk about a hypothetical enumeration standard/feature set, AST node binding/referencing and void pointer comparisons does not give me an idea how your preferred interface would look like, and to the best of my knowledge all of this is not very relevant to any actual enum implementation in current C++ compilers. This seems like highly dubious "future proofing" at best.

My original mock up enumerated members of a struct type, and it gave me access to node reference machinery and AST cursors which I feel is desirable. The future proofing is not the point, rather it's

I don't see why this "node reference machinery" is necessary for any of the facilities to
mentioned in your later post.
 

where you decide to put the future proofing: into new language extensions, or into third party abstraction libraries. I was proposing the latter despite that this SG appears to be biased towards the former.

That appearance is incorrect, we're not talking about just language extensions here, but

were certainly not striving for just third party abstraction libraries either.
 

 

To me, your previously expressed wish to *standardize* a "completely compiler version dependent" e.g. arbitrary(?) interface seems rather paradoxical. You could lobby the individual compiler maintainers for such an interface, but I don't see any role for ISO or SG7 in that effort.

Not at all. There are plenty of areas in C++ where the standard explicitly standardises intentionally no standard. A perfectly valid outcome from SG7 deliberations is to recommend to the committee to do nothing, or more likely to do some particularly useful variant of doing nothing.

What you have proposed is not a variant of doing nothing. It's worse than doing nothing.

 

But finding good, intuitive interface mechanisms for the typical use cases seems like the biggest problem to me. I am not deeply familiar with the Mirror library, but its heavy reliance on boost::MPL for most non-trivial uses might not be ideal answer for a standardized solution.

Mirror's API isn't bad, it's more that it needs a post compilation step to add in the reflection metadata which is annoying. And actually the MPL is probably one of the most core tools anyone using Boost uses - I don't think I've written much code in the past five years which didn't use the MPL. Once Boost has figured out how/whether to refactor MPL for C++11, I'm very sure MPL will enter the standard as it's incredibly useful.

I kinda doubt that MPL will enter the standard just because it's incredibly useful for some people.
 

 

A few meta-programming improvements (like Concepts Lite and some proposed support features like better parameter packs etc) to me seem like very important requirements for really good interfaces. I'm afraid (existing) TMP library capabilities are just not good enough for more advanced reflection areas (but enumerations should be doable, as we propose).

On this we agree. I'm also asserting that we simply don't have sufficient experience to design good language extensions right now, and we ought to demur to a library evolution approach.

Based on that assertion we don't have sufficient experience to decide that this "library

evolution" approach is any better. Not that the assertion is valid anyway - we have plenty
of experience from google protobuf, boost fusion, Qt, and other existing attempts at
reflection to know what we want to achieve, and a good part of how.

What we ought to do is to resist the proposal that we make a blanket statement that
an implementation is going to expose some "node reference machinery" without stating

how. If it's completely implementation-specific, a valid exposure is to expose absolutely
nothing. If it's not implementation-specific, we're already well on the way to standardize
an actual reflection API. The implementation-specific approach will accomplish absolutely

nothing and will only hinder the writing of portable code using reflection.

[Niall Douglas]

To me, your previously expressed wish to *standardize* a "completely compiler version dependent" e.g. arbitrary(?) interface seems rather paradoxical. You could lobby the individual compiler maintainers for such an interface, but I don't see any role for ISO or SG7 in that effort.

Not at all. There are plenty of areas in C++ where the standard explicitly standardises intentionally no standard. A perfectly valid outcome from SG7 deliberations is to recommend to the committee to do nothing, or more likely to do some particularly useful variant of doing nothing.

What you have proposed is not a variant of doing nothing. It's worse than doing nothing.

In your opinion Ville. I would view my proposal as an entirely valid proposal for C++1x as an information and good practice gathering exercise. I agree that long term it isn't ideal, but that's the point of it being an information and good practice gathering exercise. I never claimed - despite your best attempts to paint me so - that this proposal is for all future C++ standards. No, it's an interim measure for the near future, and I have been clear about that since the beginning.

  

On this we agree. I'm also asserting that we simply don't have sufficient experience to design good language extensions right now, and we ought to demur to a library evolution approach.

Based on that assertion we don't have sufficient experience to decide that this "library

evolution" approach is any better. Not that the assertion is valid anyway - we have plenty
of experience from google protobuf, boost fusion, Qt, and other existing attempts at
reflection to know what we want to achieve, and a good part of how.

All of which have filled me with considerable caution, because all of those proposals do not scale well, especially in the presence of C++ Modules and ODR violation resolution strategies. I freely admit I don't know which design would be good, but I am very certain that all these designs you just mentioned are deeply flawed when used at scale.

 

If it's completely implementation-specific, a valid exposure is to expose absolutely
nothing.

Correct. Another valid exposure is to supply a copy of libclang with every compiler, such that every compiler gets libclang bindings mapped into its compile-time space. Reflected AST parsing now becomes completely external to the current compiland. This, I would imagine, would be the easiest path for GCC and MSVC, perhaps even clang itself.

 

If it's not implementation-specific, we're already well on the way to standardize
an actual reflection API. The implementation-specific approach will accomplish absolutely

nothing and will only hinder the writing of portable code using reflection.

By defining Reflection to be compile-time only for now, we have defined away much of the difficulty in getting a good Reflection design right. Equally that means the present Reflection support will only EVER be for compile-time use only.

Which might actually be fine for C++-1y - this SG proposes some support and it is explicitly aimed for compile-time use and never anything more. In which case, my proposal still stands as a valid proposal for gaining information and good practice gathering for a Reflection implementation which does support dynamic Reflection after C++-1y.

It is therefore STILL a worthwhile item for this SG to consider, especially as a library based design can be evolved far faster than an ISO standard.

I assume that this thread is now pretty much spent unless someone forks clang with my proposed new feature set. I hope some keen doctoral student is reading, which is why I started this thread!

Niall

[Ville Voutilainen]

On 28 October 2013 17:34, Niall Douglas <nialldo...@gmail.com> wrote:

What you have proposed is not a variant of doing nothing. It's worse than doing nothing.

In your opinion Ville. I would view my proposal as an entirely valid proposal for C++1x as an

Of course, in my opinion.
 

information and good practice gathering exercise. I agree that long term it isn't ideal, but that's the point of it being an information and good practice gathering exercise. I never claimed - despite your best attempts to paint me so - that this proposal is for all future C++ standards. No, it's an interim measure for the near future, and I have been clear about that since the beginning.

I'm not trying to paint you anything. I'm saying that this idea is a step in the wrong direction,
and having it as an interim measure is not very useful.

 

  

On this we agree. I'm also asserting that we simply don't have sufficient experience to design good language extensions right now, and we ought to demur to a library evolution approach.

Based on that assertion we don't have sufficient experience to decide that this "library

evolution" approach is any better. Not that the assertion is valid anyway - we have plenty
of experience from google protobuf, boost fusion, Qt, and other existing attempts at
reflection to know what we want to achieve, and a good part of how.

All of which have filled me with considerable caution, because all of those proposals do not scale well, especially in the presence of C++ Modules and ODR violation resolution strategies. I freely admit I don't know which design would be good, but I am very certain that all these designs you just mentioned are deeply flawed when used at scale.

I have no reason to believe that an implementation-specific solution on top of which

libraries have to be built to support every different implementation is in any way superior.

It would be helpful to know how and why you think the aforementioned designs are

"deeply flawed when used at scale". I have a couple of reasons why they are flawed;

protobuf relies on external code-generation tools, boost fusion requires a very
intrusive and tedious adaptation to be written, and Qt assumes a very specific

opt-in class hierarchy with considerable overhead. We know those issues, too,
and the plan of this SG is to avoid those problems from the get-go. Sticking our
head into the sand and hoping that an implementation-specific solution with 3rd party
libraries on top of it will not solve any of those problems, at all.

 

If it's completely implementation-specific, a valid exposure is to expose absolutely
nothing.

Correct. Another valid exposure is to supply a copy of libclang with every compiler, such that every compiler gets libclang bindings mapped into its compile-time space. Reflected AST parsing now becomes completely external to the current compiland. This, I would imagine, would be the easiest path for GCC and MSVC, perhaps even clang itself.

That sounds like standardizing libclang. Perhaps that would be a possible outcome, but I
highly doubt it. I don't think exposing an AST of any kind is a reasonable first step.
 

 

If it's not implementation-specific, we're already well on the way to standardize
an actual reflection API. The implementation-specific approach will accomplish absolutely

nothing and will only hinder the writing of portable code using reflection.

By defining Reflection to be compile-time only for now, we have defined away much of the difficulty in getting a good Reflection design right. Equally that means the present Reflection support will only EVER be for compile-time use only.

Replacing 'present' with 'near-term', maybe so, but that doesn't mean the next step of reflection
support is compile-time-only.

Which might actually be fine for C++-1y - this SG proposes some support and it is explicitly aimed for compile-time use and never anything more. In which case, my proposal still stands as a valid

That 'never' is something I don't agree with, and I don't know where it came from.
 

proposal for gaining information and good practice gathering for a Reflection implementation which does support dynamic Reflection after C++-1y.

It is therefore STILL a worthwhile item for this SG to consider, especially as a library based design can be evolved far faster than an ISO standard.

A language TS can evolve at a speed equal to a library TS.
 

[Niall Douglas]

I'm not trying to paint you anything. I'm saying that this idea is a step in the wrong direction,
and having it as an interim measure is not very useful.

I agree it is not as useful for simple compile-time reflection as less ambitious solutions. I think your judgement here is based on the immediate needs of C++-1y - and I have no problem with that BTW, so long as it is recognised that a simple compile-time only design will only be intended as such.

 

I have no reason to believe that an implementation-specific solution on top of which

libraries have to be built to support every different implementation is in any way superior.

Depends on scope of use case. If you push Reflection far enough, it inevitably enters implementation specificness anyway.

 

It would be helpful to know how and why you think the aforementioned designs are

"deeply flawed when used at scale". I have a couple of reasons why they are flawed;

protobuf relies on external code-generation tools, boost fusion requires a very
intrusive and tedious adaptation to be written, and Qt assumes a very specific

opt-in class hierarchy with considerable overhead. We know those issues, too,
and the plan of this SG is to avoid those problems from the get-go. Sticking our
head into the sand and hoping that an implementation-specific solution with 3rd party
libraries on top of it will not solve any of those problems, at all.

This is off topic for this SG's stated intent for 1y, but my concern lies in how to sanely reflect reflecting code, reflect the reflection of reflection, and then onto infinity of recursion, while operating in an environment with C++ Modules where a definition in one module might be subtly different (in a segmentation fault causing kind of way) from another definition in another module. Part of the difficulty here is that probably the link stage will have to late stage code generate according to cross-compiland Reflection (e.g. if module 1 reflects module 2 which reflects module 1 etc) which implies the need for storing not just a program's intermediate state to be traversed during link, but a mutable (yes, mutable) database of C++ compilands which can be queried for link and type reflection. Right now, I can't think of a "good" way of handling this, but I'm fairly sure that exposing the compiler's internal state to the code being compiled is a reasonable start.

You must remember where I'm coming from and my use case: getting some precompiled C++ blob of LLVM bytecode to load and interoperate with many other and unknown precompiled C++ blobs of LLVM bytecode even if they were compiled with different STL or Boost implementations. I'm thinking of a world where you simply don't have to recompile C++ blobs of LLVM bytecode even if the world surrounding them changes, because the C++ dynamic runtime and loader can see what needs doing and can cope.

You might say "this is an extreme case that will never materialise, and therefore isn't important here". And maybe it won't. Equally it's a very useful ad absurdum thought exercise for getting Python or Ruby or Java runtimes to far more seamlessly interoperate with some arbitrary chunk of C++, because if C++ can interoperate with C++, I would argue that any other language will find it much easier to do so as well. Equally, it's hard for C++ to argue for its increased usage as a systems programming language if it can't even interoperate with itself.

I absolutely agree everything I just said is off topic for this SG. But you did ask why I think the systems you mentioned are flawed at scale, so there you go.

 

That sounds like standardizing libclang. Perhaps that would be a possible outcome, but I

highly doubt it. I don't think exposing an AST of any kind is a reasonable first step.

From my own work I believe that even an AST for every compiland is insufficient for the kind of Reflection I just described. We in fact need all the ASTs, and then some more metadata again (i.e. new C++ keywords to disambiguate what the programmer really means by "class Foo" for example).

It is therefore STILL a worthwhile item for this SG to consider, especially as a library based design can be evolved far faster than an ISO standard.

A language TS can evolve at a speed equal to a library TS.
 

Who said anything about TSs? I have been clear since the beginning that third party libraries would evolve in whatever way they think best outside of ISO. Sometime later, when we have some idea of some viable design, then we might think about ISO standardisation. For the kind of reflection I was talking about earlier, this is almost certainly a C++-2x worry. Right now we have no idea what we don't know.

Niall

[Ville Voutilainen]

On 28 October 2013 21:54, Niall Douglas <nialldo...@gmail.com> wrote:

I'm not trying to paint you anything. I'm saying that this idea is a step in the wrong direction,
and having it as an interim measure is not very useful.

I agree it is not as useful for simple compile-time reflection as less ambitious solutions. I think your judgement here is based on the immediate needs of C++-1y - and I have no problem with that BTW, so long as it is recognised that a simple compile-time only design will only be intended as such.

My judgement is based on the needs of C++1y and beyond and having a sane evolution
path from the initial step onwards.
 

 

I have no reason to believe that an implementation-specific solution on top of which

libraries have to be built to support every different implementation is in any way superior.

Depends on scope of use case. If you push Reflection far enough, it inevitably enters implementation specificness anyway.

It does, inevitably? That's news to me. I have a pretty good idea how to avoid that

alleged inevitability, and that is to avoid implementation-specific solutions and
certainly to avoid standardizing an implementation rather than an API.

This is off topic for this SG's stated intent for 1y, but my concern lies in how to sanely reflect reflecting code, reflect the reflection of reflection, and then onto infinity of recursion, while operating in an environment with C++ Modules where a definition in one module might be subtly different (in a segmentation fault causing kind of way) from another definition in another module. Part of the difficulty here is that probably the link stage will have to late stage code generate according to cross-compiland Reflection (e.g. if module 1 reflects module 2 which reflects module 1 etc) which implies the need for storing not just a program's intermediate state to be traversed during link, but a mutable (yes, mutable) database of C++ compilands which can be queried for link and type reflection. Right now, I can't think of a "good" way of handling this, but I'm fairly sure that exposing the compiler's internal state to the code being compiled is a reasonable start.

Yes, and we have already established that I think it's a dead end, not a reasonable start.
 

You must remember where I'm coming from and my use case: getting some precompiled C++ blob of LLVM bytecode to load and interoperate with many other and unknown precompiled C++ blobs of LLVM bytecode even if they were compiled with different STL or Boost implementations. I'm thinking of a world where you simply don't have to recompile C++ blobs of LLVM bytecode even if the world surrounding them changes, because the C++ dynamic runtime and loader can see what needs doing and can cope.

You might say "this is an extreme case that will never materialise, and therefore isn't important here". And maybe it won't. Equally it's a very useful ad absurdum thought exercise for getting Python or Ruby

I don't think it's extreme at all. I also don't think it needs to be solved by exposing implementation

internals.
 

or Java runtimes to far more seamlessly interoperate with some arbitrary chunk of C++, because if C++ can interoperate with C++, I would argue that any other language will find it much easier to do so as well. Equally, it's hard for C++ to argue for its increased usage as a systems programming language if it can't even interoperate with itself.

I'm all for lowering the barriers of such interoperability. I can see several ways how things like
eg. javascript<->c++ and python<->c++ interoperability can benefit from reflection. Whether
that needs to be "use any blob any way you like" is a different matter.

 

That sounds like standardizing libclang. Perhaps that would be a possible outcome, but I

highly doubt it. I don't think exposing an AST of any kind is a reasonable first step.

From my own work I believe that even an AST for every compiland is insufficient for the kind of Reflection I just described. We in fact need all the ASTs, and then some more metadata again (i.e. new C++ keywords to disambiguate what the programmer really means by "class Foo" for example).

I have trouble following that "example". Most of the reflection cases I have seen deal with

objects of class Foo type, and couldn't care less where you eg. forward declare it, as long
as they can interoperate with the data members and functions of that class. Such things

do not need ASTs nor any even-more-precise information than that. Code generators
are a different matter, reifying objects from arbitrary data is much more complex. But
that seems to be a less common use, albeit the rarity may be contributed by lack of

facilities that would allow it.

 

It is therefore STILL a worthwhile item for this SG to consider, especially as a library based design can be evolved far faster than an ISO standard.

A language TS can evolve at a speed equal to a library TS.
 

Who said anything about TSs? I have been clear since the beginning that third party libraries would

Well, I did. :) In order to be in-scope for this SG, we should imho target either a TS or a standard.

That's also why I don't think putting open-ended wording about implementation-specific exposure
of reflection facilities into the standard; either specify the facility, or leave it out, but don't

almost-specify it at a vagueness level that has nothing portable whatsoever.
 

evolve in whatever way they think best outside of ISO. Sometime later, when we have some idea of some viable design, then we might think about ISO standardisation. For the kind of reflection I was talking about earlier, this is almost certainly a C++-2x worry. Right now we have no idea what we don't know.

I daresay I disagree. Many people in this SG have a very good idea what they know and what
is more unknown, and we have a pretty good idea how to solve the first evolutionary steps

so that the subsequent steps are still possible. And your proposal doesn't fit at least
my mental image of what we should do, when, and how. It's a proposal that takes a completely
different direction, and I believe that direction would be a mistaken one.

Note that I might think quite differently if there were existing practice of implementations

even remotely agreeing on reflection-like facilities. Currently there's next to none of that,
so it may fall as a task of this SG to find common ground and plow the way.

[Mike Gibson]

On Tue, Oct 22, 2013 at 12:49 PM, Niall Douglas <nialldo...@gmail.com> wrote:

My suggestion for implementing compile-time reflection is very simple, and is the absolute minimum possible: the compiler makes available a magic namespace, let's call it namespace __compiler_internals. In this namespace you get a set of pseudo template definitions which happen to make available the present parsing state of the compiler in this compiland. As I originally pitched this to Chandler regarding clang, I'll stick to a libclang example, but I should emphasise that by this I mean that each compiler makes available its internal state in whatever format it so chooses. It is then up to one or more third party libraries - probably at least one in Boost - to convert compiler-specific namespace __compiler_internals into a unified template API in order to achieve cross-compatibility. I entirely expect such thunk libraries to get repeatedly thrown away and refactored until something emerges which is pretty good for most use cases.

Nothing prevents anyone from doing this already in the clang code.  No one has done it yet AFAIK so I don't think that we can expect that our simple creation of a magic namespace would cause library writers to suddenly jump on the bandwagon.

The only thing that you're proposing is to create a magic namespace.  Nothing else is standardized.  Not even the name of this namespace need be standard under your scheme.  So how does this have anything to do with the standardization process?  I can see this approach being used to implement some of the ideas presented in the group, but I don't really see the point in spending time on it in a standardization group.

Mike Gibson

[Niall Douglas]

I daresay I disagree. Many people in this SG have a very good idea what they know and what is more unknown, and we have a pretty good idea how to solve the first evolutionary steps so that the subsequent steps are still possible. And your proposal doesn't fit at least my mental image of what we should do, when, and how. It's a proposal that takes a completely different direction, and I believe that direction would be a mistaken one.

I'm well known for proposing completely different directions, even if I know they stand no chance (as I knew before I started this thread). I believe they can help solidify thinking in others, and my thanks to you Ville for engaging. Also, if you don't turn up to WG21 meetings then generally any proposition or opinion you have tends to fall by the wayside (and correctly so I think), so SGs such as this and scanning the WG21 mailings and emailing the authors of N papers about things I think about their papers are as far as I can contribute, as I have never had an employer which permits me to attend such meetings except as vacation time. Best I can do - i.e. act as a devil's advocate - given limited resources unfortunately.

 

Note that I might think quite differently if there were existing practice of implementations even remotely agreeing on reflection-like facilities. Currently there's next to none of that, so it may fall as a task of this SG to find common ground and plow the way.

On this point I think we are in total agreement!

Niall

[Ville Voutilainen]

On 29 October 2013 23:53, Niall Douglas <nialldo...@gmail.com> wrote:

I'm well known for proposing completely different directions, even if I know they stand no chance (as I knew before I started this thread). I believe they can help solidify thinking in others, and my thanks to you Ville for engaging. Also, if you don't turn up to WG21 meetings then generally any proposition or opinion you have tends to fall by the wayside (and correctly so I think), so SGs such as this and

It is possible to find meeting-attending champions that can present your proposals in the
meetings. Having said that, I'm not going to champion this proposal. :)

[Remotion]

IMHO adding something like "magic namespace" is totaly wrong direction.
There is already have enough problems with RTTI and all its "magic", while it only provide a minimum of information.

I think the better way would be to start with compiler time only reflection as a library.
So do everything that is possible as C++14 and add some compiler intrinsics for everything that is not possible right now.

This proposal goes this way.
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3815.html
Unfortunately is covers only enums and this is really not enough.
We need class, field, method and function informations too...
Of course this is not as as easy as enums that apparently only requeres 3 compiler intrinsics.

__enumerator_list_size(Enum) -> size_t                number of enumerators in Enum
__enumerator_value(Enum,I) -> size_t                  value of I'th enumerator in Enum
__enumerator_identifier(Enum,I) -> string literal    identifier of I'th enumerator in Enum

Remo

[Thiago Macieira]

On quarta-feira, 18 de dezembro de 2013 09:51:32, Remotion wrote:
> IMHO adding something like "magic namespace" is totaly wrong direction.
> There is already have enough problems with RTTI and all its "magic", while
> it only provide a minimum of information.
>
> I think the better way would be to start with compiler time only reflection
> as a library.
> So do everything that is possible as C++14 and add some compiler intrinsics
> for everything that is not possible right now.
>
> This proposal goes this way.
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3815.html
> Unfortunately is covers only enums and this is really not enough.
> We need class, field, method and function informations too...
> Of course this is not as as easy as enums that apparently only requeres 3
> compiler intrinsics.
>
> __enumerator_list_size(Enum) -> size_t number of enumerators
> in Enum
> __enumerator_value(Enum,I) -> size_t value of I'th
> enumerator in Enum
> __enumerator_identifier(Enum,I) -> string literal identifier of I'th
> enumerator in Enum

I've been meaning to write a paper on how to extend std::typeinfo with the
information that you're requesting. We know it's possible to do it with
minimal runtime impact because there are existing implementations doing that.
Doing it inside the compiler would only make it more efficient.

The two questions I still have are:

- what classes to extract information on? The cost should only be paid for
people using the feature, so the classes should be explicitly marked for
extraction. Doing it for every single class is too much overhead, even if
read-only sharable memory, and besides it's not backwards compatible. Existing
code needs to be able to deal with non-reflected classes.

- where to draw the line on what information to extract. I'm guessing it's
pretty much a given that for a class we'd want to extract:
* a list of base classes, their virtualness and protection level
* a list of member functions, including the protection level, return type,
parameters, cv-qualifiers, ref qualifiers, virtualness, attributes
* a list of member variables, their types, cv qualifiers and protection level
* a list of enums, with their members
* probably some metadata / annotations in the form of attributes

But should we include the parameter names of the member functions? Should we
include inline functions? Should we include typedefs and nested structs? How
about member templates? And since we're talking about templates, if the class
is a template, should it include the template parameter list? What else should
we extract metadata on: namespaces, global variables?

Then we go into the detail of making all this useful. What's the use of
knowing that a function exists if you cannot call it? So you need a PMF for
each member function, probably wrapped in a neat std::function. But the API is
generic, so we need a generic function-calling mechanism à la libffi.

Finally, there will be issues with access protection. Given the typeinfo of a
class, you will be able to enumerate all protected and private members, which
means you could indirectly call them and bypass the protection mechanism.

class [[extract_reflection]] Foo
{
void private_function();
};

int main()
{
Foo f;
f.private_function(); // error

std::callable c = typeid(f).member_function("private_function()");
c(f); // ok
}

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center
PGP/GPG: 0x6EF45358; fingerprint:
E067 918B B660 DBD1 105C 966C 33F5 F005 6EF4 5358

[Remotion]

I think extending std::typeinfo is the wrong direction.
At leas for compiler time reflection/introspection.

My requirements at compiler time reflection are following:
It must be not intrusive. So it should be possible to use it with libraries.
Using attributes like [[extract_reflection]] is not an option if one want information about a third party library.
It mus have not runtime cost if it is not used and minimal if it is used.

 > what classes to extract information on?

Every class that the user will ask for at compiler time.

 > But should we include the parameter names of the member functions?

Yes, this would be helpful in some cases.

 > Should weinclude inline functions?
Of course yes.

 > Should we include typedefs and nested structs?

For example to create automatic serialization the type of all fields will be necessary.
So if a field type is a nested struct then this information will be necessary to access it own fileds.
Not sure about typedefs.

 > How about member templates?
Yes templates makes everything more complicated.

 > if the class is a template, should it include the template parameter list?

Probably no, this would be too complicated to handle.

 > What else should we extract metadata on: namespaces, global variables?

This information may be necessary in some cases but right now I have no clear idea how this can be handled.
Especially for anonymous namespaces.

Of course references / pointers to field and member function will be necessary in some cases.
But having this information is already a big win.

 > Finally, there will be issues with access protection.

Yes this is a issue. You can not properly serialize a class if you can not ready from or write to all of it fields.

Apparently some extra meta-data is also necessary in some cases.

struct my_data
{
  int *array [[reflect::is_array_ptr]];
  int array_length [[reflect::is_array_length]];
  float  value [[reflect::range(0.01, 20.0)]];
};


Again the problem with this approach is that it is intrusive.

Right now my favorite solution are a couple of compiler intrinsics that will give you all this information.

For example t extract information about member function flowing intrinsics can be used.

__record_method_count(T) -> size_t            how many member function class/struct T contains
__record_method_type(T,I) -> Type              type of I'th member function
__record_method_identifier(T,I) -> string      name of the I'th member function

Now this intrinsics can be warped into a library.

reflect::field<my_data>::size  == 3
reflect::method<my_data>::size == 0

Remo

[Thiago Macieira]

On quarta-feira, 18 de dezembro de 2013 11:59:53, Remotion wrote:
> I think extending std::typeinfo is the wrong direction.
> At leas for compiler time reflection/introspection.
>
> My requirements at compiler time reflection are following:
> It must be not intrusive. So it should be possible to use it with libraries.
> Using attributes like [[extract_reflection]] is not an option if one want
> information about a third party library.
> It mus have not runtime cost if it is not used and minimal if it is used.

We're not going to be able to meet everyone's requirements perfectly (just see
the discussion on the main list about inserting items into a vector).

It should be ok to trigger the extraction of an existing class that didn't
declare itself with any special marker. However, how do we ensure that we
don't bloat executables at runtime by duplicating information everywhere?
Classes should be able to opt-in to having the information in a canonical
place.

This is what compilers already do for the virtual tables. The type info
structure is emitted next to the vtable, which is what gave me the idea in the
first place.

> > what classes to extract information on?
>
> Every class that the user will ask for at compiler time.

Including all parents, I guess.

> > But should we include the parameter names of the member functions?
>
> Yes, this would be helpful in some cases.

Fair enough, we have it QMetaMethod too, though it is optional since the
function declaration may not have them.

> > Should weinclude inline functions?
>
> Of course yes.

But they may not be callable. Otherwise, it means instantiating all inline
functions just by asking for the meta information.

Incidentally, this may not work at all for template classes. Some member
cannot be instantiated, and thus may not be callable. Then again, the meta
data should just include the fact that instantiation failed.

> > Should we include typedefs and nested structs?
>
> For example to create automatic serialization the type of all fields will
> be necessary.
> So if a field type is a nested struct then this information will be
> necessary to access it own fileds.
> Not sure about typedefs.

The member variables yes, but I'm asking about typedefs and nested structs
that may not be used at all. For example:

template <typename T> struct Vector
{
typedef T value_type;
typedef T *pointer_type;
typedef T *iterator;

iterator begin();
};

Do we need to extract them? And even where they got used, should the metatype
contain "iterator" or the resolved type? Since we're here, should it include
"T *" or should it show the actual type for the class instantiation?

> > if the class is a template, should it include the template parameter
>
> list?
> Probably no, this would be too complicated to handle.

So probably we can extract information about full template instantiations, but
not about the templates themselves or partial specialisations.

Similar to how you can do typeid(std::vector<int>) but not
typeid(std::vector).

> > Finally, there will be issues with access protection.
>
> Yes this is a issue. You can not properly serialize a class if you can not
> ready from or write to all of it fields.
>
> Apparently some extra meta-data is also necessary in some cases.
>
> struct my_data
> {
> int *array [[reflect::is_array_ptr]];
> int array_length [[reflect::is_array_length]];
> float value [[reflect::range(0.01, 20.0)]];
> };
>
>
> Again the problem with this approach is that it is intrusive.

Right. That's why the class needs to declare its reflection metadata by itself.
I don't see how you could elsewhere just declare that the range of the value
is 0.01 to 20.0. And what happens if it got declared differently elsewhere?
Program is ill-formed?

> Right now my favorite solution are a couple of compiler intrinsics that
> will give you all this information.
>
> For example t extract information about member function flowing intrinsics
> can be used.
>
> __record_method_count(T) -> size_t how many member function
> class/struct T contains
> __record_method_type(T,I) -> Type type of I'th member function
> __record_method_identifier(T,I) -> string name of the I'th member
> function
>
> Now this intrinsics can be warped into a library.
>
> reflect::field<my_data>::size == 3
> reflect::method<my_data>::size == 0

Right.

I'm thinking a bit lower-level, though. How would this information be stored
in memory? A good paper should provide implementation hints, to prove that
this is feasible, like we've done for the enums. And IMHO it should also prove
that this information can be mostly relocation-free.

I also want this generic, not a template. I want something like:

void dump(const std::typeinfo *ti)
{
static const char *aggregates[] = { "struct ", "class " };
std::cout << aggregates[ti->aggregate_type()] << ti->pretty_name();
if (ti->base_classes()) {
bool first = true;
for (auto base: ti->base_classes())
std::cout << (first ? (first = false, " : ") : ", ")
<< base.pretty_name();
}
std::cout << '{' << std::endl;
for (auto m: ti->member_functions())
std::cout << m.name() << std::endl;
std::cout << "};" << std::endl;
}

To cite prior art:

void dump(const QMetaObject *mo)
{
std::cout << "class " << mo->className();
if (mo->superClass()) // Qt meta object only supports one Q_OBJECT base
std::cout << " : " << mo->superClass()->className();

std::cout << '{' << std::endl;
for (int i = 0; i < mo->constructorCount(); ++i)
std::cout << mo->constructor(i).name(); std::endl;
for (int i = 0; i < mo->methodCount(); ++i)
std::cout << mo->method(i).name(); std::endl;
std::cout << "};" << std::endl;
}

This could be implemented by an entirely template solution, wrapped in a class
with virtuals to get the generic data. That's... sub-optimal. The compiler can
do better.

[Sean Middleditch]

On Wednesday, December 18, 2013 1:06:56 PM UTC-8, Thiago Macieira wrote:

It should be ok to trigger the extraction of an existing class that didn't
declare itself with any special marker. However, how do we ensure that we
don't bloat executables at runtime by duplicating information everywhere?
Classes should be able to opt-in to having the information in a canonical
place.

If it's compile-time, there is zero bloat.  There isn't any extra data output unless the compile-time reflector chooses to do so.

 

>  > Should weinclude inline functions?
>
> Of course yes.

But they may not be callable.

Inline today is just a weak hint to the compiler in most cases.  If you take the address of the function, it still works as expected, just like reflection should.  If we have to start stripping inline keywords from small functions that were marked up "in the old days" when people thought inline meant something - or if we have to start moving trivial accessors out of class definitions - just to make reflection useful, something is very broken.

 

Incidentally, this may not work at all for template classes. Some member
cannot be instantiated, and thus may not be callable. Then again, the meta
data should just include the fact that instantiation failed.

What is the use case for enumerating methods that don't actually exist?  I'm envisioning an API where every single use of a method has to be wrapped with a boiler plate "is_instantiated()" check.

 

>  > Should we include typedefs and nested structs?
>
> For example to create automatic serialization the type of all fields will
> be necessary.
> So if a field type is a nested struct then this information will be
> necessary to access it own fileds.
> Not sure about typedefs.

The member variables yes, but I'm asking about typedefs and nested structs
that may not be used at all. For example:

It makes sense to reflect only the underlying type.  If and when C++ gains opaque type aliases, then it would make sense to not "see through" those (and only those).

 

>  > if the class is a template, should it include the template parameter
>
> list?
> Probably no, this would be too complicated to handle.

So probably we can extract information about full template instantiations, but
not about the templates themselves or partial specialisations.

If talking about a compile-time system... yes.  Needed.  I use reflected templates and partial specializations currently with our reflection system to mark up that a vector is a "sequential container" so we know how to present it in property editor UIs and network serialization.  Likewise for comprehending that a smart pointer is a pointer and with which ownership semantics.  Partial specializations comes in handy more rarely, but they've come up for functions used in type conversions (think streams, lexical_cast, etc.).

>  > Finally, there will be issues with access protection.
>
> Yes this is a issue. You can not properly serialize a class if you can not
> ready from or write to all of it fields.
>
> Apparently some extra meta-data is also necessary in some cases.
>
> struct my_data
> {
>   int *array [[reflect::is_array_ptr]];
>   int array_length [[reflect::is_array_length]];
>   float  value [[reflect::range(0.01, 20.0)]];
> };
>
>
> Again the problem with this approach is that it is intrusive.

Right. That's why the class needs to declare its reflection metadata by itself.
I don't see how you could elsewhere just declare that the range of the value
is 0.01 to 20.0. And what happens if it got declared differently elsewhere?
Program is ill-formed?

ODR.  We already have this problem.  And yes, this use case does come up.  Think of a third-party library.  I can't reasonably mark up their headers but I could add a definition to my code.  Existing C++ reflection facilities already support this (partially because with most of them you have to define all metadata externally to the class).

In our current game engine we have almost zero fields that are lacking at least one attribute.

Note this is also another huge case for not doing this at runtime, in any way.  I have no need for editor attributes in a final release build of a game.  As proposed here for a runtime system I'd have to #ifdef the declaration of the metadata.  With our system we can #ifdef the attribute class definition to turn into an empty do-nothing attribute that gets further optimized way during compilation.

And yes, binary size really does matter.  When you're counting kilobytes to pack a game and its assets into the 10-50MB 3G download limits for a mobile app, things like this start to matter, as does RTTI and exceptions (even "zero cost" implementations still need unwinding tables).  It rarely comes down to that, but I don't want to be the guy stuck trying to figure out what valuable content to cut from a game because we ended up depending on a reflection system that is spewing out gobs of unneeded data.

 

> Right now my favorite solution are a couple of compiler intrinsics that
> will give you all this information.
>
> For example t extract information about member function flowing intrinsics
> can be used.
>
> __record_method_count(T) -> size_t            how many member function
> class/struct T contains
> __record_method_type(T,I) -> Type              type of I'th member function
> __record_method_identifier(T,I) -> string      name of the I'th member
> function

>
> Now this intrinsics can be warped into a library.
>
> reflect::field<my_data>::size  == 3
> reflect::method<my_data>::size == 0

Right.

Yes, yes, and more yes. :)

  

I'm thinking a bit lower-level, though. How would this information be stored
in memory? A good paper should provide implementation hints, to prove that
this is feasible, like we've done for the enums. And IMHO it should also prove
that this information can be mostly relocation-free.

Again, compile time.  How things are stored in memory is irrelevant at compile time.  How things are stored at run-time in any theoretical standard can be left as undefined as type_info.

 

I also want this generic, not a template. I want something like:

void dump(const std::typeinfo *ti)
{

Others would deeply prefer to not have this.  Adding this would mandate a -fno-runtime-reflection that quite possibly every single major player in our industry would be forced to use.  Yes, we need and use runtime reflection, but what's wanted is a very tailored and minimal runtime size/processing overhead that exposes only the subset of possible data.  We might use a compile-time reflector over methods for scripting binding and not want to expose it at all during runtime where we perhaps need only the ability to map string name to class names to look up component types from data files and property names.

Which brings up another point against runtime reflection.  It's almost certainly missing data that we need.  "Properties" are one example.  We don't need them in C++ itself, but when it comes to binding objects to editor UIs or network serialization they become important.  I _very_ often want to avoid serializing a particular member variable or end up wanting to use a specific getter/setter.  Consider a field like "size" which on some data structures is not explicitly stored but is implicitly calculated with something like end_ptr-begin_ptr.  A pure C++ reflection system is not going to comprehend this issue.  It's going to see member variables and it's going to see methods, and it won't have a way to know that two particular methods make up a getter/setter pair, or that a single method is meant to represent a read-only or write-only "property".

We can with exisitng systems write something like:

   REFLECT(MyType)

   .base<BaseType>()

   .property("foo", &MyType::m_foo, attr::range(0, 100))

   .property("bar", &MyType::GetBar, &MyType::SetBar, attr::name("Bar controls the nozzle"))

   .property("baz", &MyType::m_baz)

   .method("Stuff", &MyType::Stuff, attr::expose(kSecureDomain), attr::help("Realigns the widgets"));

That is a syntax and feature set I literally can (and do) support today.  The only macro involved is REFLECT (and a few variants for templates and few other cases), the rest are part of a ReflectionBuilder<> template.  It supports reflecting private member variables and function iff the class definition gets a small macro addition (also used to enable polymorphic reflection) though it also fully works for types that don't have the header modification (you can even add reflection to primitive types like int, necessary of course to reflect properties of primitive types).

If constexpr were available more widely this could be made even more efficient rather easily (and a future C++ reflection standard should of course assume full C++14 constexpr support).  Even more runtime-optimized systems are possible with approaches vaguely similar to clReflect (https://bitbucket.org/dwilliamson/clreflect).  The data can be extracted and then massaged and output directly to the desired in-memory data structures allowing the most efficient use (such as the generation of perfect hash functions for the set of properties/methods/classes in use) with no run-time processing overhead.

A compile-time reflection facility would greatly help in terms of reducing redundancy.  With some of the simpler compile-time features proposed, the above could trivially turn into:

   REFLECT(MyType)

   .property("foo",  attr::range(0, 100))

   .property("bar", attr::name("Bar controls the nozzle"))

   .property("baz")

   .method("stuff", attr::expose(kSecureDomain), attr::help("Realigns the widgets"));

That requires the ability to enumerate base types, to lookup fields or methods by constexpr string name, and not much else.  The .property("baz") line could be removed by allowing the enumeration of fields, those personally I prefer the requirement to explicitly declare what I want reflected.  Add in the ability to put in user-defined attributes and it wouldn't even be necessary to redeclare properties or methods just to add attributes unless of course one is marking up a class provided by a third party.

Most types could be as trivial as:

   [[reflect]]

   class MyType {

     [[reflect::range(0, 100)]

     int foo;

     [[reflect::name("Bar controls the nozzle")]]

     float bar;

     [[reflect]]

     std::string baz;

     [[expose(kSecureDomain), help("Realigns the widgets")]]

     void Stuff(int, float, char);

   };

That requires user-defined attributes as well as the ability to enumerate the attributes for each member variable/function.  Bonus points if an attribute can have a template callback of some kind invoked when it is used on a particular type/member (most used so that [[reflect]] on the class definition instantiates the reflector on itself), otherwise there's still a need to explicitly instantiate the reflector outside the class definition somewhere.

A standardized runtime reflection system would lose half of those features and bloat executable size with all kinds of other excess information.  It solves almost zero real use cases I've seen (mostly just in games, but I'd wager our industry has been using and experimenting with C++ reflection more than most other using C++-using industries).

A compile-time reflection system in C++ would be immensely useful.  A runtime reflection system is likely just going to contribute another -fno flag to the standard game development build flags in the worst case and simply be ignored in all but the most unicorn-like perfect case.  :)

[Thiago Macieira]

On quarta-feira, 18 de dezembro de 2013 17:20:03, Sean Middleditch wrote:
> > > > Should weinclude inline functions?
> > >
> > > Of course yes.
> >
> > But they may not be callable.
>
> Inline today is just a weak hint to the compiler in most cases. If you
> take the address of the function, it still works as expected, just like
> reflection should. If we have to start stripping inline keywords from
> small functions that were marked up "in the old days" when people thought
> inline meant something - or if we have to start moving trivial accessors
> out of class definitions - just to make reflection useful, something is
> very broken.

Good point. That means taking the address of the function via reflection will
cause it to be called, which means it will be emitted (unless the compiler can
prove it will never be called and then do dead-code elimination).

> > Incidentally, this may not work at all for template classes. Some member
> > cannot be instantiated, and thus may not be callable. Then again, the meta
> > data should just include the fact that instantiation failed.
>
> What is the use case for enumerating methods that don't actually exist?
> I'm envisioning an API where every single use of a method has to be
> wrapped with a boiler plate "is_instantiated()" check.

That's not my point. The point is that some member functions in a template
class may compile after replacement. For example:

template <typename T> struct Ptr
{
T *ptr;
Ptr(T *ptr) : ptr(ptr) {}
~Ptr() { free(ptr); }
static Ptr create()
{
void *ptr = malloc(sizeof(T));
new (ptr) T;
return Ptr(static_cast<T *>(ptr));
}
};

Ptr<void>::create does not compile, so it cannot exist. Should the metadata
about Ptr<void> contain any note that the function existed in the template but
doesn't in the instantiation?

I currently don't have any use-case for the reflection metadata containing
information about functions that could've existed but don't. Someone else
might.

> It makes sense to reflect only the underlying type. If and when C++ gains
> opaque type aliases, then it would make sense to not "see through" those
> (and only those).

Agreed. Next question: should the type list be names or should it be another
reflection structure? I'd go for the latter. That would mean you can "see
through" the opaque typedef by querying what it's a typedef of.

> If talking about a compile-time system... yes. Needed. I use reflected
> templates and partial specializations currently with our reflection system
> to mark up that a vector is a "sequential container" so we know how to
> present it in property editor UIs and network serialization. Likewise for
> comprehending that a smart pointer is a pointer and with which ownership
> semantics. Partial specializations comes in handy more rarely, but they've
> come up for functions used in type conversions (think streams,
> lexical_cast, etc.).

Sounds like we're really talking about two systems here. We need a template
reflection system that can be used by other template classes so do
metaprogramming, similar to <type_traits>. On top of that, the compiler should
provide a stored, generic implementation, containing mostly similar
information, but which wouldn't be evaluated at compile time.

> ODR. We already have this problem. And yes, this use case does come up.
> Think of a third-party library. I can't reasonably mark up their headers
> but I could add a definition to my code. Existing C++ reflection
> facilities already support this (partially because with most of them you
> have to define all metadata externally to the class).

I've only worked with metadata stored in the class, implemented by parsing the
code and generating the data, so I have no experience. Can you suggest a way
add the metadata on an already-existing class?

> > I'm thinking a bit lower-level, though. How would this information be
> > stored
> > in memory? A good paper should provide implementation hints, to prove that
> > this is feasible, like we've done for the enums. And IMHO it should also
> > prove
> > that this information can be mostly relocation-free.
>
> Again, compile time. How things are stored in memory is irrelevant at
> compile time. How things are stored at run-time in any theoretical
> standard can be left as undefined as type_info.

The standard can leave it undefined, but someone needs to write one PoC
implementation that shows that it won't require tons of data and relocations.

> Others would deeply prefer to not have this. Adding this would mandate a
> -fno-runtime-reflection that quite possibly every single major player in

Unless it's implemented as a second step, as an opt-in request. Let's say we
do have the compile-time reflection. Since the compiler generates the data,
it's known at compile-time and is usable by constexpr functions. It will also
be part of dead-code and dead-data elimination. But if it's not dead, it will
need to be emitted.

Just as an example:

static const int i = 1;

If you only use i as a prvalue, the compiler doesn't need to use memory for
it. It will be eliminated. However, if you take its address, it will need to
emit the variable and allocate space for it (that includes passing it by
const-ref, which is quite surprising).

> Which brings up another point against runtime reflection. It's almost
> certainly missing data that we need. "Properties" are one example. We
> don't need them in C++ itself, but when it comes to binding objects to
> editor UIs or network serialization they become important. I _very_ often
> want to avoid serializing a particular member variable or end up wanting to
> use a specific getter/setter. Consider a field like "size" which on some
> data structures is not explicitly stored but is implicitly calculated with
> something like end_ptr-begin_ptr. A pure C++ reflection system is not
> going to comprehend this issue. It's going to see member variables and
> it's going to see methods, and it won't have a way to know that two
> particular methods make up a getter/setter pair, or that a single method is
> meant to represent a read-only or write-only "property".
>
> We can with exisitng systems write something like:
>
> REFLECT(MyType)
> .base<BaseType>()
> .property("foo", &MyType::m_foo, attr::range(0, 100))
> .property("bar", &MyType::GetBar, &MyType::SetBar, attr::name("Bar
> controls the nozzle"))
> .property("baz", &MyType::m_baz)
> .method("Stuff", &MyType::Stuff, attr::expose(kSecureDomain),
> attr::help("Realigns the widgets"));

The best I can offer is that those would be rolled up in C++ attributes at the
class level.

> A compile-time reflection facility would greatly help in terms of reducing
> redundancy. With some of the simpler compile-time features proposed, the
> above could trivially turn into:
>
> REFLECT(MyType)
> .property("foo", attr::range(0, 100))
> .property("bar", attr::name("Bar controls the nozzle"))
> .property("baz")
> .method("stuff", attr::expose(kSecureDomain), attr::help("Realigns the
> widgets"));

[snip]

> Most types could be as trivial as:
>
> [[reflect]]
> class MyType {
> [[reflect::range(0, 100)]
> int foo;
>
> [[reflect::name("Bar controls the nozzle")]]
> float bar;
>
> [[reflect]]
> std::string baz;
>
> [[expose(kSecureDomain), help("Realigns the widgets")]]
> void Stuff(int, float, char);
> };
>

Sounds sane enough :-)

> That requires user-defined attributes as well as the ability to enumerate

You can make a non-user reflect attribute that takes an arbitrary payload
instead. Or a namespace: reflect::user::help("Realigns the widgets").

> the attributes for each member variable/function. Bonus points if an
> attribute can have a template callback of some kind invoked when it is used
> on a particular type/member (most used so that [[reflect]] on the class
> definition instantiates the reflector on itself), otherwise there's still a
> need to explicitly instantiate the reflector outside the class definition
> somewhere.

Not sure I understood this part. Can you clarify what you meant?

> A standardized runtime reflection system would lose half of those features
> and bloat executable size with all kinds of other excess information. It
> solves almost zero real use cases I've seen (mostly just in games, but I'd
> wager our industry has been using and experimenting with C++ reflection
> more than most other using C++-using industries).

So your proposal is to provide *only* a template & constexpr reflection system
that is able to extract almost everything under the sun and no generic
library, but then let people write their on implementations on top that would
extract and record the information that they need. Correct?

[Remotion]

For me this is the proffered way.
It should possible to build runtime reflection based on compiler time reflection.
We do not need to change the language it self so it will still be C++14 but with a couple of lightweight reflection traits that can be used by any reflection library.

For example this construct will print all the method names at runtime.

template<class T, size_t... I>
void print_methods_impl(index_sequence<I...>) {
    const char* names[] = { __record_method_identifier(T, I)... };
    for (const auto& name : names){
        printf(" %s \n", name);
    }
}
template<class T, typename Indices = make_index_sequence<__record_method_count(T)>>
void print_methods() {
    return print_methods_impl<T>(Indices());
}

The advantage of this approach is that it is already working and is relatively easy to implement at leas in Clang.
One question is what information do we really need ?

Remo

[Péter Németh]

Do you think, that this is an easy to understand, intuitive way of reflection?

template<class T, size_t... I>
void print_methods_impl(index_sequence<I...>) {
    const char* names[] = { __record_method_identifier(T, I)... };

I also prefer the compile time reflection way, but I don't like this so complicate template meta programming - vararg style hack.
I want to get an constexpr AST node object from a (template)parameter in a clean way.

Peter

[Remotion]

Not it is not easy to understand way of reflection.
This is only low level way of reflection that can be used to build library on it.

> I also prefer the compile time reflection way, but I don't like this so complicate template meta programming - vararg style hack.

This is not a hack this are only variadic templates and index_sequence from C++14 and nothing else.

> I want to get an constexpr AST node object from a (template)parameter in a clean way.

How, can you please be more concrete?

By the way I think working but may be not easy to use reflection is better as no reflection at all.

Remo

[Thiago Macieira]

On sexta-feira, 20 de dezembro de 2013 05:12:24, Remotion wrote:
> For me this is the proffered way.
> It should possible to build runtime reflection based on compiler time
> reflection.

As long as we agree that the litmus test of the compile-time reflection is the
ability to build a working runtime reflection solution, I'm good with it.

[Remotion]

Start with compiler time reflection and then build runtime reflection on top of it.
No changes to language, no new language constructs only a couple of compiler intrinsics just like current type_traits but a lot more powerful.

Right now all reflection libraries using something like this.

   REFLECT(MyType)

   .base<BaseType>()

   .property("foo", &MyType::m_foo, attr::range(0, 100))

   .property("bar", &MyType::GetBar, &MyType::SetBar, attr::name("Bar controls the nozzle"))

   .property("baz", &MyType::m_baz)

   .method("Stuff", &MyType::Stuff, attr::expose(kSecureDomain), attr::help("Realigns the widgets"));

A lot of MACROS or templates.

I hope that this can be simplified in common cases to just this:
   REFLECT(MyType)

Remo

[Péter Németh]

On Friday, December 20, 2013 5:35:44 PM UTC+1, Remotion wrote:

Not it is not easy to understand way of reflection.
This is only low level way of reflection that can be used to build library on it.

> I also prefer the compile time reflection way, but I don't like this so complicate template meta programming - vararg style hack.
This is not a hack this are only variadic templates and index_sequence from C++14 and nothing else.

Ok, I don't know the newest C++14 variadic templates feature, but I think that syntax in the way to go.
 

> I want to get an constexpr AST node object from a (template)parameter in a clean way.
How, can you please be more concrete?

In my proposal https://github.com/hun-nemethpeter/cpp-reflector-mini/blob/master/Proposal.md you get an AST node from a parameter or from whatever.

// origin
enum class EVote
{
  Yes,
  RatherYes,
  Undecided,
  RatherNo,
  No
};

// driver
class EnumDriver
{
  public:
    constexpr EnumDriver(const EnumDecl& enumDecl)
    {
      for (auto& enumerator : enumDecl.enumerators())
        enumValueNames.push_back(enumerator.getName());
    }

    // meta::vector is a constexpr vector
    meta::vector<meta::id_name> enumValueNames;
};

// template with attached driver
template<typename T> $use(EnumDriver driver)
void Json::readFrom(T& obj, const std::string& data)
{
  obj = llvm::StringSwitch<T>(data)
    $for (auto enumValueName : driver.enumValueNames) {
      .Case($enumValueName.asStr(), $enumValueName) }
  ;
}

// usage
int main()
{
  Evote vote;
  Json::readFrom(vote, "EVote::Undecided");

  return 0;
}

So with the $use keyword you will got an AST node.

 

[Remotion]

For me this code is looking more complicated as my example.
It also uses some Clangs/LLVM internals that other compiler just can not use.

The idea is just to add some new intrinsic to the compiler and no new keywords like $use.
Of course intrinsics are new keywords but the are used only internal in the reflection library.
Just like type_traits do this already.

Remo

[Péter Németh]

On Friday, December 20, 2013 6:20:57 PM UTC+1, Remotion wrote:

For me this code is looking more complicated as my example.

Really? At least i does not use template meta programming and varargs.
 

It also uses some Clangs/LLVM internals that other compiler just can not use.

No, I don't want to use clang internals, only a standardized interface of it.

 
The idea is just to add some new intrinsic to the compiler and no new keywords like $use.

But it does not leads to a general compile time reflection solution. You can not generate code with it.
It looks like as a temporary solution. But as a temporary solution I can live with it, at least it is better than nothing...

Peter
 

[Remotion]

 > Really? At least i does not use template meta programming and varargs.

This was not template meta programming but only template programming and not varargs but variadic templates from C++11.
But I agree variadic templates can look strange if you are not familiar with it.

 > No, I don't want to use clang internals, only a standardized interface of it.

But how to make such standardized interface for whole AST ?
If you look at Clang AST then it is not really easy to create simplified interface to it.

 > But it does not leads to a general compile time reflection solution. You can not generate code with it.

Yes it is not general compiler time reflection.  To be more precise it is compiler time introspection.
But I think it should be possible to generate "code" using it and templates.

 > But as a temporary solution I can live with it, at least it is better than nothing...

Yes this can be temporal solution or better first step of the solution.

Right now all this intrinsics and more are already implemented and working in experimental branch of Clang.
There is not proper library for it but it already can be used in the code.
So the example above can be compiled and work right now.

Remo

[Péter Németh]

On Friday, December 20, 2013 6:48:02 PM UTC+1, Remotion wrote:

 > Really? At least i does not use template meta programming and varargs.
This was not template meta programming but only template programming and not varargs but variadic templates from C++11.
But I agree variadic templates can look strange if you are not familiar with it.

It looks horrible! IMHO, we should deprecate the whole template programming it just a brain dead idea.
Some years ago I wrote some TMP code, it is just a dead end.
https://github.com/hun-nemethpeter/cpp-meta-sql
But maybe it just off topic here, so it my personal opinion here.
 

 > No, I don't want to use clang internals, only a standardized interface of it.
But how to make such standardized interface for whole AST ?
If you look at Clang AST then it is not really easy to create simplified interface to it.

I don't think so. I see no problem here. Maybe it takes some time to standardizing the names, but it is doable.
Maybe a temporal solution can be, that the Clang API is the standard and thats all.
Language feature with an experimental API.
 

 > But it does not leads to a general compile time reflection solution. You can not generate code with it.
Yes it is not general compiler time reflection.  To be more precise it is compiler time introspection.
But I think it should be possible to generate "code" using it and templates.

There was a paper named: "Call for Compile-Time Reflection Proposals"
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3814.html
We should solve at least that examples.

 

 > But as a temporary solution I can live with it, at least it is better than nothing...
Yes this can be temporal solution or better first step of the solution.

Right now all this intrinsics and more are already implemented and working in experimental branch of Clang.
There is not proper library for it but it already can be used in the code.
So the example above can be compiled and work right now.

Im unsure. So "Hey people! We have a temporal solution for compiler time introspection!" sounds strange to me.
So we should not standardizing it, but it should be a working use-case or something like that.

Peter
 

[Ville Voutilainen]

On 20 December 2013 20:23, Péter Németh <hun.nem...@gmail.com> wrote:
>> > Really? At least i does not use template meta programming and varargs.
>> This was not template meta programming but only template programming and
>> not varargs but variadic templates from C++11.
>> But I agree variadic templates can look strange if you are not familiar
>> with it.
> It looks horrible! IMHO, we should deprecate the whole template programming

And the dollar-infested DSL you're proposing doesn't?

[Péter Németh]

The DSL extension? I think it will be awesome for native HTML5 support, or just JSON at least. I like it. But I accept your opinion.

// full grammar is here: http://www.json.org/
struct JsonParamGrammarItem : meta::grammar
{
  meta::id_name key;
  meta::op_name colon = ':';
  meta::expr value;
};

struct JsonParamGrammarTail : meta::grammar
{
  meta::id_name comma = ',';
  JsonParamGrammarItem item;
};

struct JsonParamGrammar : meta::grammar
{
  meta::symbol open_brace = '{';
  JsonParamGrammarItem paramFirst;
  meta::vector<JsonParamGrammarTail> paramMore; // this can be used for varargs ...
  meta::symbol close_brace = '}';
};

class JsonParamDriver
{
   JsonParamDriver(JsonParamGrammar grammar);
};

/*
grammar rules
or - meta::or
and - member in struct
optional - meta::optional
any - meta::vector

so JsonParamGrammarItem is
meta::id_name & ':' & meta::expr
*/

// usage
class SomeWidget
{
  template<astnode Node>
  SomeWidget(Node) $use(JsonParamDriver driver)
  {
    ...
    // We should process with an other driver that associate member names with param names.
  }

  SomeWindow window;
  SomeLabel label;
};

SomeWidget widget({
                     window: "Hello world",
                     label:  "Foo"
                  });

[Remotion]

 > It looks horrible! IMHO, we should deprecate the whole template programming

So what you want is to deprecate half of the C++ ?
Then probably you need to look at another language, may be D  ?
IMHO we should first create working and better alternative before deprecated anything.
And we should first start with PREPROCESSOR, but this is off topic now.

 > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3814.html
 > We should solve at least that examples.

4. Enumeration of other entities can be solved by this proposal:
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2013/n3815.html

1 and 2 could probably be solved by solution that I try to describe above.

3. Compile-time context information.
Solution for this is also required by me and I have a couple of ideas how this could be solved.
But I will know more once I will try to implement them...

Remo

[Péter Németh]

On Friday, December 20, 2013 7:43:51 PM UTC+1, Remotion wrote:

 > It looks horrible! IMHO, we should deprecate the whole template programming
So what you want is to deprecate half of the C++ ?
Then probably you need to look at another language, may be D  ?
IMHO we should first create working and better alternative before deprecated anything.
And we should first start with PREPROCESSOR, but this is off topic now.

My proposal is a good for deprecating the preprocessor (except header guard). I have no problem with the template itself, but
the template meta programming is just a disaster. TMP is definitely not the half of the C++.  But yes, this is OFF now.
 
Peter