[boost] About all these metaprogramming libraries

[Louis Dionne via Boost]

Dear Boost,

We've seen two metaprogramming libraries ask for a formal review on the list
recently. I expect a third (Brigand) might join the party soon, and maybe
even a fourth (Eric Niebler's meta). I think it would be wise to have a plan
for how we're going to deal with this.

Before going further, I'd also like to clear something out. Some people have
asked whether "classic" metaprogramming libraries still had their place
given that we now have Hana. As the author of Hana, I think the answer is
yes. While Hana is (IMHO) easier to use and more powerful, it also has an
important shortcoming. Hana is too slow when dealing with very large inputs.
This is due to the fact that it can also handle values, which is much
heavier than dealing with types only. This can be alleviated to some extent,
but the truth is that our execution model (the compiler) simply has to do
more work when handling values.

Hence, I think Boost does need a modern classic TMP library to handle these
cases where a library writer needs to handle gigantic type lists. I think
marketing such a library as an advanced tool for library writers would be a
service to the overall C++ community, but that's an orthogonal concern.

However, Boost needs one such library, not four. I think we can't just do 4
reviews and include all the libraries that pass in Boost; that would be a
huge disservice to our users, who will be left with the burden of choosing.
Heck, if we can't even make our mind, how can they?

So, how do we pick? Have we ever been in a similar situation where we have
multiple competing libraries solving _exactly_ the same problem?

Louis




--
View this message in context: http://boost.2283326.n4.nabble.com/About-all-these-metaprogramming-libraries-tp4692524.html
Sent from the Boost - Dev mailing list archive at Nabble.com.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Edouard via Boost]

Hi Louis,

>We've seen two metaprogramming libraries ask for a formal review on the list recently. I expect a third (Brigand) might join the party soon, and maybe even a fourth (Eric Niebler's meta). I think it would be wise to have a plan for how we're going to deal with this.

I am one of the authors of Brigand (https://github.com/edouarda/brigand), and it's true we're considering submitting it. This is upon request by Boost library authors who would like to use Brigand to reduce the compilation time of their libraries. If the submission is deemed inappropriate or useless it will not happen.

>Hence, I think Boost does need a modern classic TMP library to handle these cases where a library writer needs to handle gigantic type lists. I think marketing such a library as an advanced tool for library writers would be a service to the overall C++ community, but that's an orthogonal concern.

I agree with you. I also think that just patching Boost.MPL to make it leverage C++ 11 will not be sufficient. That's how Brigand got started and very quickly we realized we were at a dead end. Then I read Peter Dimov's article and realized there was a better way. :-)

>However, Boost needs one such library, not four. I think we can't just do 4 reviews and include all the libraries that pass in Boost; that would be a huge disservice to our users, who will be left with the burden of choosing.
>Heck, if we can't even make our mind, how can they?

I'm afraid the main produce of 4 reviews might be an extraordinary amount of bike shedding.

>So, how do we pick? Have we ever been in a similar situation where we have multiple competing libraries solving _exactly_ the same problem?

Does a decision need to be taken urgently? The aforementioned libraries are currently available and ready to be used. To me, the inclusion into Boost is more about the process than the inclusion itself. The process will raise questions and issues and shed some light on unsuspected usage scenarii.

At CppCon we discussed if the language and the standard library needed improvements to make TMP even more useful, convenient and faster.

I submit discussing what kind of TMP library should be added to Boost will be very beneficial.

-Edouard

[Vicente J. Botet Escriba via Boost]

If there were several proposals, we could review all of them together.
We already did that for the thread/futures proposals.

I believe that some of the things we need to answer are:
* do we want (can) to use as (HOMF) high-order meta-functions? if we
nedd them at all). IIUC, Peter library doesn't works with HOMFs.
* do we want a Hana style with "concepts" and with customizations? Do we
want other data type than type lists? IIUC Peter's library works only
with template aliases as data types and almost variadic class template
with type parameters is a good candidate for a type list (even
std::variant :( )
* do we need lazy evaluation?
* do we need lambdas?
* do we want a C++11/C++14/C++17 library?
.... other you can think of.

However I see only one official proposal (mp11 - Peter Dimov), even if
there are other interesting libraries (as Meta, Brigand).


Vicente

[Louis Dionne via Boost]

> If the submission is deemed inappropriate or useless it will not happen.

No, I think submitting Brigand for review is an excellent idea. We just have
to figure out how to deal with 3-4 competing libraries in the same domain.

> I agree with you. I also think that just patching Boost.MPL to make it
> leverage C++ 11 will not be sufficient.

Agreed. We all tried that (me included a few years ago), and it's a bad
idea.

> Does a decision need to be taken urgently?

Not as far as I'm concerned. I was just trying to handle the fact that right
now, two authors have officially asked for a formal review. If we just
happily do the reviews for those two libraries without thinking, we'll get
into the scenario I'd like to avoid.




--
View this message in context: http://boost.2283326.n4.nabble.com/About-all-these-metaprogramming-libraries-tp4692524p4692531.html

Sent from the Boost - Dev mailing list archive at Nabble.com.

[Louis Dionne via Boost]

> If there were several proposals, we could review all of them together. We
> already did that for the thread/futures proposals.

Ah, that's exactly the kind of thing I was looking for, thanks. So we've had
previous experience with this.

> * do we want a Hana style with "concepts" and with customizations? Do we
> want other data type than type lists? IIUC Peter's library works only
> with template aliases as data types and almost variadic class template
> with type parameters is a good candidate for a type list (even
> std::variant :( )

As far as I can tell, none of the proposals have Hana-style customization
points and concepts, but I see no reason to explicitly request that they
provide them. I'm happy as long as there's some way to interoperating with
other libraries.

> However I see only one official proposal (mp11 - Peter Dimov), even if
> there are other interesting libraries (as Meta, Brigand).

You're right. I thought Bruno Dutra's Metal had been officially submitted,
but not yet. In any case, it's just a matter of days or weeks.

Louis




--
View this message in context: http://boost.2283326.n4.nabble.com/About-all-these-metaprogramming-libraries-tp4692524p4692532.html

Sent from the Boost - Dev mailing list archive at Nabble.com.

[Peter Dimov via Boost]

Vicente J. Botet Escriba wrote:

> * do we want (can) to use as (HOMF) high-order meta-functions? if we nedd
> them at all). IIUC, Peter library doesn't works with HOMFs.
> * do we want a Hana style with "concepts" and with customizations? Do we
> want other data type than type lists? IIUC Peter's library works only
> with template aliases as data types and almost variadic class template
> with type parameters is a good candidate for a type list (even
> std::variant :( )

These two are the distinguishing features of mp11. All other libraries have
chosen differently. To clarify, the mp11 approach does support higher-order
metaprogramming, but it chooses to make the ordinary case easier at the
expense of the HOMF case, which is made harder, whereas the other libraries
choose to make HOMF easier, at the expense of taking quoted metafunctions as
arguments, which requires aliases to be quoted.

mp11, in contrast, requires quoted metafunctions to be de-quoted (Q::invoke)
when passed to algorithms.

To clarify further, std::variant<T...> is absolutely a supported typelist in
mp11. If you have a std::variant V, you can remove the duplicate types from
it with mp_unique<V>. This is again a deliberate design decision which keeps
simple uses simple at the expense of making more complicated uses more
complicated. With the other libraries, to remove the duplicates from a
variant, you first have to convert it to the native typelist type, apply
'unique', then convert back to std::variant.

[Edouard via Boost]

Hi Peter,

>To clarify further, std::variant<T...> is absolutely a supported typelist in mp11. If you have a std::variant V, you can remove the duplicate types from it with mp_unique<V>. This is again a deliberate design decision which keeps simple uses simple at the expense of making more complicated uses more >complicated. With the other libraries, to remove the duplicates from a variant, you first have to convert it to the native typelist type, apply 'unique', then convert back to std::variant.

We took the same approach. We have a native type list for convenience, but we also thought there was no reason to prevent direct manipulation of any kind of typelist. We also think it's best to keep the simple use cases straightfoward.

Kind regards.

-Edouard

[Andrzej Krzemienski via Boost]

2017-03-18 22:49 GMT+01:00 Vicente J. Botet Escriba via Boost <
bo...@lists.boost.org>:

Maybe, during the process, to get the additional insight, we sould request
of each of the candidates to compare the librearies with one another. They
are experts in the domain, and should be able to quickly identify different
tradeoffs taken by deiifferent libraries.

Regards,
&rzej;

[Vicente J. Botet Escriba via Boost]

Le 18/03/2017 à 23:08, Peter Dimov via Boost a écrit :
> Vicente J. Botet Escriba wrote:
>
>> * do we want (can) to use as (HOMF) high-order meta-functions? if we
>> nedd them at all). IIUC, Peter library doesn't works with HOMFs.
>> * do we want a Hana style with "concepts" and with customizations? Do
>> we want other data type than type lists? IIUC Peter's library works
>> only with template aliases as data types and almost variadic class
>> template with type parameters is a good candidate for a type list
>> (even std::variant :( )
>
> These two are the distinguishing features of mp11. All other libraries
> have chosen differently. To clarify, the mp11 approach does support
> higher-order metaprogramming, but it chooses to make the ordinary case
> easier at the expense of the HOMF case, which is made harder, whereas
> the other libraries choose to make HOMF easier, at the expense of
> taking quoted metafunctions as arguments, which requires aliases to be
> quoted.
>
> mp11, in contrast, requires quoted metafunctions to be de-quoted
> (Q::invoke) when passed to algorithms.
>

I have the impression that mp11 cannot use the meta function returned by
a function as a parameter of an algorithm, but I'm missing something
that it should be evident to you. How can we define a compose
meta-function that could be used later as parameter?

algo<dt, compose<mf1, mf2>>

> To clarify further, std::variant<T...> is absolutely a supported
> typelist in mp11. If you have a std::variant V, you can remove the
> duplicate types from it with mp_unique<V>. This is again a deliberate
> design decision which keeps simple uses simple at the expense of
> making more complicated uses more complicated. With the other
> libraries, to remove the duplicates from a variant, you first have to
> convert it to the native typelist type, apply 'unique', then convert
> back to std::variant.
>

I know. I just find it weird to take std::variant as a data type that
models type list. I agree with you that given the definition you have
used in mp11 (it is also the case of Brigand), std::variant models a
mp11 type list.
I'm not saying that it is not useful and I suspect that it should reduce
the compile time.

My main concern is that Boost.Hana has more data types than type list
and that the algorithms are associated to concepts. IIUC, mp11 has only
one concept (variadic template) and all the algorithms we can define on
this very rich data structure. Maybe you are right and this is all we
need. At a first glance I would prefer a meta library that is based on
the design of Hana. I'm not saying here that I want the mixity of type
and values.
Of course, if the compile times are very different I could change my
first impression.
Vicente

[Vicente J. Botet Escriba via Boost]

Le 18/03/2017 à 22:52, Louis Dionne via Boost a écrit :
>> If there were several proposals, we could review all of them together. We
>> already did that for the thread/futures proposals.
> Ah, that's exactly the kind of thing I was looking for, thanks. So we've had
> previous experience with this.

Great.

>
>
>> * do we want a Hana style with "concepts" and with customizations? Do we
>> want other data type than type lists? IIUC Peter's library works only
>> with template aliases as data types and almost variadic class template
>> with type parameters is a good candidate for a type list (even
>> std::variant :( )
> As far as I can tell, none of the proposals have Hana-style customization
> points and concepts, but I see no reason to explicitly request that they
> provide them. I'm happy as long as there's some way to interoperating with
> other libraries.

I'm not requesting that the meta libraries customize Hana concepts as
Boost.Hana has values.
I would like we discuss if we adapting those meta-concepts to the meta
world could be useful or not.

>
>> However I see only one official proposal (mp11 - Peter Dimov), even if
>> there are other interesting libraries (as Meta, Brigand).
> You're right. I thought Bruno Dutra's Metal had been officially submitted,
> but not yet. In any case, it's just a matter of days or weeks.

Who knows ;-)

Vicente

[Bruno Dutra via Boost]

On Sat, Mar 18, 2017 at 10:52 PM, Louis Dionne via Boost <
bo...@lists.boost.org> wrote:

>
> > If there were several proposals, we could review all of them together. We
> > already did that for the thread/futures proposals.
>
> Ah, that's exactly the kind of thing I was looking for, thanks. So we've
> had
> previous experience with this.
>

This is probably the most reasonable approach.

> > However I see only one official proposal (mp11 - Peter Dimov), even if
> > there are other interesting libraries (as Meta, Brigand).
>
> You're right. I thought Bruno Dutra's Metal had been officially submitted,
> but not yet. In any case, it's just a matter of days or weeks.
>

That is correct, I haven't formally submitted Metal yet, but that has been
in the works for the past couple of weeks and it is now a matter of days
like you said.

Bruno

[Bruno Dutra via Boost]

To add to that list, we also need to ask ourselves

* do we need SFINAE friendliness?
* do we want to use pattern matching to extract data from concepts?

> However I see only one official proposal (mp11 - Peter Dimov), even if
> there are other interesting libraries (as Meta, Brigand).

> <http://lists.boost.org/mailman/listinfo.cgi/boost>
>

The formal submission of Metal is just a matter of formality and will
happen in the following days.

Bruno

[Peter Dimov via Boost]

Vicente J. Botet Escriba wrote:

> I have the impression that mp11 cannot use the meta function returned by a
> function as a parameter of an algorithm, but I'm missing something that it
> should be evident to you. How can we define a compose meta-function that
> could be used later as parameter?

This is a metafunction in mp11:

template<class... T> using F = /*...*/;

This is a quoted metafunction:

struct Q
{
template<class... T> using invoke = /*...*/
};

Since quoted metafunctions are types, you can operate on them (although mp11
doesn't yet provide any built-in means to compose them), but to pass one to
an algorithm, you need

mp_transform<List, Q::invoke>

instead of just

mp_transform<List, F>

> I just find it weird to take std::variant as a data type that models type
> list.

The need to operate on the list of variant alternatives is actually pretty
common. Hence std::variant_size (which is just mp_size in mp11),
std::variant_alternative_t (mp_at_c). Similarly std::tuple_size,
std::tuple_element_t. This endless reinvention of the wheel is completely
unnecessary.

[Paul Fultz II via Boost]

> On Mar 18, 2017, at 2:32 PM, Louis Dionne via Boost <bo...@lists.boost.org> wrote:
>
> Dear Boost,
>
> We've seen two metaprogramming libraries ask for a formal review on the list
> recently. I expect a third (Brigand) might join the party soon, and maybe
> even a fourth (Eric Niebler's meta). I think it would be wise to have a plan
> for how we're going to deal with this.
>
> Before going further, I'd also like to clear something out. Some people have
> asked whether "classic" metaprogramming libraries still had their place
> given that we now have Hana.

Of course, there is a place for portability. Hana doesn’t work with gcc 4.8 nor gcc 5 nor MSVC. So if a library would like to support those compilers and utilize a library to handle metaprogramming or heterogeneous sequences, it would be nice to have an alternative modern solution instead of having to continue to use Fusion or MPL.

> As the author of Hana, I think the answer is
> yes. While Hana is (IMHO) easier to use and more powerful, it also has an
> important shortcoming. Hana is too slow when dealing with very large inputs.
> This is due to the fact that it can also handle values, which is much
> heavier than dealing with types only. This can be alleviated to some extent,
> but the truth is that our execution model (the compiler) simply has to do
> more work when handling values.
>
> Hence, I think Boost does need a modern classic TMP library to handle these
> cases where a library writer needs to handle gigantic type lists. I think
> marketing such a library as an advanced tool for library writers would be a
> service to the overall C++ community, but that's an orthogonal concern.
>
> However, Boost needs one such library, not four. I think we can't just do 4
> reviews and include all the libraries that pass in Boost; that would be a
> huge disservice to our users, who will be left with the burden of choosing.
> Heck, if we can't even make our mind, how can they?
>
> So, how do we pick? Have we ever been in a similar situation where we have
> multiple competing libraries solving _exactly_ the same problem?

It would be nice to see better collaboration between authors to build an unifying metaprogramming library, rather than competing against each other. Of course, this can be somewhat difficult as they have different fundamental concepts.

Paul

[Peter Dimov via Boost]

Paul Fultz II wrote:

> It would be nice to see better collaboration between authors to build an
> unifying metaprogramming library, rather than competing against each
> other. Of course, this can be somewhat difficult as they have different
> fundamental concepts.

As I see it, it's the fundamental concepts competing against each other, and
that's how it should be; how else we'll find out which approach is best?

[Emil Dotchevski via Boost]

On Sat, Mar 18, 2017 at 1:32 PM, Louis Dionne via Boost <
bo...@lists.boost.org> wrote:

> However, Boost needs one such library, not four. I think we can't just do 4
> reviews and include all the libraries that pass in Boost; that would be a
> huge disservice to our users, who will be left with the burden of choosing.
> Heck, if we can't even make our mind, how can they?
>

These questions are important only under the assumption that the libraries
are interchangeable. I doubt that they are. More likely their authors have
made different design choices, probably even competing design choices.

You can say well, as long one library can do the job of another, we don't
need both. But what if one library is faster in one use case and 10 times
slower in another? What if one library is always slower but can be used in
cases when another library can not?

There are already precedents in Boost of different libraries with
overlapping domains and this is not a problem. Moreover, it is in the
spirit of C++ to allow foor multiple solutions to a single problem, and to
provide just the solution to obscure problems 99% of the C++ programmers
didn't know existed.

Emil

[Vladimir Batov via Boost]

On 2017-03-19 12:02, Peter Dimov via Boost wrote:
> Paul Fultz II wrote:
>
>> It would be nice to see better collaboration between authors to build
>> an unifying metaprogramming library, rather than competing against
>> each other. Of course, this can be somewhat difficult as they have
>> different fundamental concepts.
>
> As I see it, it's the fundamental concepts competing against each
> other, and that's how it should be; how else we'll find out which
> approach is best?

Would not that be better to contain the battle among the gurus, the lib
developers. So that they are forced to cooperate, to consider
alternatives, to come up with one best (in their collective opinion)
solution... and leave the user out of it? Otherwise users are forced to
consider 4 solutions, to evaluate those and to decide (potentially
incorrectly) which one to use. As a user I personally do not have the
resources and the knowledge to do that. Taking/choosing a Boost lib I've
always known that there was a lot of effort spent evaluating,
optimizing, improving, etc. the lib. So that I get the best I can get.

[Emil Dotchevski via Boost]

On Sat, Mar 18, 2017 at 7:48 PM, Vladimir Batov via Boost <
bo...@lists.boost.org> wrote:

> On 2017-03-19 12:02, Peter Dimov via Boost wrote:
>
>> Paul Fultz II wrote:
>>
>> It would be nice to see better collaboration between authors to build an
>>> unifying metaprogramming library, rather than competing against each other.
>>> Of course, this can be somewhat difficult as they have different
>>> fundamental concepts.
>>>
>>
>> As I see it, it's the fundamental concepts competing against each
>> other, and that's how it should be; how else we'll find out which
>> approach is best?
>>
>
> Would not that be better to contain the battle among the gurus, the lib
> developers. So that they are forced to cooperate, to consider alternatives,
> to come up with one best (in their collective opinion) solution...

There is a big difference between "best" and "best in their collective
opinion".

Also, one library may be best in one thing and suck at something else.

Emil

[Michael Caisse via Boost]

On 3/18/17 19:48, Vladimir Batov via Boost wrote:

>
> Would not that be better to contain the battle among the gurus, the lib
> developers. So that they are forced to cooperate, to consider
> alternatives, to come up with one best (in their collective opinion)
> solution... and leave the user out of it? Otherwise users are forced to
> consider 4 solutions, to evaluate those and to decide (potentially
> incorrectly) which one to use. As a user I personally do not have the
> resources and the knowledge to do that. Taking/choosing a Boost lib I've
> always known that there was a lot of effort spent evaluating,
> optimizing, improving, etc. the lib. So that I get the best I can get.
>

Users of these libraries are often library developers. Many of us have
already implemented smallish TMP libraries that we use internally. I
think the Boost community is the right community to review just this
sort of library. Abusing compilers and meta programming rich libraries
is what we are known for (for good or bad).

I expect a combined review to tease out the best design choices and
implementations and may even discover better solutions yet-to-be-discovered.

micahel

--
Michael Caisse
Ciere Consulting
ciere.com

[Vladimir Batov via Boost]

On 2017-03-19 16:42, Michael Caisse via Boost wrote:
> ...

> Users of these libraries are often library developers. Many of us have
> already implemented smallish TMP libraries that we use internally. I
> think the Boost community is the right community to review just this
> sort of library. Abusing compilers and meta programming rich libraries
> is what we are known for (for good or bad).
>
> I expect a combined review to tease out the best design choices and
> implementations and may even discover better solutions
> yet-to-be-discovered.

Probably... If you mean one review of a combined effort... essentially
one (two?) consolidated library... maybe with currently separate libs
made sections/parts of such a combined effort... and duplicate parts
trimmed. All that trimming and negotiations and arguments IMO are better
done backstage... maybe only with some bits and pieces taken to the
list.

Otherwise, if you mean several simultaneous reviews (as they are
currently conducted) of competing functionally-overlapping libs, I
suspect it to result in an overload, a lot of very public fighting and
ultimately a stalemate. As always it is just my opinion. I can be dead
wrong. I often am.

[Niall Douglas via Boost]

> However, Boost needs one such library, not four. I think we can't just do 4
> reviews and include all the libraries that pass in Boost; that would be a
> huge disservice to our users, who will be left with the burden of choosing.
> Heck, if we can't even make our mind, how can they?
>
> So, how do we pick? Have we ever been in a similar situation where we have
> multiple competing libraries solving _exactly_ the same problem?

Can I ask a stupid question?

If Hana is the slow, "full fat" metaprogramming library, is there space
remaining for exactly one fast, "lightweight" metaprogramming library?

You seem to assume this de facto to be the case, but I'd like to know why?

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

[degski via Boost]

On 18 March 2017 at 14:32, Louis Dionne via Boost <bo...@lists.boost.org>
wrote:

> However, Boost needs one such library, not four.

Maybe, maybe not. What would help though if there would be a (ONE) document
outlining what the differences are and in what domain/context one should
prefer on library over the other, complemented with pros and cons of each
lib. This (TMP) is not simple stuff unless one is heavilly involved (are a
library writer). An 'average' boost user is not capable within reasonable
time and with reasonable effort to make an informed choice from amongst all
these similar libraries. I must say, I like Peters' 'simple' mp11 though,
providing building blocks to more advanced stuff (I would say (generally)
the STL-spirit).

degski

[Bruno Dutra via Boost]

On Sun, Mar 19, 2017 at 4:22 PM, degski via Boost <bo...@lists.boost.org>
wrote:

> I must say, I like Peters' 'simple' mp11 though,
> providing building blocks to more advanced stuff (I would say (generally)
> the STL-spirit).
>

I agree being simple is a plus, but IMO there is nothing exceptionally
simple about mp11.
Metal was designed to be as a simple building block as possible as well
[1], the only difference is that it approaches this goal in different way.

[1]: http://brunocodutra.github.io/metal/#concepts

[Loïc Joly via Boost]

Le 19/03/2017 à 16:22, degski via Boost a écrit :
> On 18 March 2017 at 14:32, Louis Dionne via Boost <bo...@lists.boost.org>
> wrote:
>
>> However, Boost needs one such library, not four.
>
> Maybe, maybe not. What would help though if there would be a (ONE) document
> outlining what the differences are and in what domain/context one should
> prefer on library over the other, complemented with pros and cons of each
> lib.

Including Hana.

The best would be only one library for all TMP needs, but if there is
more than one, such a document is of the essence.

---
Loïc

[Oswin Krause via Boost]

On 2017-03-19 17:13, Bruno Dutra via Boost wrote:
> On Sun, Mar 19, 2017 at 4:22 PM, degski via Boost
> <bo...@lists.boost.org>
> wrote:
>
>> I must say, I like Peters' 'simple' mp11 though,
>> providing building blocks to more advanced stuff (I would say
>> (generally)
>> the STL-spirit).
>>
>
> I agree being simple is a plus, but IMO there is nothing exceptionally
> simple about mp11.

Hi,

I would disagree. Conceptionally it is a lot easier to grasp as
everything is defined either in language primitives or logical
constraints.

Here are the concepts:

A list is any type of the form L<T...>.
A set is a list where all elements are unique
A map is a set of keys zipped with an arbitrary list of same size.
A metafunction is a template alias with unspecified template parameters.

The definition of the concepts do not need to reference any primitive
inside the library and thus it can work well with many other of the
proposed libraries and libraries outside of boost, without having to
jump through additional hoops (e.g. renaming/"copying" arguments).

For me the biggest question is: will it work natively with mpl::vector?
This would make it a clear winner in my book.

[Peter Dimov via Boost]

Oswin Krause wrote:

> For me the biggest question is: will it work natively with mpl::vector?

Not very well at the moment; mpl::vector<int, void, float> is a list of
fixed size 20, whereas mpl::vector<int, void, float>::type is of size 3, but
still fixed. It will only work with algorithms that don't need to change the
size of the list.

It would be possible in principle to make the algorithms take mpl::vector
and return mp_list, I suppose.

[Bruno Dutra via Boost]

On Sun, Mar 19, 2017 at 7:43 PM, Oswin Krause <
Oswin....@ruhr-uni-bochum.de> wrote:

> On 2017-03-19 17:13, Bruno Dutra via Boost wrote:
>
>> On Sun, Mar 19, 2017 at 4:22 PM, degski via Boost <bo...@lists.boost.org>
>> wrote:
>>
>> I must say, I like Peters' 'simple' mp11 though,
>>> providing building blocks to more advanced stuff (I would say (generally)
>>> the STL-spirit).
>>>
>>>
>> I agree being simple is a plus, but IMO there is nothing exceptionally
>> simple about mp11.
>>
>
> Hi,
>
> I would disagree. Conceptionally it is a lot easier to grasp as everything
> is defined either in language primitives or logical constraints.
>
> Here are the concepts:
>
> A list is any type of the form L<T...>.
> A set is a list where all elements are unique
> A map is a set of keys zipped with an arbitrary list of same size.
> A metafunction is a template alias with unspecified template parameters.
>
> The definition of the concepts do not need to reference any primitive
> inside the library and thus it can work well with many other of the
> proposed libraries and libraries outside of boost, without having to jump
> through additional hoops (e.g. renaming/"copying" arguments).
>

You've pretty much described Metal, except that Metal requires lists to be
specializations of metal::list, but that is far from a shortcoming, in fact
there is a very good reason behind that design choice.

What do you expect to happen if you try to erase a value from a template,
such as std::unique_ptr, that has a default parameter using mp11? What
about inserting an element? Templates with default parameters are valid
lists if the template that represents a List is not specified, so
algorithms must be well defined for all these cases, which is very
cumbersome to handle on the library side for very little gain, not to
mention that it often leads to surprising and unexpected behavior.

For me the biggest question is: will it work natively with mpl::vector?
> This would make it a clear winner in my book.
>

Working natively with mpl::vector is only possible through adaptors, either
implicit in the library, or explicit to the user, because the type of the
concept behind mpl::vector is unspecified by design. Personally I think
there is very little gain in embedding such adaptors implicitly in any
library, because they bring along a lot of legacy and make the life of
maintainers miserable for basically no gain, since a explicit adaptor, such
as from_mpl<>, has zero cost to the user.

My choice for Metal was to provide metal::from_mpl, a metafunction that
converts any MPL concept to its equivalent in Metal parlance.

Bruno

[Bruno Dutra via Boost]

On Sun, Mar 19, 2017 at 8:09 PM, Peter Dimov via Boost <
bo...@lists.boost.org> wrote:

> Oswin Krause wrote:
>
> For me the biggest question is: will it work natively with mpl::vector?
>>
>
> Not very well at the moment; mpl::vector<int, void, float> is a list of
> fixed size 20, whereas mpl::vector<int, void, float>::type is of size 3,
> but still fixed. It will only work with algorithms that don't need to
> change the size of the list.
>
> It would be possible in principle to make the algorithms take mpl::vector
> and return mp_list, I suppose.
>

You forgot to take into account that algorithms such as mpl::insert and
mpl::erase return proxies that inherit from vectors and don't expose the
elements in their types signatures, while being themselves valid instances
of a Vector. This is the real deal breaker.

[Louis Dionne via Boost]

> If Hana is the slow, "full fat" metaprogramming library,

Hana is not a "slow, full fat" metaprogramming library. Quite far from that.
It's just that for pure type-level metaprogramming (which by the way is
quite rarely needed nowadays with deduced return types), we can't possibly
be as fast as Brigand, Metal or such type-level only solutions that is
highly optimized for that precise use case.

> is there space remaining for exactly one fast, "lightweight"
> metaprogramming library?

The problem is that the term metaprogramming is overloaded to mean several
things. It can either mean pure type-level metaprogramming (MPL world), or
algorithms on tuples (Fusion world). It HAPPENS that the latter is powerful
enough to perform the former; this is what Hana allows and then you only
need one metaprogramming library. However, that is heavier compile-time wise
and does not scale as well as a library that was crafter with type
computations only from the start. And if that was your question, I don't
think you can do what Hana does (i.e. handle Fusion + MPL) while being
significantly faster. You might be able to be some small constant factor
faster, but that's it.

By the way, these libraries for pure type-level metaprogramming have the
same problem, it's just that their constant factors are smaller than Hana's.
Take any of these libraries and try to process a type list with 200K
elements while doing something non-trivial. The fundamental problem is that
we need language-level support for metaprogramming if we want to go
completely crazy with it, and some of us are working exactly on this right
now for C++next (or maybe the one after).

Louis




--
View this message in context: http://boost.2283326.n4.nabble.com/About-all-these-metaprogramming-libraries-tp4692524p4692599.html

Sent from the Boost - Dev mailing list archive at Nabble.com.

[Paul Fultz II via Boost]

> On Mar 18, 2017, at 7:02 PM, Peter Dimov via Boost <bo...@lists.boost.org> wrote:
>
> Paul Fultz II wrote:
>
>> It would be nice to see better collaboration between authors to build an unifying metaprogramming library, rather than competing against each other. Of course, this can be somewhat difficult as they have different fundamental concepts.
>
> As I see it, it's the fundamental concepts competing against each other, and that's how it should be; how else we'll find out which approach is best?

Well brigand and mpl11 seem very similar conceptually. There is a conceptual difference with Metal where everything is explicit. However, I don’t think we have to choose between a implicit and explicit API. Rather a library could provide both APIs, much like Boost.PP provides an API for both explicit and implicit(ie deduced) re-entrance.

Furthermore, the simple API of mpl11 very likely could be implemented using Metal underneath. Perhaps instead of providing two different libraries we provide one library with metal::explicit_ and metal::implicit where the implicit API could follow mpl11 very closely(if thats what seems to be the best approach to the implicit part considering other APIs).

Paul

[Peter Dimov via Boost]

Bruno Dutra wrote:
> > It would be possible in principle to make the algorithms take
> > mpl::vector and return mp_list, I suppose.
>
> You forgot to take into account that algorithms such as mpl::insert and
> mpl::erase return proxies that inherit from vectors and don't expose the
> elements in their types signatures, while being themselves valid instances
> of a Vector. This is the real deal breaker.

On MSVC, insert/erase return numbered vectors, but on g++ and clang++,
you're right, they return some weird v_item sequences.

What is our use case here though? From where would those mpl types come?

[Bruno Dutra via Boost]

On Sun, Mar 19, 2017 at 9:29 PM, Peter Dimov via Boost <
bo...@lists.boost.org> wrote:

> Bruno Dutra wrote:
>
>> > It would be possible in principle to make the algorithms take >
>> mpl::vector and return mp_list, I suppose.
>>
>> You forgot to take into account that algorithms such as mpl::insert and
>> mpl::erase return proxies that inherit from vectors and don't expose the
>> elements in their types signatures, while being themselves valid instances
>> of a Vector. This is the real deal breaker.
>>
>
> On MSVC, insert/erase return numbered vectors, but on g++ and clang++,
> you're right, they return some weird v_item sequences.
>
> What is our use case here though? From where would those mpl types come?
>

I don't have a use case, I'm just trying to argue against the idea of
implementing any implicit native support for MPL sequences, which besides
non-trivial is also fruitless.

Moreover, providing support for numbered instances of mpl::vector only and
not for its proxies would really be a bad idea, because it doesn't solve
the common use case, where the user has some legacy metaprogram that
manipulates mpl::vectors, certainly also through insert and erase, and
wants to pass this on to the algorithms of a new library that advertises
itself as interoperating with MPL natively, only to realize the hard way
that it doesn't really.

I stand by the opinion that it can't go wrong with explicit converters.

Bruno

[Michael Caisse via Boost]

On 3/19/17 12:31, Bruno Dutra via Boost wrote:
> On Sun, Mar 19, 2017 at 7:43 PM, Oswin Krause <
> Oswin....@ruhr-uni-bochum.de> wrote:
>
>> On 2017-03-19 17:13, Bruno Dutra via Boost wrote:
>>
>>> On Sun, Mar 19, 2017 at 4:22 PM, degski via Boost <bo...@lists.boost.org>
>>> wrote:

<snip features and criticisms of various libraries>

>
> You've pretty much described Metal, except that Metal requires lists to be
> specializations of metal::list, but that is far from a shortcoming, in fact
> there is a very good reason behind that design choice.
>

<snip more things why-I-designed-my-library-this-way>

Here is a suggestions. I don't think this thread is the place for the
authors to explain the benefits of their design/implementation/usage.
There will be a lot of time for that.

In this thread, we are trying (as a community) to figure out the best
approach to deal with multiple submissions of TMP libraries. Lets
concentrate on that concern.

Don't worry ... there will be plenty of opportunity for critique and
defense (o;

michael

--
Michael Caisse
Ciere Consulting
ciere.com

[Niall Douglas via Boost]

On 19/03/2017 19:57, Louis Dionne via Boost wrote:
>
>> If Hana is the slow, "full fat" metaprogramming library,
>
> Hana is not a "slow, full fat" metaprogramming library. Quite far from that.
> It's just that for pure type-level metaprogramming (which by the way is
> quite rarely needed nowadays with deduced return types), we can't possibly
> be as fast as Brigand, Metal or such type-level only solutions that is
> highly optimized for that precise use case.

Surely you forgot to append the condition "we can't possibly be as fast
as ... *with current compilers*"?

>> is there space remaining for exactly one fast, "lightweight"
>> metaprogramming library?
>
> The problem is that the term metaprogramming is overloaded to mean several
> things. It can either mean pure type-level metaprogramming (MPL world), or
> algorithms on tuples (Fusion world). It HAPPENS that the latter is powerful
> enough to perform the former; this is what Hana allows and then you only
> need one metaprogramming library. However, that is heavier compile-time wise
> and does not scale as well as a library that was crafter with type
> computations only from the start. And if that was your question, I don't
> think you can do what Hana does (i.e. handle Fusion + MPL) while being
> significantly faster. You might be able to be some small constant factor
> faster, but that's it.

Again, Hana is slower than these other libraries because *current
compilers* have been heavily optimised for type based metaprogramming,
and not yet for deduced return type metaprogramming.

But this won't be the situation soon. The Visual Studio team are
actively targeting MSVC at Hana. Knowing them, even after it's compiling
100% clean they'll then go on to make MSVC *fast* with Hana. I don't
doubt the clang guys are also thinking the same thing, that a Hana based
benchmark could be real useful to shining a spotlight on code which is
currently highly suboptimal in their compilers.

(As an aside, a Hana based compiler performance benchmark would be
awesome, you should write one with all that free time you have :) and
put it online with graphs or something)

> By the way, these libraries for pure type-level metaprogramming have the
> same problem, it's just that their constant factors are smaller than Hana's.
> Take any of these libraries and try to process a type list with 200K
> elements while doing something non-trivial. The fundamental problem is that
> we need language-level support for metaprogramming if we want to go
> completely crazy with it, and some of us are working exactly on this right
> now for C++next (or maybe the one after).

So I've got to ask the question: Are we not jumping the gun here with
adding any metaprogramming library to Boost beyond Hana? If we wait a
few years, the performance gap between Hana and any of these other
libraries will probably close very significantly to the point of
inconsequence.

Let me put this another way: do any of the other metaprogramming
libraries add *in pure API or capability terms* anything over Hana?
Because if they add nothing new in terms of a better API, or able to do
extra stuff, there is a *strong* argument that we should just wait and
see what future compilers look like before adding more metaprogramming
libraries.

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

[Bruno Dutra via Boost]

On Mar 20, 2017 12:42 AM, "Niall .

Let me put this another way: do any of the other metaprogramming
libraries add *in pure API or capability terms* anything over Hana?

I don't mean to derail the discussion again, but since you asked, Metal
provides SFINAE-friendliness guarantees that Hana does not. Think
std::enable_if on steroids.

Bruno,

[Niall Douglas via Boost]

> Let me put this another way: do any of the other metaprogramming
> libraries add *in pure API or capability terms* anything over Hana?
>
>
> I don't mean to derail the discussion again, but since you asked, Metal
> provides SFINAE-friendliness guarantees that Hana does not. Think
> std::enable_if on steroids.

Thanks for the info.

However to continue to play devil's advocate, is that a highly
motivating reason for an additional metaprogramming library over Hana in
Boost when Concepts will ship in every major compiler this year or next?

If someone were proposing a metaprogramming library which allowed me to
write code which uses Concepts on a supporting compiler, but fell back
to a C++ 14 emulation on older compilers, that I'd find that a
compelling reason for an additional Boost metaprogramming library.

Niall

--
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/

[Deniz Bahadir via Boost]

Am 19.03.2017 um 23:00 schrieb Michael Caisse via Boost:
> On 3/19/17 12:31, Bruno Dutra via Boost wrote:
>> On Sun, Mar 19, 2017 at 7:43 PM, Oswin Krause <
>> Oswin....@ruhr-uni-bochum.de> wrote:
>>
>>> On 2017-03-19 17:13, Bruno Dutra via Boost wrote:
>>>
>>>> On Sun, Mar 19, 2017 at 4:22 PM, degski via Boost <bo...@lists.boost.org>
>>>> wrote:
>
> <snip features and criticisms of various libraries>
>
>>
>> You've pretty much described Metal, except that Metal requires lists to be
>> specializations of metal::list, but that is far from a shortcoming, in fact
>> there is a very good reason behind that design choice.
>>
>
> <snip more things why-I-designed-my-library-this-way>
>
> Here is a suggestions. I don't think this thread is the place for the
> authors to explain the benefits of their design/implementation/usage.
> There will be a lot of time for that.
>
> In this thread, we are trying (as a community) to figure out the best
> approach to deal with multiple submissions of TMP libraries. Lets
> concentrate on that concern.

As mentioned before, I think some kind of comparison between such TMP
libraries is required. A comparison between their features/concepts and
especially the users' needs they try to address with these
features/concepts.

For example, from a user's point of view I would pretty much like to see
a "drop-in" replacement for Boost.MPL which compiles (much) faster (and
uses less RAM doing so).
However, I am not really interested in using such a replacement by
myself. I am instead interested in all existing Boost libraries that
currently use Boost.MPL to "magically" become/compile faster (and use
less RAM).

Of course, if all Boost-libraries that use Boost.MPL are actively
maintained and their maintainers would apply internal changes so that
these libraries would use another (faster) TMP library (e.g. Boost.Hana
or what might be adopted next), that would be fine, too.
However, it probably is less work for the maintainer to only adjust some
includes and maybe some typedefs to use a "drop-in" replacement instead
of changing a lot of code to use another TMP library which is completely
different from Boost.MPL.

>
> Don't worry ... there will be plenty of opportunity for critique and
> defense (o;
>
> michael
>

Just my "user's view".

Deniz

[Peter Dimov via Boost]

Deniz Bahadir wrote:

> As mentioned before, I think some kind of comparison between such TMP
> libraries is required.

Error C6301: passive voice used: "is required"

Who should produce this comparison?

[Peter Koch Larsen via Boost]

On Mon, Mar 20, 2017 at 4:00 PM, Peter Dimov via Boost
<bo...@lists.boost.org> wrote:
>> As mentioned before, I think some kind of comparison between such TMP
>> libraries is required.
>
> Error C6301: passive voice used: "is required"
>
> Who should produce this comparison?

I agree with Deniz if we can rephrase his words slightly, changing
required to desirable.
I have myself some knowledge of MPL and a little bit of knowledge of
Brigand. But my knowledge is fragmented, having used only what I use
in my projects. It would be quite nice to have an overview of the
capabilities and differences of the different libraries. For metal and
brigand that should not be to difficult as I know that the authors
have followed both libraries.
For your library, I read your excellent posts on metaprogramming, but
I was unaware that it had become a public library before you submitted
it to boost.

/Peter

[Deniz Bahadir via Boost]

Am 20.03.2017 um 16:00 schrieb Peter Dimov via Boost:
> Deniz Bahadir wrote:
>
>> As mentioned before, I think some kind of comparison between such TMP
>> libraries is required.
>
> Error C6301: passive voice used: "is required"
>
> Who should produce this comparison?

The answer to that question is one of the goals of the current
discussion, if I understood Michael correctly. ;-)

But jokes aside:

I would recommend, the authors of the different TMP libraries create a
short list of the concepts/features used and the problems and use-cases
they want to address with their libraries. (Probably, such a list should
also be available for Boost.Hana and possibly Boost.MPL. Although I
doubt that anybody would be pleased to just continue using Boost.MPL
instead of one of the new TMP-libraries.)
Experienced/interested TMP developers/users could probably help doing so.

Additionally, users of TMP libraries (others or some of the
to-be-reviewed ones) should mention their use-cases or problems they
would like to see addressed. (E.g. as I did in my former email; hoping
for a "drop-in" replacement of Boost.MPL for other Boost libs.)

Checking which of the libraries addresses what needs might help to find
consensus in what really should and can be addressed and which of the
new libraries does the best job.
Maybe we will realize their might be two TMP libraries that best become
joined into a single one to address almost all of the user's needs? Then
such a joined TMP lib might be the best candidate to become reviewed and
adopted by Boost?

And just in case, you might ask me the same question. ;-)
The last step, the checking of addressed needs should probably be done
by all readers of the mailing-list that are interested in this topic via
discussion on this mailing-list.

As a result we might get a nice pre-selection for a review candidate and
as a side-effect gather more reviewers, because they already
participated in the pre-selection so that the review-barrier is/feels
lowered.

I hope that is not too unrealistic.
Deniz