Allowing string literals to match `char...` packs in templates

[Vittorio Romeo]

Currently string literals are not allowed as template arguments because the standard explicitly prevents that (see 14.3.2 Template non-type arguments).

...why?
A possible issue arises when passing the same literal twice - consider this case:

template<const char* TS> struct t{};
int main()
{
    t<"hi"> a{};
    t<"hi"> b{};
}
// Issue: does `TS` have the same address for `a` and `b`?

The problem seems to be that, since string literals really are pointers/references, there's no sensible way of dealing with them as compile-time template parameters.

But it would be really useful to have such a feature:

template<magic TS> struct tag{};
int main()
{
    database::table<tag<"user">,
        database::field<tag<"name">, std::string>,
        database::field<tag<"age">, int>
    > db_user;

    db_user.get<tag<"name">>(id(10));
    db_user.get<tag<"age">>(id(10));
}

Another example:

template <typename F, std::size_t ... I>
constexpr auto string_builder(F f, std::index_sequence<I...>)
{
  return char_sequence<f(I)...>{};
}

template<magic TS>
auto tag(TS s)
{
    return string_builder([]() constexpr { return s; }, magic_sizeof(s));
}

int main()
{
    static_assert(is_same(decltype(tag("hi")), decltype(tag("hi"))));
    static_assert(!is_same(decltype(tag("hi")), decltype(tag("bye"))));
}

The above examples show the definition of unique types based on strings in-situ without using macros. (It is already possible with macros.)
The problem is that there currently is nothing in the language that could be used as "magic".

Therefore I propose that, to avoid ambiguity, magic should simply be a char... pack.
A special rule would be added to the standard that allows string literals to be matched into char... packs as template parameters.

template<typename> struct t_typename{};
template<const char*> struct t_constchar{};
template<const char(&)[3]> struct t_charref{};
template<char...> struct t_chars{};
template<auto> struct t_auto{};
template<auto...> struct t_autos{};

int main()
{
    t_typename<"hi">{}; // still would FAIL to compile, as intended
    t_constchar<"hi">{}; // still would FAIL to compile, as intended
    t_charref<"hi">{}; // still would FAIL to compile, as intended
    t_chars<"hi">{}; // should COMPILE with the proposed changes
    t_auto<"hi">{}; // still would FAIL to compile, as intended
    t_autos<"hi">{}; // should COMPILE with the proposed changes
}

Using char... instead of pointers/reference would make the literal behave just a sequence of compile-time characters.

What do you think?
The same problem is addressed by P0424R0, but only for user-defined literals.

I think that allowing string literals to be matched into template char... argument packs would make sense, because the programmer only cares about the literal's "characters" (not its address) at compile-time.

[Matt Calabrese]

On Wed, Nov 16, 2016 at 9:04 AM, Vittorio Romeo
<vittorio....@gmail.com> wrote:
> Currently string literals are not allowed as template arguments because the
> standard explicitly prevents that (see 14.3.2 Template non-type arguments).

At the end of the Issaquah C++ standards meeting last week some of us
worked toward another approach to arrays as value-kind template
parameters (Louis Dione, Jeff Snyder, James Touton, myself, and
others), specifically because of wanting to easily pass compile-time
strings. It's a harder problem than it seems and implementors voiced
concern regarding representation as a pack of char even though the
desired semantics are similar. I'm not sure, but Louis Dionne might
already be starting on a new paper in this area. There are a few of us
who really want something like this.

[Louis Dionne]

Let me just expand a bit on what Matt said.

I think this is a brilliant idea, but I don't think it's going to work, for the same reason

as my P0424R0 paper was not warmly received. The reason is that implementers

don't want anything that encourages the use of large template parameter lists,

because the representation of that inside the compiler is too inefficient. They don't

want a single template parameter per character in the string, because the structure

that represents a template parameter in the compiler is expensive. Instead, they'd

rather have a single structure that represents all the characters. I'm not sure why they

can't just represent a character pack in a more efficient manner, but that's what I

was told when I presented in EWG.

So on Saturday, we tried looking into passing arrays as template arguments, but

that becomes messy because arrays are already supported, but they decay to

pointers so we'd need to support a new syntax. We then tried passing std::array,

but doing this cleanly would require passing arbitrary literal types as template

arguments, which is also non-trivial of a change.

I have not started writing a paper yet, because I don't have any solution in mind

at this time to be honest. However, this is one of the most valuable problems to

be solved for metaprogramming, so I'll take anything that works.

Louis

[TONGARI J]

On Thursday, November 17, 2016 at 1:28:26 PM UTC+8, Louis Dionne wrote:

Let me just expand a bit on what Matt said.

I think this is a brilliant idea, but I don't think it's going to work, for the same reason

as my P0424R0 paper was not warmly received. The reason is that implementers

don't want anything that encourages the use of large template parameter lists,

because the representation of that inside the compiler is too inefficient. They don't

want a single template parameter per character in the string, because the structure

that represents a template parameter in the compiler is expensive. Instead, they'd

rather have a single structure that represents all the characters. I'm not sure why they

can't just represent a character pack in a more efficient manner, but that's what I

was told when I presented in EWG.

So on Saturday, we tried looking into passing arrays as template arguments, but

that becomes messy because arrays are already supported, but they decay to

pointers so we'd need to support a new syntax. We then tried passing std::array,

but doing this cleanly would require passing arbitrary literal types as template

arguments, which is also non-trivial of a change.

I have not started writing a paper yet, because I don't have any solution in mind

at this time to be honest. However, this is one of the most valuable problems to

be solved for metaprogramming, so I'll take anything that works.

What about automatically generating types for string-literals?

Probably with a built-in literal operator 'lit' that does the compiler magic.

The generated type of a string-literal is guaranteed to be the same across TUs.

Such a type should provide some constexpr methods/typedefs to access its intrinsic (e.g. size(), operator[], char_type, etc).

For example:

constexpr auto str = "Hello"lit;
static_assert(is_same<decltype(str), decltype("Hello"lit)>::value);
static_assert(is_same<decltype(str)::char_type, char>::value);
static_assert(s[0] == 'H');

Has idea like this been discussed?

[Louis Dionne]

No, this has not been discussed AFAICT. That's not a bad idea, but it does seem very magic to me.

I don't know whether it's more desirable to introduce this special case or to "fix" the problem by

allowing user defined types with some restrictions as template arguments. Certainly the latter is

cleaner, but also much more difficult.

Louis

[TONGARI J]

On Thursday, November 17, 2016 at 3:23:00 PM UTC+8, Louis Dionne wrote:

No, this has not been discussed AFAICT. That's not a bad idea, but it does seem very magic to me.

I don't know whether it's more desirable to introduce this special case or to "fix" the problem by

allowing user defined types with some restrictions as template arguments. Certainly the latter is

cleaner, but also much more difficult.

It's only as magic as lambda expressions :)

The idea is very similar, the big difference is that lambda will give you different types even if the expressions are lexically the same, while this will give you the same type as long as the string-literals are the same.

Further more, we can define some useful operations to help string processing, e.g.

static_assert(is_same<decltype("Hello"lit + " World"lit), decltype("Hello World"lit)>::value);

This is certainly not as general as allowing UDTs as template arguments, but should solve a practical problem in compile-time string processing.

[Jonathan Müller]

Sorry if I'm asking the obvious question here: Why not guarantee that different string literals have different addresses and the same string literal has the same address and use const char* as non-type template parameters?

Jonathan

[Ville Voutilainen]

On 17 November 2016 at 10:13, Jonathan Müller

It's less of a problem of guaranteeing that different literals have
different addresses, and
more a problem of avoiding identical literals possibly having
different addresses. Therefore
it's better to have the template equivalence be determined by the
contents of the string
rather than the address.

[Louis Dionne]

Another problem is that "different" literals for address purposes could be the

same when doing a strcmp-style comparison. Consider:

    foo<"hello world">

    foo<"hello world\0abcd">

Should these two resolve to the same template? If we use the address of literals,

certainly not. If we compare the contents of the string, I guess that remains an

open issue but the two behaviors could be envisioned.

Louis

[Matt Calabrese]

Ville and Louis already covered a couple of reasons why this won't
work, but yet another is that not all string template arguments come
directly from literals. It's entirely possible that such strings will
be the result of compile-time string manipulations (and I predict that
this may even end up being fairly common, given metaprogramming).

[Tom Honermann]

I believe one of the complications here is systems that support shared
libraries. The opportunity to assign common addresses for equal string
literals doesn't arise until run-time.

Tom.

[Tom Honermann]

On 11/17/2016 02:10 AM, TONGARI J wrote:

What about automatically generating types for string-literals?

Probably with a built-in literal operator 'lit' that does the compiler magic.

I've given this a little thought.  My motivation comes from wanting to know the encoding of a string literal (e.g., implementation defined execution character set vs UTF-8).  Presumably, the generated/deduced type could record this information.  Such a mechanism would address the use cases that motivated P0482R0.

I think the primary difficulty with such an approach is addressing backward compatibility; in particular standard conversions and overload resolution.  The generated/deduced type would have to allow implicit conversion to an (lvalue) array of code unit type (and then allow array-to-pointer decay).  The potential difficulties with specifying such behavior are briefly discussed in http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0482r0.html#design_compat_core.

You're proposal would presumably break the following code unless special rules we're introduced to allow a different type deduction for deduced pointer types.

auto *p = "text";

Tom.

[Johannes Schaub]

Perhaps pass it not as a value, but as a (class-) type with a new kind
of type specifier. Then you can use a type template parameter, and
need to invent only syntax to denote these types. The type would have
a "value" static constexpr member that denotes the value. For string
literals, that would be a char array. Notice that this approach will
be compatible with passing std::integral_constant as a type, just that
the actual type will be some "magic" type. Using the "decltype"
keyword here, I'm making an example

template<typename Name>
struct Variable {
void print() { std::cout << Name::value; }
};

Variable<decltype<"Hello folks">> v;
v.print(); // prints "Hello folks" (without quotes)

Variable <decltype<42>> v42;
v42.print(); // prints 42

If we expose the type (bikeshed warning) as "std::constant", we could
also specialize on it

template<>
struct Variable<std::constant<int>> { };

I can imagine that with concepts, one could also specialize on the
contained value, although I'm not familiar with the exact syntax and
whether it actually could work

template<typename C> // C will be char[N]
requires constexprCompareFunction(C::value, "Hello folks")
struct Variable<std::constant<C>> { };

All kind of literals are allowed. Perhaps one can extend it to all
kinds of constant expressions even.

> --
> You received this message because you are subscribed to the Google Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d9433fac-152f-4515-b518-985a78768c03%40isocpp.org.

[Johannes Schaub]

And you need something to guarantee that decltype<0> is not the same
type as decltype<1> in order for the templates that receive the types
to be different for different constants ^_^ Perhaps you can make it a
hidden compiler type and the user could simply write the above
concepts-thingy as

template<> struct Variable<decltype<"Hello folks">> { };

To say "some string", you could catch the type type and require that
it's for a string-literal

template<typename C>
requires StringLiteralConstant<C>
struct Variable<C> { };

[Matt Calabrese]

On Thu, Nov 17, 2016 at 12:28 AM, Louis Dionne <ldio...@gmail.com> wrote:

So on Saturday, we tried looking into passing arrays as template arguments, but

that becomes messy because arrays are already supported, but they decay to

pointers so we'd need to support a new syntax. We then tried passing std::array,

but doing this cleanly would require passing arbitrary literal types as template

arguments, which is also non-trivial of a change.

FWIW, I think that either supporting std::array or supporting some kind of new built-in array type that doesn't decay (and is convertible from existing array types) is most likely the way we'd need to go. I'm not entirely convinced of the implication being that we'd need to support passing arbitrary literal types as template arguments in order to get one of these solutions -- we can just limit the element type of the array to being a type from the set of existing non-type template argument types that are also valid array element types. This seems to be a reasonable restriction and avoids all of the scary questions regarding support for literal types as non-type template parameter types.

One reason that I lean toward a new, language-level array type is that there are other problems with arrays that, in my opinion, hold the language back. Here's a brief, non-exhaustive list of issues with existing c-arrays:

* They decay

* They cannot have a 0-length extent

* There are all of the problems already mentioned in this thread regarding difficulty in supporting them as non-type template parameter types 

* If we hope for the language to ever support size-0 types, I suggest that c-style arrays would be one of the main barriers (more on this later)

* They are neither copyable nor movable

* They are not comparable

From that list, I consider the last two to be things that actually might be able to be fixed with respect to c-style arrays without introducing a new type. I may write a paper on this, specifically, but the other issues remain.

std::array fixes some of these problems but introduces others:

* Multi-dimensional arrays are not directly supported (yet) and std::array of std::array is a hairy alternative

* No simple way to dynamically allocate a std::array of N elements, where N is only known at run-time

* Template instantiations :|

* It's a library solution for "fixing" a legacy language issue (I'm usually for library solutions to problems rather than language, but not when the library is the preferred alternative to a built-in language facility)

* Special-casing support for std::array as a valid non-type template argument would be somewhat weird and puts an... interesting dependency on the library from the language.

As much as I like std::array for everything it fixes, I really think that we should, if at all possible, have a language-level array type that's simply not broken. I just suspect that it would be difficult to get consensus since we'd likely never be able to get rid of existing c-style arrays and having two fundamental array kinds would be weird. It would also mean that there would be no less than 3 ways to create an array in standard C++ (c-style, std::array, and the hypothetical new type). 

Earlier in this post I mentioned that existing c-style arrays may be a barrier for the language eventually supporting size-0 types. Hopefully not derailing the thread too much, but to elaborate on that, c-style arrays have the requirement of pointer-to-element-type being the corresponding iterator type of the array. That is to say, given an array T array[N], [array + 0, array+N) is a valid range over N elements, with T* being the iterator type. Sean Parent and others have been vocal about the desire for size-0 types, but, unfortunately, I see that as very difficult when it comes to existing assumptions with respect to arrays. Imagine, for instance, that "T" here is a hypothetical size 0 type. This means that "array" should also be size 0. This also means that [array + 0, array + N), with iterator type T*, no longer forms a range of length N. It actually forms a range of length 0. This means that any generic code that depends on T* being an appropriate iterator type may break when T happens to be a size 0 type. Other languages that support size 0 types, such as Rust, don't have this problem because their array iterators aren't required to be pointers. In the size 0 element case, the iterator type instead contains an index and so you can form proper ranges. IMO, having an array type in C++ that didn't require pointers as the iterator type would be a step in a positive direction.

There are other problems with size 0 types, such as that for desired effects, you'd want "distinct" size-0 objects to be able to have the same address (the array case is just one example of this). I consider this to be less of an issue in practice.

Anyway, the overall point is that time and time again we run into c-style arrays being a barrier to progress. std::array is a great alternative, but I don't think we should discount trying to have a new built-in array type that is "compatible" with existing c-style arrays (can extract a pointer or reference to c-style array given a new-style array in cases where it is feasible). This could potentially make the array template-parameter-kind more feasible along with solving the other existing issues with built-in arrays.

[Thiago Macieira]

But everything that depended on those literals would also be resolved at
runtime only (vague linkage).

(I'm not considering ODR-violating options like -fvisibility-inlines-hidden)

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Tom Honermann]

On 11/17/2016 5:58 PM, Thiago Macieira wrote:
> On quinta-feira, 17 de novembro de 2016 10:18:22 PST Tom Honermann wrote:
>> On 11/17/2016 03:13 AM, Jonathan Müller wrote:
>>> Sorry if I'm asking the obvious question here: Why not guarantee that
>>> different string literals have different addresses and the same string
>>> literal has the same address and use const char* as non-type template
>>> parameters?
>> I believe one of the complications here is systems that support shared
>> libraries. The opportunity to assign common addresses for equal string
>> literals doesn't arise until run-time.
> But everything that depended on those literals would also be resolved at
> runtime only (vague linkage).

Vague linkage is not supported by all compilers/linkers, at least not
when shared libraries are involved.

Consider the following example compiled with MSVC 2013:

$ cat t.h
extern inline const char* f() {
return "text";
}

$ cat dll1.cpp
#include "t.h"
__declspec(dllexport) const char* dll1() {
return f();
}

$ cat dll2.cpp
#include "t.h"
__declspec(dllexport) const char* dll2() {
return f();
}

$ cat main.cpp
#include <cassert>
__declspec(dllimport) const char* dll1();
__declspec(dllimport) const char* dll2();
int main() {
assert(dll1() == dll2());
}

$ cl /LD dll1.cpp /Fedll1.dll
...

$ cl /LD dll2.cpp /Fedll2.dll
...

$ cl main.cpp /Femain.exe dll1.lib dll2.lib
...

$ ./main
Assertion failed: dll1() == dll2(), file main.cpp, line 5

Since f() is an inline function, the returned string literal is required
to have the same address in all TUs:

C++14 [dcl.fct.spec] Function specifiers p4:
...
A string literal in the body of an extern inline function is the same
object in different translation units.
...

However, Microsoft's compiler/linker does not implement this requirement
across shared library boundaries. Since shared libraries are not
described by the standard, it is arguable whether this is a
non-conformance issue or not.

If the above code is compiled into a single executable on Windows, the
assert does not fail.

Tom.

[col...@yahoo.co.uk]

How about allowing literal types as template arguments if they have

defaulted == and != operators (and this condition also applies recursively

to all of their members). It does look fairly likely that we will get default ==

in the language in the near future. Once we have that, it would provide

a natural way to express which types are unproblematic for use as

template arguments.

This would be more than good enough to get us arrays of characters.

[Matt Calabrese]

On Fri, Nov 18, 2016 at 11:30 AM, col3435 via ISO C++ Standard - Future Proposals <std-pr...@isocpp.org> wrote:

How about allowing literal types as template arguments if they have

defaulted == and != operators (and this condition also applies recursively

to all of their members). It does look fairly likely that we will get default ==

in the language in the near future. Once we have that, it would provide

a natural way to express which types are unproblematic for use as

template arguments.

This would be more than good enough to get us arrays of characters.

This has came up independently by several of us. IMO, it's the most reasonable way to support literal types as template arguments, if we ever support them, but not everyone agrees. I think Faisal Vali and Andrew Sutton were the most recent people pursuing literal types as template parameter types (Faisal mentioned it to me only a few months ago). I'm not sure what their approach will be, assuming they are still working on that proposal.

[Tom Honermann]

On 11/18/2016 11:30 AM, col3435 via ISO C++ Standard - Future Proposals
wrote:

> How about allowing literal types as template arguments if they have
> defaulted == and != operators (and this condition also applies recursively
> to all of their members). It does look fairly likely that we will get
> default ==
> in the language in the near future. Once we have that, it would provide
> a natural way to express which types are unproblematic for use as
> template arguments.
>
> This would be more than good enough to get us arrays of characters.

For mangling purposes, I would expect a more strict requirement to be
necessary. Probably that all non-static data members are types (or
arrays thereof) that are acceptable as non-type template arguments.

Note that literal types include void, so, unless the regular void
proposal is accepted, I think a term other than "literal type" is needed
to describe the acceptable types here.

Tom.

[T. C.]

On Friday, November 18, 2016 at 11:19:17 AM UTC-5, Tom Honermann wrote:

Since f() is an inline function, the returned string literal is required
to have the same address in all TUs:

C++14 [dcl.fct.spec] Function specifiers p4:
...
A string literal in the body of an extern inline function is the same
object in different translation units.
...

This requirement has been removed by core issue 1823. 

[Tom Honermann]

Interesting, I was unaware.  This sounds like additional evidence that we cannot rely on string literals with matching contents having the same address across TUs.

The point regarding vague linkage still stands though; the example still fails the assertion if the inline function is modified to return the address of a static local variable.  The WP still requires static local variables in inline functions to be the same object across TUs (http://eel.is/c++draft/dcl.inline)

$ cat t.h
extern inline const char* f() {

  static char c = 't';
  return &c;
}

Tom.

[inkwizyt...@gmail.com]

On Thursday, November 17, 2016 at 6:28:26 AM UTC+1, Louis Dionne wrote:

Let me just expand a bit on what Matt said.

I think this is a brilliant idea, but I don't think it's going to work, for the same reason

as my P0424R0 paper was not warmly received. The reason is that implementers

don't want anything that encourages the use of large template parameter lists,

because the representation of that inside the compiler is too inefficient. They don't

want a single template parameter per character in the string, because the structure

that represents a template parameter in the compiler is expensive. Instead, they'd

rather have a single structure that represents all the characters. I'm not sure why they

can't just represent a character pack in a more efficient manner, but that's what I

was told when I presented in EWG.

Fact that storing string in template parameters list is ineffective did not stop GCC and Clang from implementing this:
https://godbolt.org/g/VjJzJi
https://godbolt.org/g/1DVftM
Thanks to that we can test how far we need push compilers to break when we use this UDL.

[Arthur O'Dwyer]

My uninformed opinion matches inkwizytor's.  Louis, I think you'll have to be more specific when you claim that "implementors" find P0424R0 objectionable.  Given that two of the let's-say-four major vendors have already managed to implement P0424R0, who could the holdouts be?  Microsoft?  EDG?  Or GCC and/or Clang being like "We'll implement this, but we'll grump about it"?

The situation seems analogous to std::make_integer_sequence<>, which I know is near and dear to your heart: we've got implementations already in the wild, and it's super useful for metaprogramming, but naive implementations are (claimed to be) slow. So, why not apply the same solution, which was to improve the compiler's internal mechanisms without compromising on the source-level API?

[Admittedly I'm oversimplifying; the most obvious reason not to "improve" the compiler mechanisms (e.g. by creating a special efficient representation for template-argument-lists all of whose elements are 'char') is that it would add code and complexity (and cognitive overhead, and test load, and bugs) to the compiler.]

Another idea to improve the shippability of P0424R0 would be to specify a separate implementation-defined limit on the number of characters in a UDL of this form; e.g., if some vendor wants to make it a hard error to use string UDLs longer than 64 characters, I think that would be 100% fine with today's working programmers, and the limit could easily be increased in C++3x if there were a consensus among the vendors.

my $.02,

–Arthur

[Tom Honermann]

I'd still prefer to be able to do something like this:

template<std::size_t N>
constexpr X operator""_udl(const char (&sl)[N]) { ... }

Tom.

[Tom Honermann]

Of course, that doesn't provide the string-literal -> unique type mapping (based on string literal contents) that would be desired for at least some use cases.

Tom.