Second draft of D0779R0 operator try proposal paper (with added native C++ macros)

[Niall Douglas]

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

The highlight of this paper is proposing native C++ macros as a better solution for implementing try, co_await, co_yield and co_return. These become:

• std::try#(T)
• std::coroutines::await#(T)
• std::coroutines::yield#(T)
• std::coroutines::return#(T)

Which I think everybody can agree is much nicer. Just plonk a "using namespace std::coroutines;" at the top of your coroutine and it's find and replace replacement, plus you can axe quite a few pages from the Coroutines TS.

Anyway I expect this discussion thread will be lively. From the previous discussion thread, people either love or hate boilerplate injection into calling scopes. But this is, at least, a generic library based alternative to constantly having to add new operators and keywords to the C++ language going into the future. Hate it as much as you like, EWG has better things to do than standardise boilerplate into the compiler.

Niall

[Nicol Bolas]

On Wednesday, October 11, 2017 at 9:02:24 PM UTC-4, Niall Douglas wrote:

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

The highlight of this paper is proposing native C++ macros as a better solution for implementing try, co_await, co_yield and co_return. These become:

• std::try#(T)
• std::coroutines::await#(T)
• std::coroutines::yield#(T)
• std::coroutines::return#(T)

Which I think everybody can agree is much nicer. Just plonk a "using namespace std::coroutines;" at the top of your coroutine and it's find and replace replacement, plus you can axe quite a few pages from the Coroutines TS.

Um, no you can't.

The Coroutines TS is a lot more complicated than you make it out to be. The presence of any of the `co_*` keywords in a function doesn't merely change the nature of the expression; it transforms the whole nature of the function. It turns a regular function into a coroutine function, which implicitly wraps the entire thing within a scope, adds a promise object, and does a host of other things.

Merely "injecting boilerplate" into an expression doesn't do that.

Furthermore, a coroutine function cannot issue a normal return statement, since its return value object is managed by the coroutine's promise object. Your mere "boilerplate injection" idea can't make you get a compile error if you return from a function.

And speaking of which, this is a good example of where your `try` idea (in either form) fails: it doesn't take into account the possibility of a coroutine function performing the `try` operation. In form 1, you're effectively requiring the compiler to do arbitrary look-ahead to see if there's a `co_*` keyword in use, and if it is, then it has to use `co_return`. In form 2, `try#` has no way to know whether it's a coroutine or not.

Effectively, in both cases, you would need a `co_try`.

Anyway I expect this discussion thread will be lively. From the previous discussion thread, people either love or hate boilerplate injection into calling scopes. But this is, at least, a generic library based alternative to constantly having to add new operators and keywords to the C++ language going into the future. Hate it as much as you like, EWG has better things to do than standardise boilerplate into the compiler.

If it's not worth standardizing, why do you want a generic ability to do it?

[Toby Allsopp]

Hi, Niall. Seeing as you dropped my name in your paper, I feel compelled to respond :)

On Thursday, 12 October 2017 14:02:24 UTC+13, Niall Douglas wrote:

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

The highlight of this paper is proposing native C++ macros as a better solution for implementing try, co_await, co_yield and co_return. These become:

• std::try#(T)
• std::coroutines::await#(T)
• std::coroutines::yield#(T)
• std::coroutines::return#(T)

Which I think everybody can agree is much nicer. Just plonk a "using namespace std::coroutines;" at the top of your coroutine and it's find and replace replacement, plus you can axe quite a few pages from the Coroutines TS.

Heh. As Nicol rather promptly pointed out, it's not that simple.

You throw around a lot of "everybody can agree" and "obvious reasons", but those are the very things that need to be justified in your paper.

For example, in what way is using co_await on an optional "not a true coroutine await"? Why, exactly, does it need to be "nipped in the bud"? I'm not necessarily arguing in favour of using co_await for that purpose, but I really don't see what's so abhorrent about it.

 

Anyway I expect this discussion thread will be lively. From the previous discussion thread, people either love or hate boilerplate injection into calling scopes. But this is, at least, a generic library based alternative to constantly having to add new operators and keywords to the C++ language going into the future. Hate it as much as you like, EWG has better things to do than standardise boilerplate into the compiler.

I like the idea of being able to implement interesting compile-time transformations, but I see your proposed native macros as not powerful enough. I think something that builds on static reflection of a function body at the statement and expression level, similar to the metaclasses stuff, will give better results and allow for more interesting transformations.

For example, I can't see how to get something like Haskell's do notation using your native macros. For that you need a way to capture the continuation of the calling function as a callable object, which means effectively wrapping it in a lambda.

Regards,

Toby.

[Nicol Bolas]

On Wednesday, October 11, 2017 at 10:08:32 PM UTC-4, Toby Allsopp wrote:

Hi, Niall. Seeing as you dropped my name in your paper, I feel compelled to respond :)

On Thursday, 12 October 2017 14:02:24 UTC+13, Niall Douglas wrote:

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

The highlight of this paper is proposing native C++ macros as a better solution for implementing try, co_await, co_yield and co_return. These become:

• std::try#(T)
• std::coroutines::await#(T)
• std::coroutines::yield#(T)
• std::coroutines::return#(T)

Which I think everybody can agree is much nicer. Just plonk a "using namespace std::coroutines;" at the top of your coroutine and it's find and replace replacement, plus you can axe quite a few pages from the Coroutines TS.

Heh. As Nicol rather promptly pointed out, it's not that simple.

You throw around a lot of "everybody can agree" and "obvious reasons", but those are the very things that need to be justified in your paper.

For example, in what way is using co_await on an optional "not a true coroutine await"? Why, exactly, does it need to be "nipped in the bud"? I'm not necessarily arguing in favour of using co_await for that purpose, but I really don't see what's so abhorrent about it.

I can explain that:

1. If you `co_await` in a function, any return must be a `co_return` as well.

2. As I pointed out in my post, using `co_await` adds a bunch of machinery to your code. Among the practical effects of that machinery is that guaranteed elision no longer works. The promise object in the coroutine manages the return value object, and it cannot share its storage with the caller's return value. So if you `co_return` a prvalue, that will be used to initialize the return value object in the promise. And when the eventual caller gets the return value, it will have to copy/move it from the promise.

3. If you're using `co_await` for optionals or expected, you cannot simultaneously use `co_await` for waiting on other futures too. That is, if your function returns `future<optional<T>>`, you can't use `co_await` on something that returns an `optional<T>`. The two "channels" of communication can't really coexist within the same function. Having a `co_try` would allow you to make `future<optional<T>>` work.

Regards,

Toby.

[Toby Allsopp]

On Thursday, 12 October 2017 15:23:22 UTC+13, Nicol Bolas wrote:

On Wednesday, October 11, 2017 at 10:08:32 PM UTC-4, Toby Allsopp wrote:

Hi, Niall. Seeing as you dropped my name in your paper, I feel compelled to respond :)

On Thursday, 12 October 2017 14:02:24 UTC+13, Niall Douglas wrote:

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

The highlight of this paper is proposing native C++ macros as a better solution for implementing try, co_await, co_yield and co_return. These become:

• std::try#(T)
• std::coroutines::await#(T)
• std::coroutines::yield#(T)
• std::coroutines::return#(T)

Which I think everybody can agree is much nicer. Just plonk a "using namespace std::coroutines;" at the top of your coroutine and it's find and replace replacement, plus you can axe quite a few pages from the Coroutines TS.

Heh. As Nicol rather promptly pointed out, it's not that simple.

You throw around a lot of "everybody can agree" and "obvious reasons", but those are the very things that need to be justified in your paper.

For example, in what way is using co_await on an optional "not a true coroutine await"? Why, exactly, does it need to be "nipped in the bud"? I'm not necessarily arguing in favour of using co_await for that purpose, but I really don't see what's so abhorrent about it.

I can explain that:

1. If you `co_await` in a function, any return must be a `co_return` as well.

That's true, but I don't get your point. Is the problem the inconvenience of having to change the function body in potentially many places if you decide to use co_await?

I don't believe this requirement is fundamental to the Coroutines TS.

2. As I pointed out in my post, using `co_await` adds a bunch of machinery to your code. Among the practical effects of that machinery is that guaranteed elision no longer works. The promise object in the coroutine manages the return value object, and it cannot share its storage with the caller's return value. So if you `co_return` a prvalue, that will be used to initialize the return value object in the promise. And when the eventual caller gets the return value, it will have to copy/move it from the promise.

This is a very good point, thanks for pointing it out.

Niall - put this in your paper as justification for nipping co_await optional in the bud!

 

3. If you're using `co_await` for optionals or expected, you cannot simultaneously use `co_await` for waiting on other futures too. That is, if your function returns `future<optional<T>>`, you can't use `co_await` on something that returns an `optional<T>`. The two "channels" of communication can't really coexist within the same function. Having a `co_try` would allow you to make `future<optional<T>>` work.

Hmm, this is an interesting point. It reminds me of monad transformers.

I suspect co_await on an optional in a coroutine returning future<optional<T>> could be made to work, but the composition would be awkward, i.e. the future's promise would need to know about optional or optional-like things.

Cheers,

Toby.

[Niall Douglas]

The Coroutines TS is a lot more complicated than you make it out to be. The presence of any of the `co_*` keywords in a function doesn't merely change the nature of the expression; it transforms the whole nature of the function. It turns a regular function into a coroutine function, which implicitly wraps the entire thing within a scope, adds a promise object, and does a host of other things.

All of that still works without issue. If the compiler can spot a "co_*" call and do the coroutinisation, it can do the same with whatever builtin coroutine related intrinsics are injected by the macro.

 

Merely "injecting boilerplate" into an expression doesn't do that.

Furthermore, a coroutine function cannot issue a normal return statement, since its return value object is managed by the coroutine's promise object. Your mere "boilerplate injection" idea can't make you get a compile error if you return from a function.

Also works just fine. We inject unmodified tokens. Whatever is in the macros gets injected as-is.

 

And speaking of which, this is a good example of where your `try` idea (in either form) fails: it doesn't take into account the possibility of a coroutine function performing the `try` operation. In form 1, you're effectively requiring the compiler to do arbitrary look-ahead to see if there's a `co_*` keyword in use, and if it is, then it has to use `co_return`. In form 2, `try#` has no way to know whether it's a coroutine or not.

try# doesn't need to know anything. The destination where the tokens are injected is the only consideration.

I'm actually fairly surprised at the lack of resistance to this idea. I think I'll go poke Gor and see how much he hates it ...

Niall

[Niall Douglas]

Heh. As Nicol rather promptly pointed out, it's not that simple.

Coroutines are actually very easy to implement under the bonnet. Getting them to compile into efficient code is much harder, but some implementation or other probably can be hacked into a compiler within a week. I say this as a former clang/LLVM port maintainer for a platform, and I don't claim the hacked implementation would be pretty nor wise.

 

You throw around a lot of "everybody can agree" and "obvious reasons", but those are the very things that need to be justified in your paper.

Deliberately so. The feedback I see here will expand out those sections before paper submission because I don't know what everybody does agree on and what reasons are obvious.

 

For example, in what way is using co_await on an optional "not a true coroutine await"? Why, exactly, does it need to be "nipped in the bud"? I'm not necessarily arguing in favour of using co_await for that purpose, but I really don't see what's so abhorrent about it.

I'm very keen that we avoid hacks instead of doing things properly. In many ways the co_* keywords are themselves a hack, some way of marking a function as being a special "coroutine function" was needed plus boilerplate needed to be injected.

Herb's metaclasses could be a solution, but that's a very big proposal very far out, and one which still wouldn't eliminate the need for C macros.  So I float the idea of native C++ macros, see if it sticks. It's one solution to the problem. Another is letting you define your own keywords local to a namespace, but I know Bjarne hates that idea.

 

 

Anyway I expect this discussion thread will be lively. From the previous discussion thread, people either love or hate boilerplate injection into calling scopes. But this is, at least, a generic library based alternative to constantly having to add new operators and keywords to the C++ language going into the future. Hate it as much as you like, EWG has better things to do than standardise boilerplate into the compiler.

I like the idea of being able to implement interesting compile-time transformations, but I see your proposed native macros as not powerful enough. I think something that builds on static reflection of a function body at the statement and expression level, similar to the metaclasses stuff, will give better results and allow for more interesting transformations.

Absolutely agreed. They're actually simplified considerably over the evil lambda idea. I was aiming for something achievable for C++ 20.

 

For example, I can't see how to get something like Haskell's do notation using your native macros. For that you need a way to capture the continuation of the calling function as a callable object, which means effectively wrapping it in a lambda.

Vicente has a paper on implementing Haskell do in the works. It is sufficiently complex it's unachievable for C++ 20.  In the meantime, these native C++ macros are (a) extremely easy to implement by the compiler vendor and (b) would be relatively uncontroversial to design seeing how closely they replicate C macros (but without the problems). They're achievable for C++ 20, in time for the Coroutines TS and Expected.

BTW Toby that was a great talk you did, I know Vicente watched it too.

Niall

[Niall Douglas]

1. If you `co_await` in a function, any return must be a `co_return` as well.

That's true, but I don't get your point. Is the problem the inconvenience of having to change the function body in potentially many places if you decide to use co_await?

I don't believe this requirement is fundamental to the Coroutines TS.

It's more that a coroutinised function is not a normal function. It can't behave the same way, it can't be compiled the same way, and it definitely can't be invoked the same way. This all greatly limits the usefulness of co_await.

 

2. As I pointed out in my post, using `co_await` adds a bunch of machinery to your code. Among the practical effects of that machinery is that guaranteed elision no longer works. The promise object in the coroutine manages the return value object, and it cannot share its storage with the caller's return value. So if you `co_return` a prvalue, that will be used to initialize the return value object in the promise. And when the eventual caller gets the return value, it will have to copy/move it from the promise.

This is a very good point, thanks for pointing it out.

Niall - put this in your paper as justification for nipping co_await optional in the bud!

I had thought that consequence very obvious. But okay.

 

 

3. If you're using `co_await` for optionals or expected, you cannot simultaneously use `co_await` for waiting on other futures too. That is, if your function returns `future<optional<T>>`, you can't use `co_await` on something that returns an `optional<T>`. The two "channels" of communication can't really coexist within the same function. Having a `co_try` would allow you to make `future<optional<T>>` work.

Hmm, this is an interesting point. It reminds me of monad transformers.

I suspect co_await on an optional in a coroutine returning future<optional<T>> could be made to work, but the composition would be awkward, i.e. the future's promise would need to know about optional or optional-like things.

It's also just wrong. We are using a language feature for purposes it was not intended for before it's even been standardised.

If that isn't a screaming alarm bell that this proposed language feature isn't the right design, then I don't know what is.

Niall

[Ville Voutilainen]

On 12 October 2017 at 15:55, Niall Douglas <nialldo...@gmail.com> wrote:
> Herb's metaclasses could be a solution, but that's a very big proposal very
> far out, and one which still wouldn't eliminate the need for C macros. So I
> float the idea of native C++ macros, see if it sticks. It's one solution to
> the problem. Another is letting you define your own keywords local to a
> namespace, but I know Bjarne hates that idea.

A couple of remarks:

1) I have toyed with the idea of "inject into the parent scope" many
times and hinted at it on this forum multiple
times. The way I expressed it was with an "inline template", that is

inline template <class T>
void f(T t) // yes, the return type is stupid, this is not a function,
it doesn't return
{
if (stars_are_properly_aligned(t))
return foo(t); // this returns from the surrounding scope
}

and using it would look like a function call,

int g()
{
f(42);
}

2) this is indeed one remaining use of macros for which we have no
superior replacement. The inline-template idea I've toyed with
doesn't suggest at the call site that injection will happen, whereas
your hash-macro perhaps would/should.

3) I think it does harm to the idea if it suggests tokens to be
injected into the surrounding scope. It seems much more palatable
to me if the "native-macro" establishes a new block and just injects
that block into the surrounding scope. In a similar vein,
the code in the native-macro should be valid, and if dependent, should
require that there's at least one specialication for which it's
valid. These were the things that were required for if-constexpr to be
acceptable, I expect the same requirements to arise here.

[dgutson .]

El 12 oct. 2017 10:04 a.m., "Ville Voutilainen" <ville.vo...@gmail.com> escribió:

On 12 October 2017 at 15:55, Niall Douglas <nialldo...@gmail.com> wrote:
> Herb's metaclasses could be a solution, but that's a very big proposal very
> far out, and one which still wouldn't eliminate the need for C macros.  So I
> float the idea of native C++ macros, see if it sticks. It's one solution to
> the problem. Another is letting you define your own keywords local to a
> namespace, but I know Bjarne hates that idea.

A couple of remarks:

1) I have toyed with the idea of "inject into the parent scope" many
times and hinted at it on this forum multiple
times. The way I expressed it was with an "inline template", that is

inline template <class T>
void f(T t) // yes, the return type is stupid, this is not a function,
it doesn't return
{
     if (stars_are_properly_aligned(t))
         return foo(t); // this returns from the surrounding scope
}

and using it would look like a function call,

int g()
{
    f(42);
}

2) this is indeed one remaining use of macros for which we have no
superior replacement. The inline-template idea I've toyed with
doesn't suggest at the call site that injection will happen,

What about adding 'inline' in the call side too?

whereas
your hash-macro perhaps would/should.

3) I think it does harm to the idea if it suggests tokens to be
injected into the surrounding scope. It seems much more palatable
to me if the "native-macro" establishes a new block and just injects
that block into the surrounding scope. In a similar vein,
the code in the native-macro should be valid, and if dependent, should
require that there's at least one specialication for which it's
valid. These were the things that were required for if-constexpr to be
acceptable, I expect the same requirements to arise here.

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.
To post to this group, send email to std-pr...@isocpp.org.

To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAFk2RUYXya9jkfdW51HPPUaGH79K9%3DimHZtb-MS5sVU%2BdSn9wQ%40mail.gmail.com.

[Niall Douglas]

On Thursday, October 12, 2017 at 2:04:17 PM UTC+1, Ville Voutilainen wrote:

A couple of remarks:

1) I have toyed with the idea of "inject into the parent scope" many
times and hinted at it on this forum multiple
times. The way I expressed it was with an "inline template", that is

inline template <class T>
void f(T t) // yes, the return type is stupid, this is not a function,
it doesn't return
{
     if (stars_are_properly_aligned(t))
         return foo(t); // this returns from the surrounding scope
}

and using it would look like a function call,

int g()
{
    f(42);
}

Heh. I actually started out with that exact design. I rejected it as being too easy to confuse a real template with a not-really-template.

 

2) this is indeed one remaining use of macros for which we have no
superior replacement. The inline-template idea I've toyed with
doesn't suggest at the call site that injection will happen, whereas
your hash-macro perhaps would/should.

3) I think it does harm to the idea if it suggests tokens to be
injected into the surrounding scope. It seems much more palatable
to me if the "native-macro" establishes a new block and just injects
that block into the surrounding scope. In a similar vein,
the code in the native-macro should be valid, and if dependent, should
require that there's at least one specialication for which it's
valid. These were the things that were required for if-constexpr to be
acceptable, I expect the same requirements to arise here.

Would this not prevent injecting new variable declarations?

Niall

[Ville Voutilainen]

On 12 October 2017 at 16:12, Niall Douglas <nialldo...@gmail.com> wrote:
>> 3) I think it does harm to the idea if it suggests tokens to be
>> injected into the surrounding scope. It seems much more palatable
>> to me if the "native-macro" establishes a new block and just injects
>> that block into the surrounding scope. In a similar vein,
>> the code in the native-macro should be valid, and if dependent, should
>> require that there's at least one specialication for which it's
>> valid. These were the things that were required for if-constexpr to be
>> acceptable, I expect the same requirements to arise here.
>
>
> Would this not prevent injecting new variable declarations?

Yes, it would. Some might consider that a feature rather than a bug.
The downside is of course
that you can't inject a nifty RAII-variable into the surrounding
scope. I don't know how big a deal
the scope would be here; it was certainly of utmost importance for
if-constexpr. The best way to
find out is to propose this and see what EWG says. But at any rate,
injection of tokens seems
like something that will meet fire and brimstone.

[Niall Douglas]

> Would this not prevent injecting new variable declarations?

Yes, it would. Some might consider that a feature rather than a bug.

One of the big advantages of C macros is that they can inject partial scopes e.g. BEGIN_BITFIELD(foo) ... END_BITFIELD(foo), and RAII variables for firing code execution on exit.

Removing that facility would keep C macros being used in new code.

 

The downside is of course
that you can't inject a nifty RAII-variable into the surrounding
scope. I don't know how big a deal
the scope would be here; it was certainly of utmost importance for
if-constexpr. The best way to
find out is to propose this and see what EWG says. But at any rate,
injection of tokens seems
like something that will meet fire and brimstone.

I don't love it either for the record. But it is well understood by all, there is little danger of unanticipated consequences in adopting it (well, ones worse than C macros).

And a truly better solution is actually quite hard to achieve in any reasonable timespan.

One thing I'd like is if these macros had constexpr-only function pointer behaviours i.e. within constexpr, you can return them and pass them around between constexpr functions, with the token injection only occurring upon invocation. I could see that as being really very useful. But it would be much harder for compiler vendors to implement as they're need to recursively reconstruct the AST being executed by constexpr, so I left that idea out.

Perhaps the partial parsing idea like a template you had might help with that, once parsed it's useful to constexpr, though one loses the ability to inject partial scopes which I think is important for total C macro replacement.

Do you know what is really weird with these C++ macros, it just occurred to me? These C++ macros are actually how MSVC has historically implemented templates: token injection. I am effectively proposing that here.

Niall

[Nicol Bolas]

The dangers and complexities of this "macro" idea are manifold. And by standardizing it as part of the language, we run the risk of creating language-sanctioned chaos in people's code.

Coupled with this is that the current proposal is little more than a sketch of a design, rather than something with actual rules in place. It talks about it being a "C++ macro", but it never really says what that means. After all, if you did actual macro expansion, you wouldn't be able to expand a multi-statement "macro" into a place that expects a single expression. `auto var = try#(some_expression);` would "macro expand" to `auto var = if(!some_expression) {return {};} some_expression;`, which isn't legal C++.

So the semantics of this are not fully formed.

So let's see if we can't find a way to minimize the potential for chaos and bring something more fully realized to the table. I suggest the following:

1: Macro-functions are not "macros" at all. Calling them is in no way equivalent to macro-expansion; except where otherwise stated, calling a macro function is identical to calling a regular function.

For simplicity's sake, I will continue to refer to them as "macro functions", because there needs to be a distinction between them and regular functions.

2: Macro functions must be inline.

3: Macro functions cannot directly access the scope of their caller. If you need a macro function to access something from the local scope, you pass it as a parameter, just like any other function. This prevents you from accidentally breaking a macro functions just because you named a local variable the same as it expected as part of some interface. If it needs stuff, pass that stuff in.

4: Macro functions cannot leak scope. That is, whatever they declare inside themselves stays inside themselves. It's in its own scope and cannot leak out to the caller's scope. They can return values like any function call, but that's it.

This mirrors the limitations of our current `if constexpr` syntax. The older `static_if` proposal allowed the stuff declared in them to conditionally leak out into the surrounding scope. The current one doesn't allow this.

5: You cannot get pointers to macro-functions. So you cannot pass them around. This ensures that the call graph is static.

6: Operations inside a macro function affects the macro function itself, not the caller's scope.

7: Macro functions can only affect the outer scope by issuing control flow changes: `return#`, `break#`, and `continue#`. `continue#` and `break#` cannot be used in places where there is a loop within the scope of a macro function.

8: The "outer scope" of a macro function is the closest non-macro function in the call stack. This allows nested calls to macro-functions.

These rules impose a degree of order on this concept. It keeps it from being "stick random code in places where that code can't work" and merely becomes "a function call that can impose some control flow on its caller."

[Nicol Bolas]

On Thursday, October 12, 2017 at 10:32:40 AM UTC-4, Niall Douglas wrote:

> Would this not prevent injecting new variable declarations?

Yes, it would. Some might consider that a feature rather than a bug.

One of the big advantages of C macros is that they can inject partial scopes e.g. BEGIN_BITFIELD(foo) ... END_BITFIELD(foo), and RAII variables for firing code execution on exit.

Removing that facility would keep C macros being used in new code.

... so what?

Not everyone believes that macros are an evil that must be eliminated. Macros are a tool, one that should be used sparingly, but they're still a tool.

This tool should not be judged based on whether it eliminates macro usage, but based on whether it comes to a coherent design and offers practical value to C++ developers.

The downside is of course
that you can't inject a nifty RAII-variable into the surrounding
scope. I don't know how big a deal
the scope would be here; it was certainly of utmost importance for
if-constexpr. The best way to
find out is to propose this and see what EWG says. But at any rate,
injection of tokens seems
like something that will meet fire and brimstone.

I don't love it either for the record. But it is well understood by all, there is little danger of unanticipated consequences in adopting it (well, ones worse than C macros).

And a truly better solution is actually quite hard to achieve in any reasonable timespan.

One thing I'd like is if these macros had constexpr-only function pointer behaviours i.e. within constexpr, you can return them and pass them around between constexpr functions, with the token injection only occurring upon invocation. I could see that as being really very useful. But it would be much harder for compiler vendors to implement as they're need to recursively reconstruct the AST being executed by constexpr, so I left that idea out.

Perhaps the partial parsing idea like a template you had might help with that, once parsed it's useful to constexpr, though one loses the ability to inject partial scopes which I think is important for total C macro replacement.

Do you know what is really weird with these C++ macros, it just occurred to me? These C++ macros are actually how MSVC has historically implemented templates: token injection. I am effectively proposing that here.

No, you're not. Your seminal example, `auto var = try#(some_expression);` cannot work by token pasting. If you token paste that out, you would get a compile error.

Not to mention what would happen if `some_expression` were not a variable but a complex expression. The token-injection version would evaluate that expression multiple times.

Token injection is not what you want.

 

Niall

[Matthew Woehlke]

On 2017-10-12 09:04, Ville Voutilainen wrote:
> 3) I think it does harm to the idea if it suggests tokens to be
> injected into the surrounding scope. It seems much more palatable
> to me if the "native-macro" establishes a new block and just injects
> that block into the surrounding scope.

My biggest complaint with that is that I'd really like to be able to write:

auto x = foo#(...);

It seems to me like the general idea as presented in the paper is
feasible, but where I'd like to see it expanded slightly is making it
more explicit that these critters can also "yield" a value.

I'm on the fence if that implies that they should specify both their
"yield" type and their "cause parent to return" type. If we limit them
to being inlined, the latter is probably less important.

That leaves how to specify a "yield" vs. a "make parent return". Having
the former as a terminal statement is plausible (and is how statement
expressions are implemented), but seems hard to read, and is limiting in
the sense that the block can't "yield early". The trouble, of course, is
that I don't see how to make this work without a new keyword.

--
Matthew

[Toby Allsopp]

On Friday, 13 October 2017 01:55:01 UTC+13, Niall Douglas wrote:

Heh. As Nicol rather promptly pointed out, it's not that simple.

Coroutines are actually very easy to implement under the bonnet. Getting them to compile into efficient code is much harder, but some implementation or other probably can be hacked into a compiler within a week. I say this as a former clang/LLVM port maintainer for a platform, and I don't claim the hacked implementation would be pretty nor wise.

The front-end for coroutines is not that tricky, I agree, and it does basically what you've described, i.e. expands into boilerplate code and calls to builtins. It's the back-end that is interesting ,where it takes the function apart and rewrites it based on the presence of the builtins. That's how it works in LLVM anyway.

For example, in what way is using co_await on an optional "not a true coroutine await"? Why, exactly, does it need to be "nipped in the bud"? I'm not necessarily arguing in favour of using co_await for that purpose, but I really don't see what's so abhorrent about it.

I'm very keen that we avoid hacks instead of doing things properly. In many ways the co_* keywords are themselves a hack, some way of marking a function as being a special "coroutine function" was needed plus boilerplate needed to be injected.

I don't think "hack" is particularly useful technical term as it doesn't have an objective definition. But I agree with you here.

 

For example, I can't see how to get something like Haskell's do notation using your native macros. For that you need a way to capture the continuation of the calling function as a callable object, which means effectively wrapping it in a lambda.

Vicente has a paper on implementing Haskell do in the works. It is sufficiently complex it's unachievable for C++ 20.  In the meantime, these native C++ macros are (a) extremely easy to implement by the compiler vendor and (b) would be relatively uncontroversial to design seeing how closely they replicate C macros (but without the problems). They're achievable for C++ 20, in time for the Coroutines TS and Expected.

Cool, I think Vicente's work on foundational concepts for Functor etc. is important. And that syntactic sugar should be based on those concepts rather than being ad-hoc.

I'm worried about putting stop-gap language features in that will be made obsolete soon. However, it's also a trap to do nothing until we have the grand unified theory of everything!

[Vicente J. Botet Escriba]

Le 12/10/2017 à 14:55, Niall Douglas a écrit :

Vicente has a paper on implementing Haskell do in the works. It is sufficiently complex it's unachievable for C++ 20.  In

Do you mean Haskell do-notation? No. I don't have any paper of such a do-notation.  This doesn't mean that I wouldn't appreciate it.

Vicente

[Toby Allsopp]

On Friday, 13 October 2017 02:00:12 UTC+13, Niall Douglas wrote:

1. If you `co_await` in a function, any return must be a `co_return` as well.

That's true, but I don't get your point. Is the problem the inconvenience of having to change the function body in potentially many places if you decide to use co_await?

I don't believe this requirement is fundamental to the Coroutines TS.

It's more that a coroutinised function is not a normal function. It can't behave the same way, it can't be compiled the same way, and it definitely can't be invoked the same way. This all greatly limits the usefulness of co_await.

Maybe I don't understand your meaning, but coroutines are invoked in the same way as normal functions; that's one of the design goals of the TS as I understand it. The caller has no way of telling whether it is calling a coroutine or not.

3. If you're using `co_await` for optionals or expected, you cannot simultaneously use `co_await` for waiting on other futures too. That is, if your function returns `future<optional<T>>`, you can't use `co_await` on something that returns an `optional<T>`. The two "channels" of communication can't really coexist within the same function. Having a `co_try` would allow you to make `future<optional<T>>` work.

Hmm, this is an interesting point. It reminds me of monad transformers.

I suspect co_await on an optional in a coroutine returning future<optional<T>> could be made to work, but the composition would be awkward, i.e. the future's promise would need to know about optional or optional-like things.

It's also just wrong. We are using a language feature for purposes it was not intended for before it's even been standardised.

If that isn't a screaming alarm bell that this proposed language feature isn't the right design, then I don't know what is.

I think I see your point here. The Coroutines TS does the two things it was explicitly designed to do (async and generators) very well and it feels like it can almost be pressed into service to solve some other problems, but doesn't quite do so elegantly (or at all for some closely related problems). It would be a shame, IMO, to introduce a completely separate abstraction to solve those other problems rather than designing a more general abstraction that can do everything. But there's that "do nothing until we can do everything" trap again :)

Cheers,

Toby.

[Niall Douglas]

On Thursday, October 12, 2017 at 3:49:57 PM UTC+1, Nicol Bolas wrote:

The dangers and complexities of this "macro" idea are manifold. And by standardizing it as part of the language, we run the risk of creating language-sanctioned chaos in people's code.

Meh. C++ is just a fancy macro assembler. Much less chaos results from this idea than many others already in the standard.

 

1: Macro-functions are not "macros" at all. Calling them is in no way equivalent to macro-expansion; except where otherwise stated, calling a macro function is identical to calling a regular function.

For simplicity's sake, I will continue to refer to them as "macro functions", because there needs to be a distinction between them and regular functions.

2: Macro functions must be inline.

I agree with both of these. 

3: Macro functions cannot directly access the scope of their caller. If you need a macro function to access something from the local scope, you pass it as a parameter, just like any other function. This prevents you from accidentally breaking a macro functions just because you named a local variable the same as it expected as part of some interface. If it needs stuff, pass that stuff in.

4: Macro functions cannot leak scope. That is, whatever they declare inside themselves stays inside themselves. It's in its own scope and cannot leak out to the caller's scope. They can return values like any function call, but that's it.

These two eliminate the whole point of the proposal which is to inject library defined boilerplate into the caller.

If you don't have that, just use a templated free function. It's basically what you're proposing.

 

5: You cannot get pointers to macro-functions. So you cannot pass them around. This ensures that the call graph is static.

6: Operations inside a macro function affects the macro function itself, not the caller's scope.

If the macro doesn't want to affect the caller's scope, it wraps stuff in a local scope.

 

7: Macro functions can only affect the outer scope by issuing control flow changes: `return#`, `break#`, and `continue#`. `continue#` and `break#` cannot be used in places where there is a loop within the scope of a macro function.

I see where you're heading. Those you quoted are other macro invocations of course. But let's replace those with keyword-> i.e. return-> expr, break->, continue->. That I think can work.

 

8: The "outer scope" of a macro function is the closest non-macro function in the call stack. This allows nested calls to macro-functions.

These rules impose a degree of order on this concept. It keeps it from being "stick random code in places where that code can't work" and merely becomes "a function call that can impose some control flow on its caller."

In all our interactions to date I don't think I've ever said this to you before, but you have some good ideas in there. And I think I'm going to adopt most of your ideas. The sole bit I disagree with is scoping, but the rest of it I think is good.

Thank you.

Niall

[Niall Douglas]

3: Macro functions cannot directly access the scope of their caller. If you need a macro function to access something from the local scope, you pass it as a parameter, just like any other function. This prevents you from accidentally breaking a macro functions just because you named a local variable the same as it expected as part of some interface. If it needs stuff, pass that stuff in.

4: Macro functions cannot leak scope. That is, whatever they declare inside themselves stays inside themselves. It's in its own scope and cannot leak out to the caller's scope. They can return values like any function call, but that's it.

I've come up with a compromise solution:

// Macro functions look just like a free function except for the # at the
// end of the function name. Note that the # counts as part of the identifier,
// so return# does not collide with the return keyword.
template<class T> inline int return#(T v)
{
  if(v > 0)
    return -> v;  // control flow keyword + '->' means it affects the
                  // calling function, not this macro function
  if(v < 0)      
    break ->;     // Also this break is executed in the calling function


  // We can inject variable declarations into the calling function
  // with typename + '->'. This is useful for RAII triggered cleanup.
  // Note that __FILE__, __COUNTER__ and __LINE__ refer to the caller,
  // not here. This allows injection of uniquely named variables to
  // prevent collision
  int -> a = 5;
 
  // Otherwise this function macro has a local scope, and code
  // executed here remains here
  size_t n = 0;
  for(; n < 5; n++)
  {
    // We can also refer to variables in the caller's scope like this
    // This lets an initialising macro function call inject state later
    // retrievable by a second macro function call
    (-> a) ++;
  }
 
  // This returns a value from this function macro to the caller
  return -1;
}

In use:

static int values[];  // some array somewhere
int foo(int n)
{
  int acc = 0;
  for(;;)
  {
    // Macro functions are invoked just like normal functions,
    // no token pasting. So n will be incremented exactly once.
    int ret = return#(values[n++]);
   
    // int a was injected here by the macro function
    acc += a;
  }
}

Acceptable?

Niall

[Niall Douglas]

On Friday, October 13, 2017 at 2:44:38 AM UTC+1, Niall Douglas wrote:

3: Macro functions cannot directly access the scope of their caller. If you need a macro function to access something from the local scope, you pass it as a parameter, just like any other function. This prevents you from accidentally breaking a macro functions just because you named a local variable the same as it expected as part of some interface. If it needs stuff, pass that stuff in.

4: Macro functions cannot leak scope. That is, whatever they declare inside themselves stays inside themselves. It's in its own scope and cannot leak out to the caller's scope. They can return values like any function call, but that's it.

I've come up with a compromise solution:

I've actually ended up implementing exactly your recommendations for sanity. Injected variables now get unique identifiers based on the fully qualified name of the macro which injected them. That effectively isolates each macro from any other, and from the calling function except that it fires any destructors on injected variables on scope exit. New example macro function:

// Macro functions look just like a free function except for the # at the
// end of the function name. Note that the # counts as part of the identifier,
// so return# does not collide with the return keyword.
template<class T>
inline int return#(T v)
{
  if(v > 0)
    return -> v;  // control flow keyword + '->' means it affects the
                  // calling function, not this macro function
  if(v < 0)      
    break ->;     // Also this break is executed in the calling function

                  // If break isn't valid at the point of use
                  // in the calling function, it will not compile

  // We can inject variable declarations into the calling function
  // with typename + '->'. This is useful for RAII triggered cleanup

  // i.e. these get destructed when the scope of the call point exits.
  // Note that the actual name of the variable injected will
  // be some very unique identifier which cannot collide with any
  // other variable, including those injected by other macro functions

  int -> a = 5;
 
  // Otherwise this function macro has a local scope, and code
  // executed here remains here
  size_t n = 0;
  for(; n < 5; n++)
  {

    // We can also refer to variables previously injected into the
    // caller's scope by this macro function like this.
    // This lets one keep state across invocations of the macro function

    (-> a) ++;
  }
 
  // This returns a value from this function macro to the caller

  // If you wrote return -> a, that would be a compile error
  // as there is no variable called a in this scope.
  return (-> a);
}

I'm going to bed now. I'm exhausted.

Niall

 

[Andrzej Krzemieński]

W dniu czwartek, 12 października 2017 03:02:24 UTC+2 użytkownik Niall Douglas napisał:

This paper is much closer to finished than the previous half arsed paper posted here at Vicente's request. It is also pretty much completely rewritten, and Vicente no longer wishes to author it.

Let me copy the reply from similar topic in SG14.

Instead of "macro functions" consider the following language extension as an alternative. It appears to me less invasive, more focused, and seems to address your problem.

You can declare a function to have -- apart form normal return -- an "exported return slot":

T op(X x, Y y) export return U;

This means: function `op` can normally return a `T`, it takes arguments X and Y normally, but it also receives a *slot* where it can return another value of an unrelated type `U`. Everyone can see from declaration that this function is special: it has exported return slot.

Now, I can use such special function as follows:

U test(X x)
{
  T v = op(x, Y{});
  U u = transform(v);
  return u;
}

The call to op is special because op has been declared as special. Apart from passing arguments X and Y we are passing a return slot to `op` behind the scenes. If function returns normally (a `T`) it is stored in variable `v` normally. If `op` returns through an "exported slot", we immediately return the value from the slot out of our own function `test`.

With this functionality your operator TRY could be implemented as:

template <class T, class E, class T2>
T try_(std::expected<T, E> v) export return std::expected<T2, E>
{
  if (v)
    return v.value(); // return normally
  else
    export return v.error(); // return through exported slot
}

And you can use it as follows:

expected<float> get_float() noexcept
{
  int _int = try_(get_int());
  float ret = (float) _int;

  if ((int) ret != _int)
    return unexpected(std::errc::result_out_of_range);

  return ret;
}

Of course the type of exported return slot of the called function should be convertible to (or deduced from) the type of the calling function. Also, in the compiled code there would be no slots any more.

It looks like it could solve the problem with operator TRY. I do not know if it can address the case of coroutines, as I do not fully understand them.

Regards,

&rzej;

[gmis...@gmail.com]

This is the winner.

[Niall Douglas]

Instead of "macro functions" consider the following language extension as an alternative. It appears to me less invasive, more focused, and seems to address your problem.

This idea looks very similar to Emil's alternative design to Outcome/Expected, https://zajo.github.io/boost-noexcept/. It uses a "side channel" or "out of band" channel for returning alternatives to the normal return.

I'm not fond of OOB based designs for this problem domain. Your proposal I think would also require changes to ABI, specifically mangling. That's a big ask for a fixed function feature with limited reusability.

It looks like it could solve the problem with operator TRY. I do not know if it can address the case of coroutines, as I do not fully understand them.

Coroutines as currently proposed require the co_* operators to be called from the coroutinised function. You cannot, for example, call co_await via a helper function without coroutinising the helper function. I find this greatly annoying personally when writing coroutine code. I feel myself reaching for C macros to wrap boilerplate around co_await rather than being bothered to do it properly and write up a new awaitable type just to hide boilerplate.

Native C++ macro functions solve that too. They solve a ton of stuff, including lots of problems we don't know about yet.

Your proposal is much more fixed featured, and without changes to how a function becomes coroutinised, I don't think could implement co_await/co_yield/co_return. But more experienced folk on here may jump in with alternative opinions on that.

Niall

[Andrzej Krzemieński]

W dniu piątek, 13 października 2017 17:39:24 UTC+2 użytkownik Niall Douglas napisał:

Instead of "macro functions" consider the following language extension as an alternative. It appears to me less invasive, more focused, and seems to address your problem.

This idea looks very similar to Emil's alternative design to Outcome/Expected, https://zajo.github.io/boost-noexcept/. It uses a "side channel" or "out of band" channel for returning alternatives to the normal return.

Emil's library simply uses a thread-local storage. What I propose is just a construct that when compiled generates exactly same code as your macros. The "slot" is only for the purpose of describing the feature, when compiled, it will be replaced with a branch instruction and return.

 

I'm not fond of OOB based designs for this problem domain. Your proposal I think would also require changes to ABI, specifically mangling. That's a big ask for a fixed function feature with limited reusability.

I do not think it needs to affect ABI. Similarly to constexpr functions, you can require of functions with exported return slot that their body is visible in places where it is invoked. It is close to a macro, except that it is namespace-scoped and does adhere to name lookup rules of normal functions.

It looks like it could solve the problem with operator TRY. I do not know if it can address the case of coroutines, as I do not fully understand them.

Coroutines as currently proposed require the co_* operators to be called from the coroutinised function. You cannot, for example, call co_await via a helper function without coroutinising the helper function. I find this greatly annoying personally when writing coroutine code. I feel myself reaching for C macros to wrap boilerplate around co_await rather than being bothered to do it properly and write up a new awaitable type just to hide boilerplate.

Native C++ macro functions solve that too. They solve a ton of stuff, including lots of problems we don't know about yet.

Your proposal is much more fixed featured, and without changes to how a function becomes coroutinised, I don't think could implement co_await/co_yield/co_return. But more experienced folk on here may jump in with alternative opinions on that.

Now, ere you might be right. I do not know about coroutines that much.

Regards,

&rzej;

[Nicol Bolas]

On Friday, October 13, 2017 at 11:51:15 AM UTC-4, Andrzej Krzemieński wrote:

W dniu piątek, 13 października 2017 17:39:24 UTC+2 użytkownik Niall Douglas napisał:

Instead of "macro functions" consider the following language extension as an alternative. It appears to me less invasive, more focused, and seems to address your problem.

This idea looks very similar to Emil's alternative design to Outcome/Expected, https://zajo.github.io/boost-noexcept/. It uses a "side channel" or "out of band" channel for returning alternatives to the normal return.

Emil's library simply uses a thread-local storage. What I propose is just a construct that when compiled generates exactly same code as your macros. The "slot" is only for the purpose of describing the feature, when compiled, it will be replaced with a branch instruction and return.

 

I'm not fond of OOB based designs for this problem domain. Your proposal I think would also require changes to ABI, specifically mangling. That's a big ask for a fixed function feature with limited reusability.

I do not think it needs to affect ABI. Similarly to constexpr functions, you can require of functions with exported return slot that their body is visible in places where it is invoked. It is close to a macro, except that it is namespace-scoped and does adhere to name lookup rules of normal functions.

If you can get a pointer to such a function, then it has to be part of the ABI. Calling a pointer to a `constexpr` function is no longer a `constexpr` operation (unless the pointer itself is `constexpr`). But calling a pointer to one of these functions would still need to have that behavior.

So either it's part of the ABI, or you must forbid getting pointers to them. Which includes forbidding virtual functions of this type.

At which point, it becomes what I outlined earlier in the thread.

Also, there's one thing I really believe we should have: special syntax for calling such a function. I believe it is vitally important that the person reading code can know how it's going to behave. With the way you've suggested the syntax, calling such a function looks exactly like calling any other function. So the reader of the code has no real idea that there is a potentially hidden `return` statement somewhere.

We allow this for exceptions, because they're supposed to be for exceptional circumstances. These circumstances are not "exceptional"; they're "expected". And therefore, I say that we need to explicitly see all expected control flow in the program.

Lastly, you may want to consider how an exported return statement should work when called from a function that itself exports a return. I believe that the caller of such a function may sometimes want to have the inner exported return be exported to its own exported return. And sometimes, the caller will want the inner exported return value to be its own normal return value. Why?

Consider that there is a `try` function for `std::expected/optional` which unpacks the value or exports its non-value status. You might write a wrapper around this function that has similar behavior, but does some error logging. To do that, you have to return its return value, but also export its exported return value exactly as it exports it.

If you can't do that, you would have to re-implement the standard `try` function. That's not good, since that implementation could be complicated, and new `expected`-like types can be added to `std::try` that your new function would have to be expanded to handle.

It looks like it could solve the problem with operator TRY. I do not know if it can address the case of coroutines, as I do not fully understand them.

Coroutines as currently proposed require the co_* operators to be called from the coroutinised function. You cannot, for example, call co_await via a helper function without coroutinising the helper function. I find this greatly annoying personally when writing coroutine code. I feel myself reaching for C macros to wrap boilerplate around co_await rather than being bothered to do it properly and write up a new awaitable type just to hide boilerplate.

Native C++ macro functions solve that too. They solve a ton of stuff, including lots of problems we don't know about yet.

Your proposal is much more fixed featured, and without changes to how a function becomes coroutinised, I don't think could implement co_await/co_yield/co_return. But more experienced folk on here may jump in with alternative opinions on that.

Now, ere you might be right. I do not know about coroutines that much.

Don't feel bad about that; the original "macro" idea wasn't going to replace `co_*` either, for reasons that have been outlined earlier in the thread.

 

Regards,

&rzej;

[Niall Douglas]

This idea looks very similar to Emil's alternative design to Outcome/Expected, https://zajo.github.io/boost-noexcept/. It uses a "side channel" or "out of band" channel for returning alternatives to the normal return.

Emil's library simply uses a thread-local storage. What I propose is just a construct that when compiled generates exactly same code as your macros. The "slot" is only for the purpose of describing the feature, when compiled, it will be replaced with a branch instruction and return.

My issue was Emil's choice of design pattern i.e. the use of OOB. Not his implementation.

 

 

I'm not fond of OOB based designs for this problem domain. Your proposal I think would also require changes to ABI, specifically mangling. That's a big ask for a fixed function feature with limited reusability.

I do not think it needs to affect ABI. Similarly to constexpr functions, you can require of functions with exported return slot that their body is visible in places where it is invoked. It is close to a macro, except that it is namespace-scoped and does adhere to name lookup rules of normal functions.

My proposed native C++ macro functions have the same lookup rules as normal functions, and are explicitly banned from having linkage and thus cannot affect ABI.

Niall

[Niall Douglas]

Don't feel bad about that; the original "macro" idea wasn't going to replace `co_*` either, for reasons that have been outlined earlier in the thread.

Except your reasons made no sense, as I pointed out.

The paper's current formulation of C++ macro functions follow all the rules for sanity you laid out, and most definitely can replace all the co_* operators entirely.

Niall 

[Matthew Woehlke]

On 2017-10-13 12:04, Nicol Bolas wrote:
> Lastly, you may want to consider how an exported return statement should

> work when called from a function that *itself* exports a return. I believe

> that the caller of such a function may sometimes want to have the inner
> exported return be exported to its own exported return. And sometimes, the
> caller will want the inner exported return value to be its own normal
> return value.

Provided it's the *caller* that makes this decision, I already showed
how this can be accomplished.

--
Matthew

[Nicol Bolas]

On Friday, October 13, 2017 at 12:24:20 PM UTC-4, Niall Douglas wrote:

Don't feel bad about that; the original "macro" idea wasn't going to replace `co_*` either, for reasons that have been outlined earlier in the thread.

Except your reasons made no sense, as I pointed out.

You earlier said:

> If the compiler can spot a "co_*" call and do the coroutinisation

The language should not interact with calls to standard library functions in this fashion. The act of calling a library function should not radically modify a function; if a function's nature is going to be changed, it ought to be changed by an explicit language keyword, not the presence of a mere function call.

I don't know of any feature of C++ where the act of calling a library function is specially treated by the language. We have places where users aren't able to replicate the library's behavior, but that's very different from having the language build key features based on the use of a library function.

The paper's current formulation of C++ macro functions follow all the rules for sanity you laid out, and most definitely can replace all the co_* operators entirely.

Your macro idea doesn't follow my sanity rules. They can access the calling function's scope, and their declarations leak into the calling function.

[Nicol Bolas]

It would have to be the caller that decides this. It wouldn't make sense if the called function made the decision, since whether to export the inner return value is based entirely on being called from a function that exports a return.

If you're referring to this post, sure, you can build some library machinery to handle this. Or you could make it part of the language and have it be cheap and elidable. After all, we know that we want some kind of syntax at the call site to distinguish between calling functions which cannot export a return and calling functions which do. So we simply need to have two sets of syntax: one for "exported returns go into my return", and one for "exported returns are exported into my exported return."

Your `yield_or_return` mechanism break elision, and if exported return values are going to be part of the ABI, we really need them to work with guaranteed elision where possible.

 

--
Matthew

[Nicol Bolas]

On Friday, October 13, 2017 at 12:22:21 PM UTC-4, Niall Douglas wrote:

This idea looks very similar to Emil's alternative design to Outcome/Expected, https://zajo.github.io/boost-noexcept/. It uses a "side channel" or "out of band" channel for returning alternatives to the normal return.

Emil's library simply uses a thread-local storage. What I propose is just a construct that when compiled generates exactly same code as your macros. The "slot" is only for the purpose of describing the feature, when compiled, it will be replaced with a branch instruction and return.

My issue was Emil's choice of design pattern i.e. the use of OOB. Not his implementation.

Well, that rather depends on how you define "out of band". I, for example, use that term to represent a case where the error is not present in the function signature, where it's stored in something "out of band" from the call.

In Andrzej's idea, the error is directly tied to the signature. It's as much "in band" as returning `expected<T, E>`. You effectively are giving functions two "bands" of return values.

[Niall Douglas]

The language should not interact with calls to standard library functions in this fashion. The act of calling a library function should not radically modify a function; if a function's nature is going to be changed, it ought to be changed by an explicit language keyword, not the presence of a mere function call.

I agree up to the point that an explicit language keyword ought to be required to change a function's nature.

There are already plenty of function calls which change a function's nature. All the cmath ones for example change their nature when you change rounding etc.

So really it comes down to documentation. If some macro documents "this changes the calling function's nature by X, Y and Z" then we're good.

 

The paper's current formulation of C++ macro functions follow all the rules for sanity you laid out, and most definitely can replace all the co_* operators entirely.

Your macro idea doesn't follow my sanity rules. They can access the calling function's scope, and their declarations leak into the calling function.

No, they get their own macro-local scope with lifetime attached to the calling function's scope. Big difference (and you persuaded me of the wisdom of that change incidentally)

Niall

 

[Nicol Bolas]

OK, there's been a lot of discussion here, and we seem to have agreement on some points and some variation on others. I'm posting what I think is a summary of where the various discussions are at the present time. Feel free to add corrections if I'm mistaken.

For the purposes of this post, I'm going to call these "exporting functions". There seems to be agreement that:

1: Exporting functions are functions, not macros. They do not have access to the definitions in their callers, and they do not leak declarations into their callers, with a few very specific exceptions. Compilers can certainly inline them (and may be required to do so, per the stuff below), but they nominally behave the way we expect functions to behave.

2: Exporting functions use special syntax in their prototypes which marks them as being an exporting function.

Here are some things that either are in disagreement or haven't been really talked about much:

1: The specific suite of operations that an exporting function can export. There's broad agreement on "return", but nothing much has been discussed about other operations. It should be noted that the use cases outlined in the original paper are solved by just exporting `return`, not more esoteric operations like `continue` and `break`, so `return` may be all we need.

2: Whether the exported operation (including the type being export-returned) is a part of the function's signature. Also, if we require the exported-return type (and we should), we should of course allow placeholders, as for regular return types.

3: Should exporting functions be static or dynamic? By "dynamic", I mean can you get pointers to exporting functions and pass them around; can you make virtual exporting functions? Making them "dynamic" means that exporting functions become a first-class part of the compiler's ABI. Obviously if it's part of the ABI, then it must be part of the function's signature, along with the exported return type.

To restrict them to just "static" use means restricting them to being inline, visible to all translation units that use them. And of course, not being able to declare them `virtual` or getting function pointers to them in any way.

4: Should the invocation of an exporting function have special syntax, to make it visually distinct from calling functions which cannot export operations?

5: Should there be special syntax to invoke an exporting function within an exporting function, such that if it exports a return, the return is exported to the outer function's caller? This is somewhat orthogonal to #4, in that even if we don't require a syntax for export return cases, we could still add a syntax to cover this particular case. That is, even if we allow `try(expression)` to implicitly export a return, `export try(expression)` could be used to mean to export the exported return value from the `try(expression)` (resulting in the value of `expression` otherwise). And therefore, `return export try(expression)` would mean to export-return the exported return of `try(expression)`, and return the return result of `try(expression)` if it resulted in a regular value.

[Niall Douglas]

On Friday, October 13, 2017 at 8:10:07 PM UTC+1, Nicol Bolas wrote:

OK, there's been a lot of discussion here, and we seem to have agreement on some points and some variation on others. I'm posting what I think is a summary of where the various discussions are at the present time. Feel free to add corrections if I'm mistaken.

For the purposes of this post, I'm going to call these "exporting functions". There seems to be agreement that:

1: Exporting functions are functions, not macros. They do not have access to the definitions in their callers, and they do not leak declarations into their callers, with a few very specific exceptions. Compilers can certainly inline them (and may be required to do so, per the stuff below), but they nominally behave the way we expect functions to behave.

Agreed. I've chosen the term "macro functions" but the current formulation is mostly function, little macro.

I am semi-tempted to add the 'control flow keyword ->' markup to lambdas, but it's too important that these things get ADL discovery like functions.

 

2: Exporting functions use special syntax in their prototypes which marks them as being an exporting function.

Can we please use any adjective other than "export"?

Export makes me think of Modules. Or template export, the memory of which makes me queasy.

These functions don't export anything. They affect a caller's control flow.

 

Here are some things that either are in disagreement or haven't been really talked about much:

1: The specific suite of operations that an exporting function can export. There's broad agreement on "return", but nothing much has been discussed about other operations. It should be noted that the use cases outlined in the original paper are solved by just exporting `return`, not more esoteric operations like `continue` and `break`, so `return` may be all we need.

My macro function have the specific ability to call intrinsics as if they were the calling function. So, they could call setjmp() and be cast iron guaranteed that the frame stored is that if the caller.

The reason I mention setjmp is because the Coroutines TS only covers half the use cases for coroutines. The other half would enormously benefit from native C++ macro functions. Any coroutines implementation right now is using unwieldy and brittle C macros. Those could be made to permanently go away.

 

2: Whether the exported operation (including the type being export-returned) is a part of the function's signature. Also, if we require the exported-return type (and we should), we should of course allow placeholders, as for regular return types.

3: Should exporting functions be static or dynamic? By "dynamic", I mean can you get pointers to exporting functions and pass them around; can you make virtual exporting functions? Making them "dynamic" means that exporting functions become a first-class part of the compiler's ABI. Obviously if it's part of the ABI, then it must be part of the function's signature, along with the exported return type.

To restrict them to just "static" use means restricting them to being inline, visible to all translation units that use them. And of course, not being able to declare them `virtual` or getting function pointers to them in any way.

4: Should the invocation of an exporting function have special syntax, to make it visually distinct from calling functions which cannot export operations?

5: Should there be special syntax to invoke an exporting function within an exporting function, such that if it exports a return, the return is exported to the outer function's caller? This is somewhat orthogonal to #4, in that even if we don't require a syntax for export return cases, we could still add a syntax to cover this particular case. That is, even if we allow `try(expression)` to implicitly export a return, `export try(expression)` could be used to mean to export the exported return value from the `try(expression)` (resulting in the value of `expression` otherwise). And therefore, `return export try(expression)` would mean to export-return the exported return of `try(expression)`, and return the return result of `try(expression)` if it resulted in a regular value.

I really think an adjective other than "export" would greatly clarify the exposition above. I find the above confusing.

Niall

 

[Matthew Woehlke]

On 2017-10-13 15:10, Nicol Bolas wrote:
> 4: Should the invocation of an exporting function have special syntax, to
> make it visually distinct from calling functions which cannot export
> operations?

Yes, please. (Maybe there isn't "consensus" per-se on that point, but I
got the impression that the overall direction was leaning that way.)

> 5: Should there be special syntax to invoke an exporting function *within*
> an exporting function

Argh... I almost hate to suggest this, but (with respect to both above
points):

int ->foo()
{
...
// call
<-exported_function();
...
// call, and a hoisted return will be re-hoisted; this allows
// "tweaking" of what possibly-exported things are re-hoisted
<-exported_function() <-> return;
...
}

("Hatred" mainly due to the implied addition of "operators" `<-` and
`<->`, and all the grammar fun that brings...)

--
Matthew

[Nicol Bolas]

On Friday, October 13, 2017 at 3:48:29 PM UTC-4, Niall Douglas wrote:

On Friday, October 13, 2017 at 8:10:07 PM UTC+1, Nicol Bolas wrote:

OK, there's been a lot of discussion here, and we seem to have agreement on some points and some variation on others. I'm posting what I think is a summary of where the various discussions are at the present time. Feel free to add corrections if I'm mistaken.

For the purposes of this post, I'm going to call these "exporting functions". There seems to be agreement that:

1: Exporting functions are functions, not macros. They do not have access to the definitions in their callers, and they do not leak declarations into their callers, with a few very specific exceptions. Compilers can certainly inline them (and may be required to do so, per the stuff below), but they nominally behave the way we expect functions to behave.

Agreed. I've chosen the term "macro functions" but the current formulation is mostly function, little macro.

I am semi-tempted to add the 'control flow keyword ->' markup to lambdas, but it's too important that these things get ADL discovery like functions.

 

2: Exporting functions use special syntax in their prototypes which marks them as being an exporting function.

Can we please use any adjective other than "export"?

Export makes me think of Modules. Or template export, the memory of which makes me queasy.

These functions don't export anything. They affect a caller's control flow.

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far. And he uses the keyword "export". It's not the best term, but it's more reasonable for the current idea than "macro" or "inline". It's not so much that it's the best word, but it's the least bad thus far.

Also, "export" is a verb, so it's possible to talk about a function "exporting" something as a separate action from it "returning" something. If you can come up with a good, single-word term for "causes my caller to return this", feel free.

Here are some things that either are in disagreement or haven't been really talked about much:

1: The specific suite of operations that an exporting function can export. There's broad agreement on "return", but nothing much has been discussed about other operations. It should be noted that the use cases outlined in the original paper are solved by just exporting `return`, not more esoteric operations like `continue` and `break`, so `return` may be all we need.

My macro function have the specific ability to call intrinsics as if they were the calling function.

What is an "intrinsic"? That's not a concept defined by the standard.

You seem to say that `setjmp` is an "intrinsic". Which standard library functions are "intrinsics" and which are not? Can users write new "intrinsics," by this definition, or is that reserved solely to the standard library?

 

So, they could call setjmp() and be cast iron guaranteed that the frame stored is that if the caller.

The reason I mention setjmp is because the Coroutines TS only covers half the use cases for coroutines. The other half would enormously benefit from native C++ macro functions. Any coroutines implementation right now is using unwieldy and brittle C macros. Those could be made to permanently go away.

What "coroutines implementations" are you talking about that use C macros? I don't use the Coroutines TS, but I've seen some example code that does. I've yet to see any that needed macros in order to do their jobs.

Niall

[gmis...@gmail.com]

Can we please use any adjective other than "export"?

Export makes me think of Modules. Or template export, the memory of which makes me queasy.

These functions don't export anything. They affect a caller's control flow.

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far. And he uses the keyword "export". It's not the best term, but it's more reasonable for the current idea than "macro" or "inline". It's not so much that it's the best word, but it's the least bad thus far.

Also, "export" is a verb, so it's possible to talk about a function "exporting" something as a separate action from it "returning" something. If you can come up with a good, single-word term for "causes my caller to return this", feel free.

 

I had sketched out something to myself that I hadn't sent yet that was very much like what Andrzej suggested except in my sketch I'd used "inline return" as the keyword. But I have to say, I quite like "export return" though and I don't find it at all confusing with modules. I guess it might stand on some feature that modules might come up with in the future, but of course I can't imagine what right now.

requires return; anyone?

[Niall Douglas]

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far.

I hate to be pedantic, but remember P0779R0 proposes two methods of implementing operator try. The first, to remind you, is this:

template <class T, class E>
constexpr auto operator try(std::expected<T, E> v) noexcept
{
  struct tryer
  {
    std::expected<T, E> v;

    constexpr bool try_return_immediately() const noexcept { return !v.has_value(); }
    constexpr auto try_return_value() { return std::move(v).error(); }
    constexpr auto try_value() { return std::move(v).value(); }
  };
  return tryer{ std::move(v) };
}

If you just want to do "var = try expr" where expr is user definable types and be done with it, the above delivers that.
 

And he uses the keyword "export". It's not the best term, but it's more reasonable for the current idea than "macro" or "inline". It's not so much that it's the best word, but it's the least bad thus far.

Also, "export" is a verb, so it's possible to talk about a function "exporting" something as a separate action from it "returning" something. If you can come up with a good, single-word term for "causes my caller to return this", feel free.

Ideally I'd like to see no new keywords and no dual band or second band channelling.

 

What "coroutines implementations" are you talking about that use C macros? I don't use the Coroutines TS, but I've seen some example code that does. I've yet to see any that needed macros in order to do their jobs.

 

The Coroutines TS only standardises one kind of coroutine. Various experts think about half of coroutine using code can be made to use the Coroutines TS. The rest of coroutines are supposedly coming in a later standards proposal and will require even more injection of boilerplate into calling functions.

It's like range for functions. They should have been implemented as a foreach# macro function, not into expanding boilerplate in the language itself.

Niall

[Andrzej Krzemieński]

W dniu sobota, 14 października 2017 02:58:11 UTC+2 użytkownik Niall Douglas napisał:

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far.

I hate to be pedantic, but remember P0779R0 proposes two methods of implementing operator try. The first, to remind you, is this:

template <class T, class E>
constexpr auto operator try(std::expected<T, E> v) noexcept
{
  struct tryer
  {
    std::expected<T, E> v;

    constexpr bool try_return_immediately() const noexcept { return !v.has_value(); }
    constexpr auto try_return_value() { return std::move(v).error(); }
    constexpr auto try_value() { return std::move(v).value(); }
  };
  return tryer{ std::move(v) };
}

If you just want to do "var = try expr" where expr is user definable types and be done with it, the above delivers that.

If I may express my opinion (and this is just opinion -- I have nothing objective to back it up) I like your first solution best. The scope is clearly defined (only operation `try`), the implementation is clear, and it is also clear that the feature would be easy to use correctly and hard to use incorrectly.

 

And he uses the keyword "export". It's not the best term, but it's more reasonable for the current idea than "macro" or "inline". It's not so much that it's the best word, but it's the least bad thus far.

Also, "export" is a verb, so it's possible to talk about a function "exporting" something as a separate action from it "returning" something. If you can come up with a good, single-word term for "causes my caller to return this", feel free.

Ideally I'd like to see no new keywords and no dual band or second band channelling.

One can easily replace a keyword with a `#` or a `??`, similarly I could reword my proposal so that it does not suggest "channeling". It is not in these things that I think the heart of the disagreement lays.

 

What "coroutines implementations" are you talking about that use C macros? I don't use the Coroutines TS, but I've seen some example code that does. I've yet to see any that needed macros in order to do their jobs.

 

The Coroutines TS only standardises one kind of coroutine. Various experts think about half of coroutine using code can be made to use the Coroutines TS. The rest of coroutines are supposedly coming in a later standards proposal and will require even more injection of boilerplate into calling functions.

It's like range for functions. They should have been implemented as a foreach# macro function, not into expanding boilerplate in the language itself.

Ok, now I think I understand where you are going. I suppose the solution I proposed is just the worse version of your solution #1, and it would probably not be able to handle the coroutine usage. I understand that people here suggested that a "generic" solution is provided that can handle both `try` and coroutines and the future extensions, and that this facility would require for the programmer to be able to define their own control-flow instructions. Your implementation injects code, but theoretically this could be implemented differently as long as I am able to define my own custom control flow.

One way is to invert the control flow and this is what the functional languages do:

expected<T> o = fun();

o.on_success_call(x).on_failure_call(y);

But I understand that we want the end user to program in an imperative language. That's fine.

But while the ability to define custom flow control is powerful enough to solve the problems of operation `try`, coroutines, foreach statement and more, it has also power to introduce incomprehensible bugs. This is what I fear when using macros:

if (condition)
  LOG_WARNING(logger) << "bad condition";
else
  process();

if `condition` is false will `process()` be called? That depends on how macro `LOG_WARNING` is defined. If it is defined as:

#define LOG_WARNING(L) if(L.warning_enabled()) L.stream()




Then I have a bug that will be super-hard to find. And the solution with injecting code (even if it is not a macro) will expose this new kind of bugs for which we are not prepared. I suspect that it will be too easy to use it incorrectly.  With my proposal I tried to narrow down the generation of the new control flows to minimum, but I think it is not going to work.

Therefore I prefer that the new control statements are only added by experts and are introduced with keywords and standardized, so that it is only a handful of them to learn, and they are well explained and taught.

Regards,
&rzej;

[Viacheslav Usov]

On Fri, Oct 13, 2017 at 9:10 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> 1: The specific suite of operations that an exporting function can export. There's broad agreement on "return", but nothing much has been discussed about other operations. It should be noted that the use cases outlined in the original paper are solved by just exporting `return`, not more esoteric operations like `continue` and `break`, so `return` may be all we need.

The point that the whole proposal is nothing but exceptions in disguise has been beaten to death.

So why can't we just say these are indeed exceptions, albeit of a special kind?

The syntax for them in the "exporting function" is almost natural:

A. throw return foo;

B. throw break;

C. throw continue;

In case A, if the caller's return type is compatible with foo, the caller returns an appropriately converted value..

In case B or C, if the caller is a context where break or continue, respectively, is meaningful, it is effected.

In every other case, an exception of a special kind is thrown and handled in the normal way.

All of that can be implemented on top of whatever exception architecture a given implementation already has, as if by wrapping every function call with:

try

{

    bar();

}

catch (return_exception<return_type> &ex)

{

    return ex;

}

catch (break_exception &)

{

    break;

}

catch (continue_exception &)

{

    continue;

}

The break and continue catch clauses are generated only if appropriate in the current context, which is known at the compile time.

This works with separate translation units and can possibly be done without ABI changes.

Then each implementation is free to use an optimised dispatch of such exceptions. It is rather plain to see that modern auto-inlining whole-program-optimising implementations already have all the right tools in their repertoire to achieve that. if there are serious doubts that such an optimisation is possible, we might require that "exporting functions" carry a special decoration, to ensure that at least calls to "ordinary" functions are not negatively affected.

Cheers,

V.

[Nicol Bolas]

On Friday, October 13, 2017 at 8:58:11 PM UTC-4, Niall Douglas wrote:

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far.

I hate to be pedantic, but remember P0779R0 proposes two methods of implementing operator try.

Yes, but that's really not what we've been talking about. Discussion in this thread has been focused on "macro-functions" and ideas derived from it, not the "operator try" part.

Probably because it's far less interesting of an idea, even if it's more viable of a proposal.

 

The first, to remind you, is this:

template <class T, class E>
constexpr auto operator try(std::expected<T, E> v) noexcept
{
  struct tryer
  {
    std::expected<T, E> v;

    constexpr bool try_return_immediately() const noexcept { return !v.has_value(); }
    constexpr auto try_return_value() { return std::move(v).error(); }
    constexpr auto try_value() { return std::move(v).value(); }
  };
  return tryer{ std::move(v) };
}

If you just want to do "var = try expr" where expr is user definable types and be done with it, the above delivers that.
 

And he uses the keyword "export". It's not the best term, but it's more reasonable for the current idea than "macro" or "inline". It's not so much that it's the best word, but it's the least bad thus far.

Also, "export" is a verb, so it's possible to talk about a function "exporting" something as a separate action from it "returning" something. If you can come up with a good, single-word term for "causes my caller to return this", feel free.

Ideally I'd like to see no new keywords and no dual band or second band channelling.

Whether you want to think of it that way or not, "second band channelling" is exactly what happens when a "macro-function" forces its caller to return a particular value. Whether that band is directly visible in the function signature or not, it's still there.

What "coroutines implementations" are you talking about that use C macros? I don't use the Coroutines TS, but I've seen some example code that does. I've yet to see any that needed macros in order to do their jobs.

 

The Coroutines TS only standardises one kind of coroutine. Various experts think about half of coroutine using code can be made to use the Coroutines TS. The rest of coroutines are supposedly coming in a later standards proposal and will require even more injection of boilerplate into calling functions.

... Oh! You're not talking about "coroutines, the technical specification"; you're talking about "coroutines, the concept". Poor naming is yet another reason I hate the Coroutines TS.

OK, that makes a lot more sense now. To be more specific, what you're talking about is P0534: call/cc.

The thing is, you're wrong that such a coroutine mechanism requires "injection of boilerplate". Call/cc works based on whole function stacks; it does not care how many functions are in the way. If it suspends execution, it suspends execution of the entire call tree down to its root. If it resumes execution, it resumes execution of the entire call tree down to its root.

As such, await-style suspend/resume can be done via utility functions/types. Those functions don't have to be "injected" into a specific function's stack.

The closest to "injection" that call/cc requires is a feature that it already has: the ability to resume a stack, but doing so by pushing another function on top of that stack (so that you can do some testing/checking/whatever). When that function returns, its value is provided to the suspension code on top of the stack.

[Nicol Bolas]

On Saturday, October 14, 2017 at 9:15:45 AM UTC-4, Viacheslav Usov wrote:

On Fri, Oct 13, 2017 at 9:10 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> 1: The specific suite of operations that an exporting function can export. There's broad agreement on "return", but nothing much has been discussed about other operations. It should be noted that the use cases outlined in the original paper are solved by just exporting `return`, not more esoteric operations like `continue` and `break`, so `return` may be all we need.

The point that the whole proposal is nothing but exceptions in disguise has been beaten to death.

So why can't we just say these are indeed exceptions, albeit of a special kind?

Because that doesn't solve any of the use cases the OP wanted to use them for.

The issue with `try/catch` is that they're statements at the site of use. The principle, driving use case that the OP wants solved is to have a way to test the result of the expression, evaluating to either a value generated by that expression or issuing a return statement from the function that tested the expression. The purpose of this is to make error testing code via `expected` or similar types more natural and reasonable than having a lot of `if(blah) return unexpected(blah);` statements.

`try/catch` doesn't solve that problem.

 

[Viacheslav Usov]

On Sat, Oct 14, 2017 at 5:20 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Because that doesn't solve any of the use cases the OP wanted to use them for.

I am afraid you misunderstood everything that I wrote. You did that so thoroughly it is astonishing.

> The issue with `try/catch` is that they're statements at the site of use.

I did not suggest using 'try/catch'. I did say that the implementation of my proposal is logically equivalent to a try/catch as if emitted by the compiler at the site of use.

I'd suggest you read what I wrote originally again.

Cheers,

V.

[Niall Douglas]

If I may express my opinion (and this is just opinion -- I have nothing objective to back it up) I like your first solution best. The scope is clearly defined (only operation `try`), the implementation is clear, and it is also clear that the feature would be easy to use correctly and hard to use incorrectly.

Spending so much time with you must be rubbing off on me Andrzej!

I figured for the paper to give WG21 opposite extremes for a solution. Though I am told by Ville and JF that this "syntax change proposal" stands no chance without lots more proof of why it's needed.

And for that, well it's easier just to let the idiom become common practice due to a Boost.Outcome becoming extremely popular due to an excellently written tutorial :) If enough people starting arriving here and saying "why can't I write try expr like in all the other languages?" then we can point them at P0779R0 and tell them the committee didn't think it worthwhile back in 2017. Meanwhile, no doubt abuse of co_await will continue to rise. But then, this potential problem was raised now. Warning was given.

 

But while the ability to define custom flow control is powerful enough to solve the problems of operation `try`, coroutines, foreach statement and more, it has also power to introduce incomprehensible bugs. This is what I fear when using macros:

if (condition)
  LOG_WARNING(logger) << "bad condition";
else
  process();

if `condition` is false will `process()` be called? That depends on how macro `LOG_WARNING` is defined. If it is defined as:

#define LOG_WARNING(L) if(L.warning_enabled()) L.stream()




Then I have a bug that will be super-hard to find. And the solution with injecting code (even if it is not a macro) will expose this new kind of bugs for which we are not prepared. I suspect that it will be too easy to use it incorrectly.  With my proposal I tried to narrow down the generation of the new control flows to minimum, but I think it is not going to work.

One of the major value adds of my proposed macro functions is that they can't suffer from problems like the above. Injection of control flow is predictable.

But maybe I ought to spell it out more with examples e.g. implement range for loops and so on. I'm on full time childcare as usual this weekend, but I'll see how I feel after the kids are in bed tonight. I only got four hours of sleep, so I am not hopeful.

Niall

[Niall Douglas]

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far.

I hate to be pedantic, but remember P0779R0 proposes two methods of implementing operator try.

Yes, but that's really not what we've been talking about. Discussion in this thread has been focused on "macro-functions" and ideas derived from it, not the "operator try" part.

Probably because it's far less interesting of an idea, even if it's more viable of a proposal.

Dunno, JF thinks it's dead on arrival without lots more proof of its need. See https://groups.google.com/a/isocpp.org/d/msg/sg14/6L5h2OKg75U/UDcY_ZgGAwAJ

 

OK, that makes a lot more sense now. To be more specific, what you're talking about is P0534: call/cc.

Actually, I'm not. Oliver who proposed that is a fellow Booster. We know each other. That proposal was one of many of a rearguard action by domain experts in coroutines to dissuade WG21 of adopting the Coroutines TS due to it being inferior to library based alternatives around a very minimal compiler intrinsics API. Indeed I may have commented on the design of that API at one point. The dissuasion failed, some would feel for political not technical reasons.

But little birdies tell me that work is ongoing to complement the Coroutines TS with something "more useful" (not my words) by filling in the gaps. That's not my endeavour, my interaction with that group has mainly been to coordinate the coroutinised i/o in AFIO with what they're planning, and even with that it is mostly to tell them that coroutinised i/o is pointless, and not useful on modern storage. Still, AFIO supports it for those who want it, with a big warning sign in the docs saying "don't use this, it's slower".

 

The thing is, you're wrong that such a coroutine mechanism requires "injection of boilerplate". Call/cc works based on whole function stacks; it does not care how many functions are in the way. If it suspends execution, it suspends execution of the entire call tree down to its root. If it resumes execution, it resumes execution of the entire call tree down to its root.

As such, await-style suspend/resume can be done via utility functions/types. Those functions don't have to be "injected" into a specific function's stack.

The closest to "injection" that call/cc requires is a feature that it already has: the ability to resume a stack, but doing so by pushing another function on top of that stack (so that you can do some testing/checking/whatever). When that function returns, its value is provided to the suspension code on top of the stack.

All of what you just said would be disproved by the several coroutines implementations in Boost. They all use C macros or template based mechanisms of injecting boilerplate. Extensively.

You may also find https://github.com/jamboree/co2 of interest. It's a library and C macro implementation of the Coroutines TS. No language changes needed. If we had the native C++ macro functions I proposed, it actually could entirely replace the Coroutines TS entirely.

Niall

[Niall Douglas]

> Because that doesn't solve any of the use cases the OP wanted to use them for.

I am afraid you misunderstood everything that I wrote. You did that so thoroughly it is astonishing.

In fairness, it took me several readings to understand what you were actually saying. What you are actually saying has nothing to do with what you appear to be saying. 

Niall

[Viacheslav Usov]

On Sat, Oct 14, 2017 at 6:30 PM, Niall Douglas <nialldo...@gmail.com> wrote:

> In fairness, it took me several readings to understand what you were actually saying. What you are actually saying has nothing to do with what you appear to be saying. 

That is unfortunate, but I have had the same problem in this and in an earlier thread on the same issue. I'll try to rephrase:

In a recent analysis (to which I responded), Nicol established that an essential part of your proposal is a way for a function to inject a return statement into the caller function; and possibly also break and continue statements. These injected, and also referred to as exported, control-flow statements execute in the caller function (Caller), as if they were executed at the site of the call of the injecting/exporting function (Injector).

I proposed that Injector could use the following syntax to achieve such injection:

A. throw return foo;

B. throw break;

C. throw continue;

Omitting details that can be consulted with in my original message, (A), (B) & (C) inject a return, a break, and a continue, respectively, into Caller.

Because it has caused confusion, I specifically stress that Caller does not need to use try/catch to allow injection to happen.

Cheers,

V.

[Nicol Bolas]

On Saturday, October 14, 2017 at 12:27:33 PM UTC-4, Niall Douglas wrote:

This is getting into bikeshedding, but I picked "export" because Andrzej's post represents the most fully formed post on this thread that represents our current thinking thus far.

I hate to be pedantic, but remember P0779R0 proposes two methods of implementing operator try.

Yes, but that's really not what we've been talking about. Discussion in this thread has been focused on "macro-functions" and ideas derived from it, not the "operator try" part.

Probably because it's far less interesting of an idea, even if it's more viable of a proposal.

Dunno, JF thinks it's dead on arrival without lots more proof of its need. See https://groups.google.com/a/isocpp.org/d/msg/sg14/6L5h2OKg75U/UDcY_ZgGAwAJ

OK, that makes a lot more sense now. To be more specific, what you're talking about is P0534: call/cc.

Actually, I'm not. Oliver who proposed that is a fellow Booster. We know each other. That proposal was one of many of a rearguard action by domain experts in coroutines to dissuade WG21 of adopting the Coroutines TS due to it being inferior to library based alternatives around a very minimal compiler intrinsics API. Indeed I may have commented on the design of that API at one point. The dissuasion failed, some would feel for political not technical reasons.

Coroutines isn't in C++17 or 20 yet, so it's hard to say that it has "failed".

But little birdies tell me that work is ongoing to complement the Coroutines TS with something "more useful" (not my words) by filling in the gaps.

It better still be ongoing. We still need actual coroutines in C++ ;).

call/cc (or any similar stackful solution) is such a different thing from the Coroutines TS that there isn't really competition between them. They're just such fundamentally different constructs, with different use cases and the like. The places where I would want to use call/cc are not the places where I would want to (or even be able to) use Coroutines TS. And vice versa.

 

That's not my endeavour, my interaction with that group has mainly been to coordinate the coroutinised i/o in AFIO with what they're planning, and even with that it is mostly to tell them that coroutinised i/o is pointless, and not useful on modern storage. Still, AFIO supports it for those who want it, with a big warning sign in the docs saying "don't use this, it's slower".

 

The thing is, you're wrong that such a coroutine mechanism requires "injection of boilerplate". Call/cc works based on whole function stacks; it does not care how many functions are in the way. If it suspends execution, it suspends execution of the entire call tree down to its root. If it resumes execution, it resumes execution of the entire call tree down to its root.

As such, await-style suspend/resume can be done via utility functions/types. Those functions don't have to be "injected" into a specific function's stack.

The closest to "injection" that call/cc requires is a feature that it already has: the ability to resume a stack, but doing so by pushing another function on top of that stack (so that you can do some testing/checking/whatever). When that function returns, its value is provided to the suspension code on top of the stack.

All of what you just said would be disproved by the several coroutines implementations in Boost. They all use C macros or template based mechanisms of injecting boilerplate. Extensively.

I was referring to the use of "call/cc" and similar things. That is, regular function calls and wrappers work just fine with that abstraction; you don't need to "inject" anything anywhere to use them.

How something gets implemented is essentially irrelevant. If using macros makes an implementation a bit more convenient... so be it. Macros are a part of the standard precisely because they can be convenient.

You seem to take "using C macros" as a priori wrong. I don't. It's just another tool the language gives you.

 

You may also find https://github.com/jamboree/co2 of interest. It's a library and C macro implementation of the Coroutines TS. No language changes needed.

It took me less than 5 seconds to find exactly what I expected to find: macros bracketing the function body. Note that the Coroutines TS includes no such thing; not even a perfunctory `async` on the function declaration.

So CO2 is still just a hack.

If we had the native C++ macro functions I proposed, it actually could entirely replace the Coroutines TS entirely.

You keep saying this, but you never address the salient point I keep bringing up. And CO2 provides a perfect example.

CO2 requires that you bracket the coroutine function with macros because it needs to insert stuff at the top and bottom of the function body. The Coroutines TS has no such bracketing. It injects the boilerplate around the function body by magic; by simply detecting that `co_*` have been used, it causes these things to appear.

Macros cannot do that. A macro invoked within a function cannot inject stuff at the top and bottom of it. And no version of your C++ macros can do that either. Therefore, they cannot "replace the Coroutines TS entirely".

Oh, you can make a version of the general design of the Coroutines TS with this feature and library stuff alone. But it would be a much more cumbersome feature to use, requiring a lot more manual work and with added fragility.

Much like attempts to make lambdas in C++ through a library. Which was prime justification for us getting lambdas as a language feature.

[Nicol Bolas]

OK, but that doesn't explain how one can tell, from reading the code, that a return can be injected at a particular location. That is, there needs to be some syntax at the call site for the caller allowing it. Plus, we still want to have the injected operation be part of the function's signature.

And we need a way for a function to "pass through" an injected operation, so that they can nest. And much like the above, it needs to be clear at the point of invocation that such a pass through is happening.

While exceptions might be a good conceptual fit, they are a poor fit syntactically. Sure, `throw return X;` is adequate as a syntax for injecting a return, but the rest of the `throw`-style machinery (`try` and `catch`) works poorly syntactically at the site of reception and in the function declaration.

V.

[Niall Douglas]

Final edition of the paper just posted to WG21 is attached. Unless it is rejected by John Spicer, this is the paper submitted.

You will note many changes have occurred since the previous draft. My thanks to all of you for your feedback, it was valuable and made this paper much better for it.

Niall

[Viacheslav Usov]

On Sun, Oct 15, 2017 at 6:29 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> OK, but that doesn't explain how one can tell, from reading the code, that a return can be injected at a particular location. That is, there needs to be some syntax at the call site for the caller allowing it. 

"A return can be injected" is exactly equivalent to "the callee may throw". Somehow we do just fine without any syntax allowing exceptions at the call site. What makes this different?

> Plus, we still want to have the injected operation be part of the function's signature.

Why? We have tried and rejected that with exceptions.

> And we need a way for a function to "pass through" an injected operation, so that they can nest. And much like the above, it needs to be clear at the point of invocation that such a pass through is happening.

Such nested use has not been motivated as far as I can tell. So it is not obvious to me we need that.

Yet I think we can also have that. We just need to say that we can also inject a throw, then 'throw throw return foo' injects 'throw return foo' into the caller.

> While exceptions might be a good conceptual fit, they are a poor fit syntactically. Sure, `throw return X;` is adequate as a syntax for injecting a return, but the rest of the `throw`-style machinery (`try` and `catch`) works poorly syntactically at the site of reception and in the function declaration.

I do not understand what you mean by that. The rest of the 'machinery' is not mandatory in what I proposed, it was used only as an illustration. Even the throw keyword is not essential, we might as well use return to the same effect, as in:

return return foo;

return break;

return continue;

Cheers,

V.

[Todd Fleming]

The final edition has a couple claims about coroutines that I don't think are true:

• "the function’s return type must be an awaitable". Not if the function's callers don't co_await on it.
  
• "you cannot await on something different to the coroutine return type". This restriction, if it were true, would hamper both the async and generator use cases.
  

Todd

 

[Nicol Bolas]

To expand on this:

1) A coroutine's return type does not have to be awaitable. The requirement of the coroutine return type is that there must exist coroutine machinery to translate the function's signature into a promise type. And while this is usually just a 1:1 mapping between the return type and the promise (`future<T>` translates to `promise<T>`), the machinery is passed the function's full signature, so it could in theory use parameter types as well.

Now, the return type usually is awaitable, since coroutines exist primarily to deal with asynchronous return values. But it doesn't have to be. Generators, for example, would not be awaitable types, since their values are not assumed to be asynchronous.

2) It's not the return value that must be the same as what you `co_await` on. What has to happen is that what you `co_await` on must be compatible with the promise machinery that is being used in the coroutine. For example, let's say you have a coroutine function that returns a `future<T>`. Well, you can `co_await` on any `future<U>` value just fine, since the `future`-based coroutine machinery will detect the promise/future type and use `.then` to schedule the continuation. However, you could also `co_await` on some non-`std::future` by giving your type `co_await` machinery that will use `.then` continuations for `std::future`.

But this also means that incompatible machinery is incompatible. A `generator` coroutine cannot `co_await` on a `future`, since `generator` coroutines are intended to resolve to values. You could write a specialized `future_generator` that could be compatible with `co_await`ing on a `future`, but this would be more expensive than a `generator`.

 

[Niall Douglas]

The final edition has a couple claims about coroutines that I don't think are true:

• "the function’s return type must be an awaitable". Not if the function's callers don't co_await on it.
  
• "you cannot await on something different to the coroutine return type". This restriction, if it were true, would hamper both the async and generator use cases.
  

On the first point, I was not aware that an awaitable is strictly something upon which co_await can be directly called. I thought it included anything which could be annotated in order to turn it into something upon which co_await can be called.

On the second, yes it's sloppy language. And as always in C++, with enough perseverance anything is possible. Doesn't make it wise though.

Niall