About captured arrays in lambda expressions

[Vlad from Moscow]

It looks strange enough that arrays captured by copy in lambda expressions are not implicitly converted to a pointer to the first element. What is the reason that the common rule about the implicit conversion of arrays was broken?

[Nicol Bolas]

On Monday, January 14, 2013 6:32:39 PM UTC-8, Vlad from Moscow wrote:

It looks strange enough that arrays captured by copy in lambda expressions are not implicitly converted to a pointer to the first element. What is the reason that the common rule about the implicit conversion of arrays was broken?

They didn't break the rule, because there is no rule that says that arrays always decay to pointers. The rule is that arrays decay to pointers under certain specific circumstances. Lambda expressions represented new circumstances not covered by those rules. Thus they could invent new rules.

Consider this code:

int arr[5];

[=] {arr[2] = 4;}();

Nowhere in this code does it explicitly do anything that would cause `arr` to decay into a pointer. It doesn't pass it to a function (in a visual sense). It doesn't explicitly convert it into a pointer. From a user perspective, there is no reason to expect `arr` to be anything other than exactly what it appears to be: `int arr[5]`.

The constructor to this functor causes the array parameter to decay to a pointer, but the contents of the functor is still an array. It simply copies the elements from the pointer into the internal array.

It's no different from this:

struct Functor
{
  Functor(int _arr[]) {std::copy(std::begin(arr), std::end(arr), _arr);}

  int arr[5];

  void operator()() {arr[2] = 4;}
}

The functor class stores the exact type of what it captures.

Furthermore, there's the obvious reason: you wouldn't be able to reference them by value. Therefore, the above code would be highly dangerous (if you returned the lambda, for example), and you could never trust [=] to do the right thing if you use an array anywhere.

[Vlad from Moscow]

вторник, 15 января 2013 г., 8:10:46 UTC+4 пользователь Nicol Bolas написал:

Well, I read you explanation but I did not find arguments that prove that  arrays captured by copy shall not be implicitly converted to a pointer. That is I did not see "new  circumstances". It would be more natural and expected if arrays were implicitly converted to pointers. If a user need a reference to array it can capture it by reference.

Let consider the first example

 

.

int arr[5];

[=] {arr[2] = 4;}();

I do not see here any problem if the array would be converted to pointer. You even write the following code

 

const int N = 5;

int arr[N];

[=] { std::fill( arr, arr + N, 4 ); }();

 

As N is not ODR-used  it can be present in the lambda expression without capturing. I think that it is very ineffective when huge arrays will be copied. Of course I coulod explicitly specify a pointer instead of an array but this requires additional declarations and definitions of objects.  If arrays were implicitly converted to pointer we would have unified  approach.

 

It is only the lambda expressions that break the rule of implicit converting arrays to pointers when arrays ODR-used.

[Alberto Ganesh Barbati]

Il giorno 15/gen/2013, alle ore 15:08, Vlad from Moscow <vlad....@mail.ru> ha scritto:

It would be more natural and expected if arrays were implicitly converted to pointers. If a user need a reference to array it can capture it by reference.

If array were converted to pointers, capturing by-copy would effectively be *the same as* capturing by-reference.

const int N = 5;

int arr[N];

[=] { std::fill( arr, arr + N, 4 ); }();

 

As N is not ODR-used  it can be present in the lambda expression without capturing. I think that it is very ineffective when huge arrays will be copied. Of course I coulod explicitly specify a pointer instead of an array but this requires additional declarations and definitions of objects.  If arrays were implicitly converted to pointer we would have unified  approach.

If you don't want the copy, then why are you capturing by-copy? It just doesn't make sense! Just capture by-reference. On the other hand, if you insist in thinking that capture by-copy should copy just the pointer (and not the elements), then think about how much extra code would you need to write, in order to emulate a capture that actually copies the elements.

Ganesh

[Vlad from Moscow]

вторник, 15 января 2013 г., 18:08:29 UTC+4 пользователь Vlad from Moscow написал:

In my opinion this question should be considered by the Committee as a defect of the C++ Standard. Lambda expressions were introduced as alternative of functional objects based on old classes std::unary_function and std::binary_function that to create functional objects on the flight. The main application of lambda expressions is their usage in algorithms. I never saw the code where a functional object is built such a way that it would copy a whole array in its constructor. It is very ineffective and has no a greate sense for functional objects. 

[Vlad from Moscow]

вторник, 15 января 2013 г., 18:59:11 UTC+4 пользователь Alberto Ganesh Barbati написал:

In fact you statement has the same sense as passing an array by value as a function argument is the same as passing the same array by reference. Maybe in this case we should abolish either passing arrays by value or passing arrays by reference? What do you think? 

[Nevin Liber]

On 15 January 2013 08:59, Alberto Ganesh Barbati <alberto...@gmail.com> wrote:

If you don't want the copy, then why are you capturing by-copy?

+1.

Plus, we have enough of a "gotcha" that member variables are (effectively) only capturable by reference; we don't need more of that.

--
 Nevin ":-)" Liber  <mailto:ne...@eviloverlord.com>  (847) 691-1404

[Nevin Liber]

On 15 January 2013 09:06, Vlad from Moscow <vlad....@mail.ru> wrote:

In my opinion this question should be considered by the Committee as a defect of the C++ Standard.

Not fitting your mental model is not a defect in the Standard.

[Nevin Liber]

On 15 January 2013 09:13, Vlad from Moscow <vlad....@mail.ru> wrote:

In fact you statement has the same sense as passing an array by value as a function argument is the same as passing the same array by reference. Maybe in this case we should abolish either passing arrays by value or passing arrays by reference? What do you think? 

I wish we could, but I believe that breaks C compatibility, and it would most would certainly break a lot of code out there.  Adopting your proposal (and it is not a defect) besides being undesirable for most of us, would also be a breaking change.

[Vlad from Moscow]

вторник, 15 января 2013 г., 19:22:42 UTC+4 пользователь Nevin ":-)" Liber написал:

I would prefer to have unified approach. And I do not think that this break a lot of code because I doubt that somebody wrote such functional objects that copy the whole array. It could be done only in case when the coder did not know that the array that he captured by default with specifier = will be copied  in whole within the lambda expression. At least I did not see such a code. And if such code exists I am sure it could be rewritten in more effective way.

[Ville Voutilainen]

On 15 January 2013 17:17, Nevin Liber <ne...@eviloverlord.com> wrote:
> On 15 January 2013 09:06, Vlad from Moscow <vlad....@mail.ru> wrote:
>> In my opinion this question should be considered by the Committee as a
>> defect of the C++ Standard.
> Not fitting your mental model is not a defect in the Standard.

The by-value capture initialization is specified to perform
direct-initialization. It's not attempting
to be equivalent to a function parameter, it's equivalent to what
happens when you copy class
members, so copying something like

struct foo
{
int foo[42];
};

will result in the copy of foo having newval.foo direct-initialized
from original.foo. Similarly, capturing
an array by value will result in an array being direct-initialized
from the captured value. I don't think
it should decay, and I don't think it's a defect. Lambda captures in
general attempt to avoid changing
the type of the original value captured (except adding const for
non-mutable lambdas), and decaying
would be an exception to that rule.

[Ville Voutilainen]

On 15 January 2013 17:30, Vlad from Moscow <vlad....@mail.ru> wrote:
> I would prefer to have unified approach. And I do not think that this break

Unified with member initialization or argument passing? Currently it's unified
with the former, you want it to be unified with the latter. The
captures are effectively
members, so why shouldn't they follow the rules of member initialization? Why
would they follow the rules of argument passing?

[Vlad from Moscow]

вторник, 15 января 2013 г., 19:30:53 UTC+4 пользователь Ville Voutilainen написал:

This explanation indeed has a sense if to forget the main application of lambda expressions. Their main application is the usage them with standard algorithms. I do not think that such functional obejct as struct foo has a greate sense taking into account that usually arrasy have sizes that much greater than 42. In this case I would prefer to capture an array by reference.

[Ville Voutilainen]

On 15 January 2013 17:40, Vlad from Moscow <vlad....@mail.ru> wrote:
> This explanation indeed has a sense if to forget the main application of
> lambda expressions. Their main application is the usage them with standard
> algorithms. I do not think that such functional obejct as struct foo has a
> greate sense taking into account that usually arrasy have sizes that much
> greater than 42. In this case I would prefer to capture an array by
> reference.

Well, as people have said, by all means capture by reference. Capturing by value
has performance implications for a far wider range of types than just arrays,
so why should arrays get special treatment there?

[Vlad from Moscow]

вторник, 15 января 2013 г., 19:40:41 UTC+4 пользователь Vlad from Moscow написал:

And I would like to append that this explanation is reasonable if I would have a built functional object and were going to copy it . But the situation is different. It is we who are building the functional object ourselves. So I do not see a greate sense to build a functional object such a way. 

[Vlad from Moscow]

вторник, 15 января 2013 г., 19:43:01 UTC+4 пользователь Ville Voutilainen написал:

Because arrays already have the special treatment and such treatment is usual. 

[Ville Voutilainen]

On 15 January 2013 17:51, Vlad from Moscow <vlad....@mail.ru> wrote:
>> Well, as people have said, by all means capture by reference. Capturing by
>> value
>> has performance implications for a far wider range of types than just
>> arrays,
>> so why should arrays get special treatment there?
> Because arrays already have the special treatment and such treatment is
> usual.

Arrays have no special treatment for member initialization. Also, note that
if you return a by-value-capturing lambda, or store it as a member and exit
the scope that created it, and it captured an array, your lambda now contains
a pointer to potentially destroyed data. Do you really want that?

[Vlad from Moscow]

вторник, 15 января 2013 г., 20:07:45 UTC+4 пользователь Ville Voutilainen написал:

What you have described has place in the current realization of lambda expressions.

 

Consider the following code

 

int a[10];

 

lm = [=]( int x ) { return ( a[4] = x, a ); };

 

int *p = lm( 10 );

 

We have gotten a pointer to a lambda scope array.

 

So this approach that is capture arrays by coping only confuses ysers.; 

[Nevin Liber]

On 15 January 2013 09:40, Vlad from Moscow <vlad....@mail.ru> wrote:

This explanation indeed has a sense if to forget the main application of lambda expressions. Their main application is the usage them with standard algorithms.

I've found that when passing lambdas directly to algorithms, 99% of the time I capture everything by reference.  Why do you find the need to capture by value for passing to algorithms? 

Also, while algorithms are *your* main (only?) use case, it most certainly isn't the only important use case.  Another major use case for lambdas is for deferred callbacks, which is important in the asynchronous and multithreaded world, and value semantics are an indispensable way of making sure the object lifetime is long enough for the callback to be executed.

[Ville Voutilainen]

On 15 January 2013 20:32, Vlad from Moscow <vlad....@mail.ru> wrote:
> What you have described has place in the current realization of lambda
> expressions.
> Consider the following code
> int a[10];
> lm = [=]( int x ) { return ( a[4] = x, a ); };
> int *p = lm( 10 );
> We have gotten a pointer to a lambda scope array.

The implementation I use diagnoses the return of the address of a
local variable, so I don't
know what the problem is supposed to be.

> So this approach that is capture arrays by coping only confuses ysers.;

The approach that capturing by value doesn't copy would confuse them even more.

Feel free to propose a change. Be ready for opposition.

[Sylvester Hesp]

On Tuesday, January 15, 2013 7:32:17 PM UTC+1, Vlad from Moscow wrote:

What you have described has place in the current realization of lambda expressions.

 

Consider the following code

 

int a[10];

 

lm = [=]( int x ) { return ( a[4] = x, a ); };

 

int *p = lm( 10 );

 

We have gotten a pointer to a lambda scope array.

 

So this approach that is capture arrays by coping only confuses ysers.; 

I can't speak for everyone, but I would be surprised if capturing an array by value (e.g., using [=]{...}) would actually capture by reference. I was also confused by the fact that capturing a class member by value didn't actually capture by value as it was referenced through 'this' (even though the lambda doesn't use 'this' in it's body). Luckily that last part is being addressed.

But say for argument's sake that arrays should indeed need to decay into a pointer. How would you propose someone would capture the entire array by value? Or would that be an impossibility after your suggested change, like capturing members by value is an impossibility with the current state of affairs?

- Sylvester

[Nevin Liber]

On 15 January 2013 17:20, Sylvester Hesp <s.h...@oisyn.nl> wrote:

I was also confused by the fact that capturing a class member by value didn't actually capture by value as it was referenced through 'this' (even though the lambda doesn't use 'this' in it's body). Luckily that last part is being addressed.

What paper addresses that?

[Richard Smith]

On Tue, Jan 15, 2013 at 3:20 PM, Sylvester Hesp <s.h...@oisyn.nl> wrote:
>
> On Tuesday, January 15, 2013 7:32:17 PM UTC+1, Vlad from Moscow wrote:
>>
>> What you have described has place in the current realization of lambda
>> expressions.
>>
>> Consider the following code
>>
>> int a[10];
>>
>> lm = [=]( int x ) { return ( a[4] = x, a ); };
>>
>> int *p = lm( 10 );
>>
>> We have gotten a pointer to a lambda scope array.
>>
>> So this approach that is capture arrays by coping only confuses ysers.;
>
>
> I can't speak for everyone, but I would be surprised if capturing an array
> by value (e.g., using [=]{...}) would actually capture by reference. I was
> also confused by the fact that capturing a class member by value didn't
> actually capture by value as it was referenced through 'this' (even though
> the lambda doesn't use 'this' in it's body). Luckily that last part is being
> addressed.

I concur on both counts. Also, given:

int arr[10];

We know that "[=] { return sizeof(arr); } ();" returns sizeof(int) *
10, because 'arr' is not captured. It would be extremely disturbing if
"[=] { arr[0] = 0; return sizeof(arr); } ();" returned sizeof(int*)
instead.

[Nicol Bolas]

On Tuesday, January 15, 2013 3:32:20 PM UTC-8, Nevin ":-)" Liber wrote:

On 15 January 2013 17:20, Sylvester Hesp <s.h...@oisyn.nl> wrote:

I was also confused by the fact that capturing a class member by value didn't actually capture by value as it was referenced through 'this' (even though the lambda doesn't use 'this' in it's body). Luckily that last part is being addressed.

What paper addresses that?

N3424 (PDF)

From the Evolution Paper Status publication, EWG didn't much care for the changes to const vs. mutable, but they wanted to take the changes on this capturing.
 

[Ville Voutilainen]

On 16 January 2013 01:41, Nicol Bolas <jmck...@gmail.com> wrote:
>>> I was also confused by the fact that capturing a class member by value
>>> didn't actually capture by value as it was referenced through 'this' (even
>>> though the lambda doesn't use 'this' in it's body). Luckily that last part
>>> is being addressed.
>> What paper addresses that?
> N3424 (PDF)
> From the Evolution Paper Status publication, EWG didn't much care for the
> changes to const vs. mutable, but they wanted to take the changes on this
> capturing.

Paraphrasing much? That's not what the EWG said, nor is it what the paper status
says. The EWG is encouraging the paper authors to submit a new paper proposing
potential changes for this/member capturing, but that's still at least
4 steps away
from making an actual change (pass through EWG, get the wording right in CWG(*),
pass a full committee vote, pass the scrutiny of committee experts not
participating the meeting
but reviewing the changes post-meeting).

(*) This one is rarely a single and trivial step.

[Nicol Bolas]

I never said that they approved it or had somehow incorporated it into the next standard. I said, "they wanted". If they didn't want the `this` changes, they'd have rejected the entire paper. So it's not unreasonable to conclude by the fact that they didn't dismiss it out of hand that they consider the idea reasonable in principle. That is, it passed the smell test (which the `const` stuff did not) and is something they'd be interested in seeing done.

I don't see how what I said was particularly misleading.

[Vlad from Moscow]

среда, 16 января 2013 г., 3:20:45 UTC+4 пользователь Sylvester Hesp написал:

Let consider a simple example. Assume that we need to write an unary predicate in the frames of the C++2003 Standard. It could look the following way

 

 

class UnaryPredicate : public std::unary_function<int, bool>
{
public:
 UnaryPredicate( const int a[] ) : a( a ) {}
 bool operator ()( int x ) { return ( x == *a++ ); }
protected:
 const int *a;
};

Now if we would rewtite it as a lambda expression it would be natural to suppose that the equivalent lambda expression will look the following way

 

[a]( int x ) mutable { return ( x == *a++ ); }

 

where a is some one-dimensional integer array.

 

Here [a] corresponds to  UnaryPredicate( const int a[] ) : a( a ) {}

and  return ( x == *a++ );  corresponds to  bool operator ()( int x ) { return ( x == *a++ ); }

 

I will repeat that this supposing is natural.

 

However this code will not be compiled in the frames of the C++ 2011 Standard. Instead of implicitly converting the array to a pointer the compiler will copy the whole array. Do I need this?! I I would like to deal with the whole array I would write [&a] and use sizeof( a ) if it is very needed.

 

Ville says that here arrays captured by copy the same way as it would be done when one cstructure is assigned to other structure. But here there is no any assignments of structures or using a default copy constructor. Here an array itself is used and arrays has no such operation as the assignment.

 

I do not know where it can be required for standard algorithms to copy the whole array in a functional object. At least till now I did not see such code. I think that such functional object will be ineffective. If you indeed need to copy a whole array and make it a local object then you can write a separate function. Lambda expressions should not be overloaded with such resourse consuming operations.

 

Returning to the example I showed that to get the same rresult I need to introduce an intermediate variable. This makes the code less obvious becuse the link between the array and lambda will be lost. For example

 

int a[N] = { /* some values */ };

int *p = a;

 

auto lm = [p]( int x ) mutable { return ( x == *p++ ); };

 

So every time some intermediate variable has to be used. It hides the linkage between the lambda and the array.

So I cannot get equivalent code for a C++ 2003 functional object without some tricks.

[Ville Voutilainen]

On 16 January 2013 15:40, Vlad from Moscow <vlad....@mail.ru> wrote:
> class UnaryPredicate : public std::unary_function<int, bool>
> {
> public:
> UnaryPredicate( const int a[] ) : a( a ) {}
> bool operator ()( int x ) { return ( x == *a++ ); }
> protected:
> const int *a;
> };
> Now if we would rewtite it as a lambda expression it would be natural to
> suppose that the equivalent lambda expression will look the following way
> [a]( int x ) mutable { return ( x == *a++ ); }
> where a is some one-dimensional integer array.
> Here [a] corresponds to UnaryPredicate( const int a[] ) : a( a ) {}
> and return ( x == *a++ ); corresponds to bool operator ()( int x ) {
> return ( x == *a++ ); }
> I will repeat that this supposing is natural.

You can repeat it as many times you like, but you say it's natural
that "a[]" corresponds to "a".
In your view, it would be unnatural if "a[]" corresponds to "&a".

> Ville says that here arrays captured by copy the same way as it would be
> done when one cstructure is assigned to other structure. But here there is
> no any assignments of structures or using a default copy constructor. Here
> an array itself is used and arrays has no such operation as the assignment.

The array value capture uses a similar "default copy constructor" as copying
an array member of a struct would. I have never claimed it has anything to
do with assignment, it has to do with how members are direct-initialized from
source members when copy construction is performed.

> Returning to the example I showed that to get the same rresult I need to
> introduce an intermediate variable. This makes the code less obvious becuse
> the link between the array and lambda will be lost. For example
> int a[N] = { /* some values */ };
> int *p = a;
> auto lm = [p]( int x ) mutable { return ( x == *p++ ); };

You can do the same by stopping insisting on using value capture. Either

auto lm = [&p]( int x ) mutable { return ( x == *p++ ); };
or
auto lm = [&]( int x ) mutable { return ( x == *p++ ); };

will work.

[Vlad from Moscow]

среда, 16 января 2013 г., 18:23:51 UTC+4 пользователь Ville Voutilainen написал:

This code has a side effect that the original object will be changed. 

[Ville Voutilainen]

On 16 January 2013 16:33, Vlad from Moscow <vlad....@mail.ru> wrote:
>> You can do the same by stopping insisting on using value capture. Either

> This code has a side effect that the original object will be changed.

Well, write a proposal about what you want and see if it goes through, then.
It doesn't seem we'll get anywhere by discussing it. Thus far every committee
member that has voiced one's opinion has opposed this proposed change,
so it would
perhaps be a good idea to found some who actually support it.

[Nevin Liber]

On 16 January 2013 08:33, Vlad from Moscow <vlad....@mail.ru> wrote:

This code has a side effect that the original object will be changed. 

n3485 25.1p10:

Unless otherwise specified, algorithms that take function objects as arguments are permitted to copy
those function objects freely.

In other words, the function objects you write are not designed for use with most of the standard algorithms (no matter what has been claimed), since side effects result in unpredictable behavior.

[Ville Voutilainen]

On 16 January 2013 17:30, Nevin Liber <ne...@eviloverlord.com> wrote:
>> This code has a side effect that the original object will be changed.
> n3485 25.1p10:
> Unless otherwise specified, algorithms that take function objects as
> arguments are permitted to copy
> those function objects freely.
> In other words, the function objects you write are not designed for use with
> most of the standard algorithms (no matter what has been claimed), since
> side effects result in unpredictable behavior.

For what it's worth, I don't think that was the "side effect" referred
to. And just
holding a pointer to something external in a functor doesn't cause unpredictable
behaviour as far as I understand, it's just that functor authors
shouldn't expect
a certain amount of copies to be made.

[Nevin Liber]

On Jan 16, 2013, at 9:42 AM, Ville Voutilainen

<ville.vo...@gmail.com> wrote:

> And just
> holding a pointer to something external in a functor doesn't cause unpredictable
> behaviour as far as I understand, it's just that functor authors
> shouldn't expect
> a certain amount of copies to be made.

Holding and dereferencing a pointer is fine. He is incrementing the
pointer that is stored in the function object; that is the side effect
which results in unpredictable behavior.

[Ville Voutilainen]

On 16 January 2013 19:14, Nevin Liber <nli...@gmail.com> wrote:
> Holding and dereferencing a pointer is fine. He is incrementing the
> pointer that is stored in the function object; that is the side effect
> which results in unpredictable behavior.

Ah, thanks, indeed - holding state inside a functor is most unwise, so
in order to make that
work, he'll need to pass-by-reference anyway.