Multi-Range For Loop

[Izzy Coding]

I have recently come across some legacy code that I am trying to update to the latest standards and STL.
In doing so I have come across something that I feel is missing from the current standard and I believe would be quite easy to add to the standard.

When working with STL style containers we can iterate over every item in a single container, however we are currently unable (without external libraries) to iterate over multiple containers.

For example:

given 2 or more equally sized containers (in my case 2 vectors of strings)
I want to iterate both and call an external function with both dereferenced values (i.e. passing http headers off to underlying API or class).

my options are:

1)

auto iter1 = vec_str1.begin();
auto iter2 = vec_str2.begin();
while (iter1 != vec_str1.end() && iter2 != vec_str2.end())
{
    http_client.add_header(*iter1, *iter2);

    ++iter1;
    ++iter2;
}

2)

for (auto iter1 = vec_str1.begin(), iter2 = vec_str2.begin(); iter1 != vec_str1.end() && iter2 != vec_str2.end(); ++iter1, ++iter2)
{
    http_client.add_header(*iter1, *iter2);
}

3) using boost

template<class... Conts>
auto zip_range(Conts&... conts) -> decltype(boost::make_iterator_range(
        boost::make_zip_iterator(boost::make_tuple(conts.begin()...)),
        boost::make_zip_iterator(boost::make_tuple(conts.end()...))
        ))
{
    return {
        boost::make_zip_iterator(boost::make_tuple(conts.begin()...)),
        boost::make_zip_iterator(boost::make_tuple(conts.end()...))
    };
}

for(auto&& t : zip_range(vec_str1, vec_str2))
{
    http_client.add_header(t.get<0>(), t.get<1>());
}

All of these to me seem like a lot of work, and are not particularly intuitive.


I wonder if it would be much more elegant and expressive to be able to write the following:

for (auto header_name, header_value : vec_str1, vec_str2)
{
    http_client.add_header(header_name, header_value);
}

This could collapse the comma delimited definitions into a tuple that would allow incrementing all the iterators together.
This would then also map the dereferenced tuple iterators into the defined names.

I am just wondering what the community might think about this?

[Ville Voutilainen]

On 11 August 2015 at 12:28, Izzy Coding <matthew.i...@gmail.com> wrote:
> I wonder if it would be much more elegant and expressive to be able to write
> the following:
> for (auto header_name, header_value : vec_str1, vec_str2)
> {
> http_client.add_header(header_name, header_value);
> }
> This could collapse the comma delimited definitions into a tuple that would
> allow incrementing all the iterators together.
> This would then also map the dereferenced tuple iterators into the defined
> names.
> I am just wondering what the community might think about this?

See http://cplusplus.github.io/EWG/ewg-active.html#43. So, this has been
suggested all the way to the committee, but we haven't seen an actual proposal
for it yet.

[Izzy Coding]

Great stuff.

Wonder if any paper has been done on this yet?
Would be good to get involved in this as it would make a lot of my legacy code so much simpler.

[Ville Voutilainen]

On 11 August 2015 at 13:05, Izzy Coding <matthew.i...@gmail.com> wrote:
> Great stuff.
> Wonder if any paper has been done on this yet?

No. As I said, there hasn't been a proposal for it.

[David Krauss]

On 2015–08–11, at 5:28 PM, Izzy Coding <matthew.i...@gmail.com> wrote:

for(auto&& t : zip_range(vec_str1, vec_str2))
{
    http_client.add_header(t.get<0>(), t.get<1>());
}

This isn’t so bad, given the zip_range facility. (Since there’s nothing else we can zip, perhaps std::zip would be a good enough name.)

Calling get is ugly but that’s the “multiple return value” problem which should be solved more generally. Hmm, I have an idea about that… will start another thread.

[Izzy Coding]

Granted the boost option is not too bad given the required boilerplate but it does rather hide the identity of what the returned values are.

I have written up a quick draft proposal for this. Any feedback would be greatly appreciated.
https://drive.google.com/open?id=0B26UZ0CbMay5V25ZZUFXZDlILXM

[Ville Voutilainen]

Any particular reason why you're suggesting a syntax different from
the one suggested
in EWG 43?

[Izzy Coding]

The one suggested in EWG to me seems a little less friendly when reading the code to determine what the loop body is doing. In most C++ code I write I always declare all before defining them.

E.G.
for ( var1 = x, var2 = 2; var1 <= var2; ++var1) { ... }

So to follow the same flow I wrote it as is.
I do not mind changing it if the masses prefer the other syntax.

[Richard Smith]

I agree: I'd rather see us fix the general problem of naming multiple return values / destructuring initialization than add a special syntax for zip iteration. (Of course, we don't have to choose between these and could have both.)

[T. C.]

I'm not sure I understand the argument.

var1 = x, var2 = 2 maps quite well to var1 : vector1; var2 : vector2.

 

Also, the comma operator can currently be used in the range initializer, and your proposal doesn't handle it.

Finally, the current rules is that auto x = f(), y = g(); must deduce the same type for x and y. Your syntax would either make the behavior inconsistent, or unnecessarily constrain it (can only iterate over ranges with the same value type).

for (auto& x : v; auto& y : w) avoids all of these problems, and is more expressive because you can do, say, for(auto& x : v; const auto& y : w).

[Izzy Coding]

In regards to the mapping I have always viewed the : to essentially be the "in" definition. As such I see it as essentially equivalent to the ; of the legacy for statement, which defines the iteration requirements.

for ( declaration ; iteration requirement; iteration action )...

compared to:

for ( declarations : iteration requirements and actions ) ...

As I mentioned before I am not tied to any particular syntax and would be happy to change. I just feel that the way I have defined it is easier to reason about the loop content given a long list of containers.

About the issue of the single auto&& in the declaration. This is just my simple way of writing it as this is a quick draft. I would indeed expect it to be more similar to:

for ( auto&& var1, auto&& var2 : con1, cont2 ) { ... }

Which would then allow for multiple different types in the declaration list. Unfortunately I am not a standardese pro and as such I more than likely have something wrong in my draft proposal.

In regards to the unpacking of multiple values to improve the boost::zip_iterator version of the code, I wonder if then we are suggestions that the zip_iterator or something similar should also be introduced into the standard? Also I would like to see what the ideas are around how the multiple return value stuff would affect this idea? Could someone provide an example of what using the multiple return value for statement would look like?

[Andrey Semashev]

On Wed, Aug 12, 2015 at 10:00 AM, Izzy Coding
<matthew.i...@gmail.com> wrote:
> In regards to the mapping I have always viewed the : to essentially be the "in" definition. As such I see it as essentially equivalent to the ; of the legacy for statement, which defines the iteration requirements.
>
> for ( declaration ; iteration requirement; iteration action )...
>
> compared to:
>
> for ( declarations : iteration requirements and actions ) ...
>
> As I mentioned before I am not tied to any particular syntax and would be happy to change. I just feel that the way I have defined it is easier to reason about the loop content given a long list of containers.
>
> About the issue of the single auto&& in the declaration. This is just my simple way of writing it as this is a quick draft. I would indeed expect it to be more similar to:
>
> for ( auto&& var1, auto&& var2 : con1, cont2 ) { ... }

To me, this syntax is slightly confusing. It looks like it might
iterate over permutations of the containers. Also, what is the
behavior if the containers have different sizes?

Personally, I like the zip solution most because it makes clear what
is being iterated over. But besides zip, I like the EWG proposed
syntax more because the element variable declarations are
syntactically close to the related ranges; with that syntax it is
easier to add or remove ranges to iterate over.

> Which would then allow for multiple different types in the declaration list. Unfortunately I am not a standardese pro and as such I more than likely have something wrong in my draft proposal.
>
> In regards to the unpacking of multiple values to improve the boost::zip_iterator version of the code, I wonder if then we are suggestions that the zip_iterator or something similar should also be introduced into the standard? Also I would like to see what the ideas are around how the multiple return value stuff would affect this idea? Could someone provide an example of what using the multiple return value for statement would look like?

I think most of Boost.Iterator and Boost.Range would be a good
addition to the standard library.

[Izzy Coding]

I see your points with regards to syntax style.
I will amend the proposal to use EWG 43 syntax. Just thinking about it I think the standard changes might even be simpler with that syntax as it will just be a change to the existing range-based for statement rather than a new syntax form.

With regards to what if the containers are different in size, that is mentioned in my draft proposal. Basically the __end is equal the the first container end reached. Therefore no out of range exceptions can occur.

With regards to preference of the zip_iterator version. My main bugbare is the required boilerplate code in order for it to work. If the new multi-range-based for syntax could be used but essentially do the zip... Version stuff under the covers then I would be happy with that. My only other issues are those to do with clarity of the code.

for ( var1:cont1, ... )
UseVars(var9, var2, ...);

To me is much clearer and less prone to errors than:

for( auto&& t : zip_iterator( cont1, ... ) )
UseVars(t.get<9>(), t.get<2>(), ...);

Using the zip format it is difficult to know what each t.get refers to (I do understand that this is essentially the multiple value return idea, but that also then to me seems like it could bloat the declarations in the for statement). I am however not against that also being possible in the standard, but feel a clearer syntax would be better than depending on library iterator adapters and tuple style multiple return values. To me it is just much more clear and precise and easy to reason about.

[Izzy Coding]

Here is a quick updated version of my draft proposal.
I hope this sits better with the original idea. Not sure if my wording is anywhere near clear as I am not a standardese person.

https://drive.google.com/open?id=0B26UZ0CbMay5cXpRSmtqd09YR00

[T. C.]

The wording needs a lot of work, but that's sort of expected.

 I'd still argue that the x : v pairs should be separated by semicolons, not commas. Otherwise you need to constrain the initializer to disallow comma operators, and introduce a breaking change along the way.

for-range-statement: 
 for-range-statement-list , ...

I don't understand this production at all. What are you trying to do here?

[T. C.]

On Wednesday, August 12, 2015 at 5:49:37 AM UTC-4, Izzy Coding wrote:

let the range-init be equivalent to a tuple-like structure containing the
required range-init for each of the declarations of a for-range-pair-
declaration. 
 tuple{ (expression1), ..., (expression{n}) }; 

This will have potential lifetime issues, and you didn't specify what this tuple-like structure contains - references? values?

I think trying to pack them is the wrong way to go. Just define auto&& __range1 through __rangeN. Ditto for __begin and __end.

[Izzy Coding]

I agree, maybe the semi-colon would be a better separator to use.

The format "something , ..." Is what I thought denoted continuous optional additions. My intent is to make the list essentially variable in length.

Could you give an example of what you mean by defining auto&& __range1 through __rangeN? I am not sure how this could be written in standardese terms.

[Matthew Woehlke]

On 2015-08-12 04:12, Izzy Coding wrote:
> With regards to preference of the zip_iterator version. [...] My only

> other issues are those to do with clarity of the code.
>
> for ( var1:cont1, ... )
> UseVars(var9, var2, ...);
>
> To me is much clearer and less prone to errors than:
>
> for( auto&& t : zip_iterator( cont1, ... ) )
> UseVars(t.get<9>(), t.get<2>(), ...);

...which is exactly why we need unpacking :-). (One of many reasons,
anyway.)

...or argument expansion:

for (auto&& iter : zip(cont1, ...))
call([*]iter);

(Requires that the argument order matches the order in the tuple, but in
this case you can do that by giving the containers in the correct order
to 'zip'. If we could slice tuples, that could be somewhat relaxed,
though the syntax would still be somewhat ugly.)

--
Matthew

[Izzy Coding]

This implementation still depends on something external to the for statement its self which to me is still a bit too dirty.
Unless something could be done to allow definition for the containers in the same format. however, that I believe would also not look very intuative.
However with a simple change to the for statement syntax rules it could be very simple and easily understood what a users intention is.
Also I would prefer not depending this idea on a library template class meaning that it can not be modified due to the tight coupling to this statement syntax.
Surely a dependancy like that would do more harm than good to the language?

[Izzy Coding]

Just been thinking. What if you have a range of tuples that you wish to map internally to separate variables? Also then if you have several ranges that you wish to iterate and unpack?

I would think the syntax defined in the draft proposal woul more easily allow this as well as being much more easy to understand and reason about.

for ( { auto&& item1, auto&& item2 } : tuple-range; auto&& single-value : container; { auto&& key1, auto&& value1 } : pair-range)
{
if (DoSomething(item1, item2, single-value))
{
DoSomethingElse(key1, value1);
}
}
How would something like this be expressed with only the multiple value return syntax?

[Matthew Woehlke]

On 2015-08-15 05:42, Izzy Coding wrote:
> Just been thinking. What if you have a range of tuples that you wish to map internally to separate variables?

Sounds like a good argument why we should have both :-).

Note: It wasn't my intention to be speaking against the proposal
(looking back, I can see where that wouldn't be clear), only to note
that unpacking at least makes the 'zip' form less ugly.

> How would something like this be expressed with only the multiple value return syntax?

Well, you *could* do it with a form of 'zip' that is aware that one or
more of its inputs should be unpacked first...

--
Matthew

[Izzy Coding]

>Sounds like a good argument why we should have both :-).

>Note: It wasn't my intention to be speaking against the proposal
(looking back, I can see where that wouldn't be clear), only to note
that unpacking at least makes the 'zip' form less ugly.

I agree, having both options would be good and allow for developer preference. In my proposal there would still be the possibility to add the multiple return syntax as part of the other proposal or an extension of.
I will be completing my amendements to my proposal and hopefully getting a document number too. I would be greatful for any comments / error corrections to what I have as I know my wording is not quite right. I would be willing to have any help to perfect this proposal, and maybe afterwards I could also look at a proposal to extend the existing multiple return syntax with relation to my proposal also. Again any help / comments are welcome as I want to try and ensure the best possible fit for the proposal.

[Izzy Coding]

Does anyone know what the current proposal document number is for the multiple return value work?
I need to reference it in my new version of my proposal document.
Regards

[Daniel Krügler]

2015-08-25 14:38 GMT+02:00 Izzy Coding <matthew.i...@gmail.com>:
> Does anyone know what the current proposal document number is for the multiple return value work?
> I need to reference it in my new version of my proposal document.

You will a document number assigned when submitting to the lwgchair
address. I guess you are talking about "Multi-Range for loops", right?
From my recent feedback you should notice that the document number
will be P0026. In any case: Please always refer with such questions to
the lwgchair address.

Thanks,

- Daniel

[Izzy Coding]

What I meant was for the "multiple return value" problem mentioned by David Krouss in this thread. I am not sure if there is any current proposal for this but I am just ensuring my proposal could work along side it enabling the use of both as has been discussed in this thread before.

[David Krauss]

> On 2015–08–25, at 9:58 PM, Izzy Coding <matthew.i...@gmail.com> wrote:
>
> What I meant was for the "multiple return value" problem mentioned by David Krouss in this thread. I am not sure if there is any current proposal for this but I am just ensuring my proposal could work along side it enabling the use of both as has been discussed in this thread before.

I’m not aware of any current proposal either. There’s just been a lot of discussion on this mailing list.

[Izzy Coding]

Great stuff, for now I have just mentioned that my proposal should allow for those discussions to be a potential proposal, however that it is currently out-of-scope for this proposal, but should allow for its possible addition in future revisions.

Current Draft Proposal Version:
https://drive.google.com/open?id=0B26UZ0CbMay5OGRORzkzZ1BiVWs

I hope it is much clearer than before. Any changes and amendments are welcome.

[T. C.]

From the motivation section:

3) Using legacy for loop with boost::zip_iterator

That example is using a range-based for loop.

Also with option 3 as I see it, there is a possibility for undefined behaviour if the for loop body modifies the ranges used. This is due to explicitly having stored each ranges end() iterator, which could potentially become an invalidated iterator on addition/removal of items from the container.

That's not being fixed by this proposal; the end() iterators are still obtained once and stored. I'd omit it.

Design decisions:

the subsequent dereferenced iterator is then assigned to the declared variable for use within the for statement body.

 

It's an initialization, not an assignment. Better be precise.

 the __end of iteration should at the point of hitting the first of any range ends

 

No need for double underscore here.

 

However, when compiling with high warning levels it would be useful when possible to have a compiler warning about such mismatched ranges.

I don't see how that's possible. Whether there's a length mismatch is a runtime property.

Technical Specifications:

auto&& __range-declaration-N = *__beginN;

That's incorrect. It should be

for-range-declaration-N = *__beginN;

for-range-declaration already includes whatever the user specified.

 for ( auto __begin = begin-expr, __end = end-expr; iterator-continuation-comparison; ++__begin )

__begin and __end need to be moved outside the for loop header. The iterators for different ranges are not necessarily of the same type.

Perhaps change range_init to (possibly rename to something like range-and-iter-init)

auto&& __rangeN = ( expression ) /* or braced-init-list */ ;
auto __beginN = begin_exprN, __endN = end_exprN;

It might be worth mentioning the change made by the ranges proposal should be incorporated - which uses instead

auto __beginN = begin_exprN;
auto __endN = end_exprN;

The difference being that this allows __beginN and __endN to have different types.

++__begin is wrong. It needs to be (void)++__begin1, (void) ++ __begin2, ..., (void) ++__beginN.

This makes me wonder if it would be simpler to specify the behavior using a while loop instead. At least you don't need the (void) casts.

if any of the comparisons return a false result then the expression as a whole should be false and iteration should stop

That's obvious from the expression. At best it's a note. I'd just delete it.

The final section still references __range, __begin, and __end, which can be confusing and should be changed.

Also, N is being used as a placeholder for any value in range in some places, and to refer to a particular value corresponding to the number of for-range-pair-declaration's in other places. I'd use different letters. 

[Izzy Coding]

Agree with all changes. Am applying them now.

>It might be worth mentioning the change made by the ranges >proposal should be incorporated - which uses instead

>...

What is the document number for the ranges proposal so I can add a reference to this as suggested?

[T. C.]

 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4382.pdf

[Izzy Coding]

Great thanks.

Newly updated version is here:
https://drive.google.com/open?id=0B26UZ0CbMay5NGFDV3hUQk9hbjA

I hope this is closer to what it should be.

[T. C.]

As cute as 

Note: this is an early draft. It’s known to be incomplete and incorrect, and it has lots of bad formatting.

 

is, I don't think you'll find it in many proposals. It's also less cute when you fixed the spelling and formatting :)

This proposal should also consider changes based on the current working paper for "Ranges" [N4382] 

where the iterator initialization for begin and end iterators are defined separately as they could 

potentially be of differing types

That feels a little strong. I'd go with something like "The specification below also incorporates a small change proposed in N4382 that allows begin and end iterators of different types." with a disclaimer that it's not essential to the proposal and trivial to remove if desired. The main point of the paper is multiple ranges, not sentinels. The latter is a minor drive-by change.

range-and-iter-init needs semicolons. The original one didn't have it because it's one declaration used as range-init;. Now that it's many separate declarations, it needs its own semicolons.

A multi-range-based for statement 

The rest of the specification section still calls it just "range-based for". I'd just stick with that name.

where iterator-continuation-comparison is equivalent to comparing

__beginN to its respective __endN using the not equal comparison for each 

of the ranges declared

__beginN != __endN

I think you deleted too much. The && is important, and is not described in this text.

and iteration-action is equivalent to incrementing __beginN for each of the ranges declared 

 

Likewise. That spot can only accommodate one expression, hence the comma operator.

Minor nitpick: I assume you copied part of the spec from a working draft - you have ligatures like fi in "defined" and "unqualified", which is a bit jarring with a monospace font. Compare: defined vs. defined

[Izzy Coding]

Great stuff.

New version can be found here:
https://drive.google.com/open?id=0B26UZ0CbMay5RG5NRERxWlBFNG8

Hope I have not missed anything.

[Klaim - Joël Lamotte]

I have read this version except the formal wording.

I think that there might be a part missing explaining why a library solution, an algorithm, would not work as well as a language change.

Or maybe suggestions to add both language and library extension (in which case another proposal would be refered to)

I mean: would it be possible to implement something like this?

    for_each( range_a, range_b, range_c, []( A& a, B& b, C& c ) { ... } ); ?

or 

    for_each( parallel_iter{range_a, range_b, range_c}, []( A& a, B& b, C& c ) { ... } ); // specialization

which would allow paramet:

    for_each( parallel_iter_default_value{range_a, range_b, range_c}, []( A& a, B& b, C& c ) { ... } );  // specialization

which would iterate for each value of any of the ranges even if there is no values anymore in the other ranges,

and will provide value-initialized values to the callback for these missing values instead of stopping as soon any range is finished.

I'm not suggesting to fully explore these suggestions, I would prefer the proposed solution, but I think

something like this will be suggested as alternative, so providing comparison might help the proposal.

--

---
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.
To post to this group, send email to std-pr...@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposals/.

[Izzy Coding]

I agree with the idea to add information about possible library support. I believe both should be implemented and have added a comment that this should be proposed separately.

Here is the new version:
https://drive.google.com/open?id=0B26UZ0CbMay5NGhsNmlnMjQtV0U

[T. C.]

That doesn't seem sufficient to me. The bar for a language feature is considerably higher than that of a library feature, and you really need to show that a language feature can do something a library feature cannot. Merely "nobody is proposing this at this time" is insufficient.

Here are some arguments for the language version:

- Easier to teach, read, and write. Doesn't require lambdas. Natural extension of existing syntax.

- Allows use of break & return, so you can terminate early. The library equivalent would require abusing exceptions.

- If the library version takes iterators, it's considerably more verbose. If the library version takes ranges, it's hard to implement the special begin/end lookup rules for range-for (in particular, the final step performs an ADL-only lookup that is not used anywhere else in C++), introducing subtle semantic differences.

- A std::for_each(Ranges..., Function) is possible but complex to implement (a function parameter pack not at the end is a non-deduced context; to avoid that, you basically have to do for_each(Args...),  taking the whole thing as a pack, and then do some special processing to unpack). A std::for_each(Function, Ranges...) can be implemented as is, but doesn't match existing version of `for_each` and a call to it is hard to read (because it puts the ranges last).

- Reduced library dependency, and usable on freestanding implementations, which may not have <iterator> or <algorithm>.

[Izzy Coding]

Thanks for the info. I knew I was missing something really important. I have now added this into the proposal. I hope this will now sit much better with everyone. Although it may still need a few weeks here and there I feel this is now much more complete. Please do keep any comments coming in.

Latest version can be viewed here:
https://drive.google.com/open?id=0B26UZ0CbMay5THZlSFlPLXRBNzQ

Fingers crossed

[Izzy Coding]

I meant to say tweeks not weeks :-)

[Ville Voutilainen]

On 31 August 2015 at 01:49, Izzy Coding <matthew.i...@gmail.com> wrote:
> I meant to say tweeks not weeks :-)

It's EWG, not EWB, when referring to the Evolution Working Group.

[Izzy Coding]

Aha, yes, just spotted that one myself.
Consider that to be rectified.

[Klaim - Joël Lamotte]

I tried to implement this version (that you didn't mention in the paper) of an overload of for_each():

    for_each( parallel_iter{range_a, range_b, range_c}, []( A& a, B& b, C& c ) { ... } ); // specialization

 

Unfortunately I lost my work when it was almost finished and have to write it again (some key combination in wnaderbox delete everything);

I plan to write it again before the end of this day, so that we have an actual thing to compare to.

On 31 August 2015 at 09:03, Izzy Coding <matthew.i...@gmail.com> wrote:

Aha, yes, just spotted that one myself.
Consider that to be rectified.

[Izzy Coding]

I would be very interested to see that implementation. I only left that out as I don't currently understand how it would be possible to map from parallel_iter ranges to each of the parameters of the lamda. Also did not know if it was possible as how does the algorithm know the count of ranges in order to be able to specify the required parameters for the lamda?

As I said I would however be very interested in seeing a possible implementation for it.

I was possibly thinking about adding this addition into the proposal so that both possibilities are available as I believe that they have a close relation and should probably be implemented together.

[Klaim - Joël Lamotte]

On 31 August 2015 at 20:47, Izzy Coding <matthew.i...@gmail.com> wrote:

I would be very interested to see that implementation. I only left that out as I don't currently understand how it would be possible to map from parallel_iter ranges to each of the parameters of the lamda. Also did not know if it was possible as how does the algorithm know the count of ranges in order to be able to specify the required parameters for the lamda?

As said I didn't get to the end of the thing, but I used C++14 features (a lot of variadics and lambdas) plus std::experimental::apply() function to do the following in parallel_iter:

 - extract begin and end iterators from all the ranges on construction (in two tuples);

 - provide a function to iterate all initially-begin iterators for each range, one time;

 - provide a function to check if any of these iterator reached end; << this one could be changed to get another semantic, like generating default values for iterators that are already end

 - provide a function calling a function with each value defered from each iterator from each range (this required a double apply to extract values by reference)

Then add a function to "make" this type given a bunch ranges, and the for_each algorithm becomes (from memory)

template< class Func, class... RangeList >

void for_each( parallel_iter<RangeList...>&& range_list, Func&& func ) // here parallel_iter is a specific type, so this is a specialization

{

    while(range_list.valid_iterators())

    {

        range_list.apply( std::forward<Func>(func) );

        range_list.next(); // increment iterators

    }

}

At this point the algorithm knows how many ranges exists as it have the RangeList.

(damn I was so close....)

Note that I think such function might be better with a different name than for_each.

As I said I would however be very interested in seeing a possible implementation for it.

I'm getting back at you when I get somewhere (in the night certainly).

 

I was possibly thinking about adding this addition into the proposal so that both possibilities are available as I believe that they have a close relation and should probably be implemented together.

Yes if it is possible to complete.

Also, I'm not a specialist so I don't know if I was breaking any standard guarantees.

[Klaim - Joël Lamotte]

On 31 August 2015 at 22:48, Klaim - Joël Lamotte <mjk...@gmail.com> wrote:

I'm getting back at you when I get somewhere (in the night certainly).

 

I was possibly thinking about adding this addition into the proposal so that both possibilities are available as I believe that they have a close relation and should probably be implemented together.

Yes if it is possible to complete.

Also, I'm not a specialist so I don't know if I was breaking any standard guarantees.

OK I managed to make it work :)

See: http://melpon.org/wandbox/permlink/m3bbQn601kQzZevb

(I'll copy the full code at the end of this email to avoid losing it)

Not sure if it's actually generic and all, but this:

int main()

{

    using namespace experiment;

    

    const std::vector<int> v { 1, 2, 3, 4, 5, 0, 1, 2, 3 };

    const std::list<float> l { 1.0f, 2.0f, 0.5f, 0.0f };

    const auto i = { "hello", "world", " !", " wow", "???" };

    

    std::vector<std::pair<double,std::string>> results;

    

    // don't allow write yet

    for_each( in_parallel( v, l, i ), [&]( int a, float b, auto&& c ){

        results.emplace_back( a * b, std::forward<decltype(c)>(c) );

    } );

    

    std::cout << "RESULTS : " << std::endl;

    for( const auto& pair : results )

        std::cout<< "    " << pair.first << " : " << pair.second << std::endl;

    std::cout << "RESULTS - END " << std::endl;    

}

Compiles and run on Wandbox with:

 - gcc 6.0/trunk C++14 and C++1z

 - gcc 5.2.0 C++14 and C++1z

 - clang 3.8/head C++14 and C++1z

 It does not compile with clang 3.6 because the <experimental/utility> 

header is not available.

The implementation is certainly optimizable (though I avoided most copies).

Also, I'm not sure how to determine which iterator type I should use for

each range. But I guess a function just to build a tuple type of iterator types using a sequence of range type might allow this.

Also I'm not a specialist with metaprog so I suspect that some of the recursion is not as efficient as some other recent techniques.

Anyway, it's possible to implement a for_each this way( though with different guarantees?).

That was fun. :D

 

 

[Klaim - Joël Lamotte]

On 1 September 2015 at 01:36, Klaim - Joël Lamotte <mjk...@gmail.com> wrote:

OK I managed to make it work :)

See: http://melpon.org/wandbox/permlink/m3bbQn601kQzZevb

(I'll copy the full code at the end of this email to avoid losing it)

The full code:

---------------------------------

#include <iostream>

#include <list>

#include <vector>

#include <string>

#include <tuple>

#include <initializer_list>

#include <experimental/tuple>

#include <type_traits>

#include <algorithm>

#include <utility>

namespace stdex = std::experimental;

namespace experiment

{

    template< class Iterator >

    void increment( Iterator&& it )

    {

        ++std::forward<Iterator>(it);

    }

    template< class FirstIter, class... OtherIters >

    void increment( FirstIter&& first, OtherIters&&... others )

    {

        increment( std::forward<FirstIter>(first) );

        increment( std::forward<OtherIters>(others)... );

    }

       

    template< class LeftIter, class RightIter >

    bool is_equal( LeftIter&& left, RightIter&& right )

    {

        return std::forward<LeftIter>(left) == std::forward<RightIter>(right);

    }

    

    // I'm using structs to be able to do partial specialization for index == 0

    template< std::size_t index >

    struct any_equal

    {

        template< class LeftIterTuple, class RightIterTuple >

        constexpr bool operator()( LeftIterTuple&& lefts, RightIterTuple&& rights ) const

        {

            if( is_equal( std::get<index>( std::forward<LeftIterTuple>(lefts) )

                , std::get<index>( std::forward<RightIterTuple>(rights) )

                ) 

            )

                return true;

            else 

                return any_equal<index - 1>{}( std::forward<LeftIterTuple>(lefts)

                                               , std::forward<RightIterTuple>(rights) );

        }        

    };

    

    template<>

    struct any_equal<0>

    {

        template< class LeftIterTuple, class RightIterTuple >

        constexpr bool operator()( LeftIterTuple&& lefts, RightIterTuple&& rights ) const

        {

            return is_equal( std::get<0>( std::forward<LeftIterTuple>(lefts) )

                , std::get<0>( std::forward<RightIterTuple>(rights) )

                ) ;

        }        

    };

    

    

    template< class... RangeRefList >

    struct parallel_iterator

    {

        

        static constexpr std::size_t rangelist_size = sizeof...(RangeRefList);

            static_assert( rangelist_size > 0, "range list too low");

        // Note: sentinel types and non-const-iterator types are not considered for this example.

        template< class RangeType >

        using IteratorType = typename std::decay_t<RangeType>::const_iterator;

        using IteratorTuple = std::tuple<IteratorType<RangeRefList>...>;

        

        IteratorTuple ranges_end;

        IteratorTuple ranges_it;

        

        template< class FirstRange, class SecondRange, class... OtherRanges >

        parallel_iterator( FirstRange&& first, SecondRange&& second, OtherRanges&&... others )

            : ranges_end{ end(first), end(second), end(others)... }

            , ranges_it{ begin(first), begin(second), begin(others)... }

        {}

        

        

        bool valid_iterators() const

        {

            static constexpr std::size_t start_index = rangelist_size - 1;

            static_assert( start_index >= 0, "Invalid index" );

            

            return !any_equal<start_index>{}( ranges_it, ranges_end );

        }

        

        template< class Func >

        void apply( Func&& func )

        {

            stdex::apply( [&]( auto&&... iterators ){

                stdex::apply( [&] ( auto&&... values ){ 

                    func( std::forward<decltype(values)>(values)... );

                }, std::forward_as_tuple( *iterators... ) );

            }, ranges_it );

        }

        

        void next()

        {

            stdex::apply( []( auto&&... iter ){

                increment( std::forward<decltype(iter)>(iter)... );

            }, ranges_it );

        }

        

    };

    template< class... RangeList >

    auto in_parallel( RangeList&&... range_list )

    {

        return parallel_iterator<RangeList...>{ std::forward<RangeList>(range_list)... };

    }

    template< class Func, class... RangeList >

    void for_each( parallel_iterator<RangeList...>&& range_list, Func&& func )

    {

        while(range_list.valid_iterators())

        {

            range_list.apply( std::forward<Func>(func) );

            range_list.next();

        }

    }

}

[Klaim - Joël Lamotte]

On 1 September 2015 at 01:37, Klaim - Joël Lamotte <mjk...@gmail.com> wrote:

On 1 September 2015 at 01:36, Klaim - Joël Lamotte <mjk...@gmail.com> wrote:

OK I managed to make it work :)

See: http://melpon.org/wandbox/permlink/m3bbQn601kQzZevb

(I'll copy the full code at the end of this email to avoid losing it)

The full code:

After some thinking:

 -  all the functions can be constexpr apparently (from my tests)

 - the actual iterator to use can be infered using decltype with the original range type and begin() end()

It seems to work as expected (though the test is not as complete as it should be for a library solution):

http://melpon.org/wandbox/permlink/r7yEVwS70ZVEFjBV

[Klaim - Joël Lamotte]

Wooops, wrong link, sorry for the spam: http://melpon.org/wandbox/permlink/gqhKDPZNl0Bp3mMA

I'm stopping here for now.

 

[Izzy Coding]

Sorry for late reply, sadly my daughter passed away so I have been with family.

I hope you don't mind. I have added your example as a reference in my proposal. Hopefully this might spark further discussions around the library implementation version. However I do believe both versions have their own merits and probably should both be considered for standardisation.

Please find my latest version of the draft here:
https://drive.google.com/open?id=0B26UZ0CbMay5V2JWNUpfbkdyTUE

Still looking for any feedback. I will be submitting this proposal in the next few weeks ready to go out on committee mailing list before next committee meeting.

[Klaim - Joël Lamotte]

On 10 September 2015 at 10:04, Izzy Coding <matthew.i...@gmail.com> wrote:

Sorry for late reply, sadly my daughter passed away so I have been with family.

I am so sorry to hear about your loss. Don't worry about the timing, I wasn't expecting immediate feedback anyway.

 

I hope you don't mind. I have added your example as a reference in my proposal. Hopefully this might spark further discussions around the library implementation version. However I do believe both versions have their own merits and probably should both be considered for standardisation.

I agree. Though I used an unusual style of arguments passing and I don't know if there is a reason for

avoiding this style.

 

Please find my latest version of the draft here:
https://drive.google.com/open?id=0B26UZ0CbMay5V2JWNUpfbkdyTUE

Still looking for any feedback. I will be submitting this proposal in the next few weeks ready to go out on committee mailing list before next committee meeting.

The proposal looks good to me in this version. It might trigger interesting discussions too.

By the way you might want to ask Eric Niebler his input if not already done.

 

[Izzy Coding]

By the way you might want to ask Eric Niebler his input if not already done.

How would I go about doing this?
Sorry I am relativley new to these things.... lol

[Izzy Coding]

By the way you might want to ask Eric Niebler his input if not already done.

How would I go about doing this?

Think I got it. Just sent him an email to get his opinion. Just hope he does not delete it thinking it is spam...lol

[T. C.]

On Thursday, September 10, 2015 at 4:04:37 AM UTC-4, Izzy Coding wrote:

Sorry for late reply, sadly my daughter passed away so I have been with family.

I'm really sorry to hear that. Hope that everything is OK.

 <snip>

Please find my latest version of the draft here:
https://drive.google.com/open?id=0B26UZ0CbMay5V2JWNUpfbkdyTUE

Still looking for any feedback. I will be submitting this proposal in the next few weeks ready to go out on committee mailing list before next committee meeting.

At the risk of getting eggs on my face, I'm now convinced that it is possible to simulate an ADL-only lookup in ordinary C++ code. The point about subtle semantic differences still stands, though, because I'm fairly sure that step two (looking for member begin/end) is impossible to perfectly emulate in code, as it requires distinguishing between having a member called "begin" (or "end") that is inaccessible/not a function/have the wrong number of parameters etc. and not having one at all.

[Izzy Coding]

Surely the check for begin/end member functions would be done via concepts? Also if a template function calls a member function that does not exist surely it would not compile.
Not sure if concepts can do this, or if they would affect the lookup results?

I agree I am sure there is a way to be able to provide the functionality as a library feature. However I just don't know enough to be able to create it. Would be interested to work with someone who is more knowledgable in that area.

[T. C.]

It's fairly straightforward to do "if r.begin() compiles, call it, otherwise call begin(r)" (with the usual caveats about immediate context).

I'm not aware of any way to do "if the type of r has a member called 'begin', attempt to call r.begin(), otherwise call begin(r)" in normal C++ code, concepts or not. And the latter is what range-for does.

In other words, the library solution would accept some types that range-for rejects.

[Klaim - Joël Lamotte]

I was thinking about mailing him OR posting a heads-up on the Ranges discussion group.

[Klaim - Joël Lamotte]

On 12 September 2015 at 11:23, T. C. <rs2...@gmail.com> wrote:

It's fairly straightforward to do "if r.begin() compiles, call it, otherwise call begin(r)" (with the usual caveats about immediate context).

I'm not aware of any way to do "if the type of r has a member called 'begin', attempt to call r.begin(), otherwise call begin(r)" in normal C++ code, concepts or not. And the latter is what range-for does.

In other words, the library solution would accept some types that range-for rejects.

If I understood you correctly (I'm not sure), this is already what std::begin does: http://en.cppreference.com/w/cpp/iterator/begin

Although you need to use a using namespace declaration to benefit from ADL:

template< class Container >

void foo( Container&& container )

{

     using namespace std; // allow to use std::begin if no Container's namespace specific begin() function is provided

     auto&& it = begin(container); // if no other option than std::begin() is selected here, if container.begin() exists, use it, otherwise fails to compile.

     ...

}

 

[T. C.]

On Saturday, September 12, 2015 at 4:24:04 PM UTC-4, Klaim - Joël Lamotte wrote:

If I understood you correctly (I'm not sure), this is already what std::begin does: http://en.cppreference.com/w/cpp/iterator/begin

Although you need to use a using namespace declaration to benefit from ADL:

template< class Container >

void foo( Container&& container )

{

     using namespace std; // allow to use std::begin if no Container's namespace specific begin() function is provided

     auto&& it = begin(container); // if no other option than std::begin() is selected here, if container.begin() exists, use it, otherwise fails to compile.

     ...

}

... and you fell into one of the subtle traps here. You can't use an overload set containing both `std::begin` and a `begin` found by ADL, because the latter can be equally greedy (think something like boost::begin - though they added an ADL barrier for that) and - voila - you get an ambiguous call. See also http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3257.pdf.

Here's an example of what I mean:

namespace foo {
    struct A { int begin; };
    struct B { using end = int; };
    class C { int* begin(); int *end(); }; // inaccessible
    struct D { int* begin(int); int* end();};
    struct E {};

    template<class T> int* begin(const T&) { return nullptr; }
    template<class T> int* end(const T&) { return nullptr; }
}

int main() {
    foo::A a; foo::B b; foo::C c; foo::D d; foo::E e;
    for(auto i1 : a) { } // doesn't compile
    for(auto i2 : b) { } // doesn't compile
    for(auto i3 : c) { } // doesn't compile
    for(auto i4 : d) { } // doesn't compile
    for(auto i5 : e) { } // compiles!
}

What I'm saying is that I believe it's impossible to write something in normal C++ that has equivalent behavior (fail on types like A/B/C/D but succeed on types like E).

[Klaim - Joël Lamotte]

I was right thinking I certainly misunderstood. Thanks for the clarification.

By the way, is the "customization points" proposal from Eric Niebler a potential solution for this? 

My understanding is not totally complete on these subjects.

 

[Izzy Coding]

Would it not be better to have something along the lines of the following:

template<typename F, typename T...>
// requires F to be callable with all Ts as parameters
void for_each(T&& t..., F&& fn)
{
auto ranges = make_multirange(forward(t)...);
for(;ranges.current() != ranges.end(); ++ranges)
{
fn(ranges.get<>()...);
}
}

I know this kind of thing is not currently possible but this would seem the best approach and also allow other useful possibilities too. This code is assuming a tuple-like type for the collection of ranges.

[Klaim - Joël Lamotte]

It would be best if there was not the current for_each() overloads yes.

Unfortunately, even with a new stl version, there would still be a need for for_each() overloads taking a pair of iterators, which mean still the same problem, except if you use concepts to define iterator's concept.

If I'm following correctly.

That's the main reason I provided a working example which don't conflict with the current signatures: a note could be added that such a signature would be nice but then would be 

dependent on concept TS/proposals which add a dependence on this proposal so should not be mandatory.

[Izzy Coding]

I was essentially thinking this in addition to the existing for_each definitions.
The assumption of dependacy on concepts is correct and would be needed for this ability as you have suggested.
I was merely suggesting what the ideal solution might look like given other changes in the standard that are currently not available.
For example I dont think the "ranges.get<>()..." is currently possible to expand out all the items of a tuple like object?
Also I did not specify exactly what the "make_multirange" function does. I guess in reality it could be down to that function to be specialised based on the T's that are passed. I only called it "make_multirange" to illustrate the focus of this current proposal. I assume it would actually be named differently in order to accept various differing range specifications.
The only current options I currently know of however are:

1. iterators begin() and end()
2. ranges (as defined in the ranges proposal)
3. and the new multi ranges which essentially could be a variable list of types that expose a begin() and end() member.

ultimatley they would all be wrapped into an STL type that exibhits the required interface to the values passed in for the "for" statement to execute correctly.

Anything that is unable to be implemented within the wrapper type would be classed as invalid code.

I hope that all makes sense? I know my descriptive ability leave a fare bit to be desired. LOL

[Izzy Coding]

Hopefully the last draft of the proposal.
I hope these additions will help to spark more debate about the possible library implementation of multi-range based loops.

The latest version can be found here:
https://drive.google.com/open?id=0B26UZ0CbMay5dnZrWU5mcW9ZTFU

As always all comments and suggestions are welcome.

[denis...@gmail.com]

Hi, Are there any updates on this proposal?

[denis...@gmail.com]

It should be more important since parallel algorithms were introduced, because if I want to perform some operation on several arrays simultaneously I cannot do it at the moment.
Moreover, it could replace transform function that take two arrays as input with more general solution.

[Izzy Coding]

At present this proposal was discussed and went on the mailing list, but was apparently dismissed.
I was unable to attend the standards meeting, so I am not exactly sure what happened. The most I can gather is that it was not accepted because the ranges proposal would offer the same or similar ability.
I am however, still open to comments and any arguments and examples to help support this proposal.
Also, I am still interested in help to get a good library implementation of this together.
If there was a standards meeting in England it would be easier for me to attend. Other than that I am also looking for someone who can attend the meetings and champion this proposal on my behalf.

[Klaim - Joël Lamotte]

You might want to contact Micheal Wong, if I remember correctly he
manages the group in england, which would be a good place to find help
on that.

> --

> You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.

> To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/8b446b1b-5944-4a3f-b899-f88604c41612%40isocpp.org.