A way around _HAS_ITERATOR

Bonita Montero

unread,

Jun 2, 2021, 8:49:30 AM6/2/21

to

Because i needed something like ptr = &*vec.end() and MSVC has iterator
-debugging by default I had a little header which I included before all
other headers.

#pragma once
#if defined(_MSC_VER)
#if defined(_HAS_ITERATOR_DEBUGGING) && _HAS_ITERATOR_DEBUGGING != 0
#error _HAS_ITERATOR_DEBUGGING must be set to 0
#elif !defined(_HAS_ITERATOR_DEBUGGING)
#define _HAS_ITERATOR_DEBUGGING 0
#endif
#endif

But now I found that to_address( it ) of C++ 20 translates to
it.operator ->(), so I don't use the above header any more but
just it.operator ->() to stick with C++17 - although the latest
MSVC++-upddate is C++20 feature-complete and I could use to_address().

Alf P. Steinbach

unread,

Jun 2, 2021, 9:07:27 AM6/2/21

to

On 2 Jun 2021 14:49, Bonita Montero wrote:
> Because i needed something like ptr = &*vec.end() and MSVC has iterator
> -debugging by default I had a little header which I included before all
> other headers.
>
> #pragma once
> #if defined(_MSC_VER)
>     #if defined(_HAS_ITERATOR_DEBUGGING) && _HAS_ITERATOR_DEBUGGING != 0
>         #error _HAS_ITERATOR_DEBUGGING must be set to 0
>     #elif !defined(_HAS_ITERATOR_DEBUGGING)
>         #define _HAS_ITERATOR_DEBUGGING 0
>     #endif
> #endif

Hm. If you've defined the macro as anything but 0, then this code errs
out. Otherwise it defines the macro as 0. So what does it tell you other
than that it's Visual C++, which `_MSC_VER` already tells you?

> But now I found that to_address( it ) of C++ 20 translates to
> it.operator ->(), so I don't use the above header any more but
> just it.operator ->() to stick with C++17 - although the latest
> MSVC++-upddate is C++20 feature-complete and I could use to_address().

This seems unrelated to above, perhaps you inadvertently failed to
include some more code?

Anyway, it's practically safe to take the address of `*s.end()` for a
`std::string`, because since C++11 the buffer has a guaranteed
null-terminator, and (at least as I see it) only a perverse
implementation would not let `*s.end()` be that final null item.

But in general, e.g. for a `std::vector`, there is no guaranteed item
beyond the last one "in" the collection, so that the `collection.end()`
iterator can even be a nullvalue itself. To get a pointer beyond the
collection use `collection.data() + collection.size()`.

However, that's pretty annoying for a `std::string_view` or (disclaimer:
haven't used it) `std::span`.

So, good idea to define it yourself, agreed, but not by dereferencing
the end iterator.

- Alf

Bonita Montero

unread,

Jun 2, 2021, 9:13:11 AM6/2/21

to

>> Because i needed something like ptr = &*vec.end() and MSVC has iterator
>> -debugging by default I had a little header which I included before all
>> other headers.
>>
>> #pragma once
>> #if defined(_MSC_VER)
>>      #if defined(_HAS_ITERATOR_DEBUGGING) && _HAS_ITERATOR_DEBUGGING != 0
>>          #error _HAS_ITERATOR_DEBUGGING must be set to 0
>>      #elif !defined(_HAS_ITERATOR_DEBUGGING)
>>          #define _HAS_ITERATOR_DEBUGGING 0
>>      #endif
>> #endif

> Hm. If you've defined the macro as anything but 0, then this code errs
> out. Otherwise it defines the macro as 0. So what does it tell you other
> than that it's Visual C++, which `_MSC_VER` already tells you?

I want to set _HAS_ITERATOR_DEBUGGING to 0 and if it is already set the
compilation should stop.

>> But now I found that to_address( it ) of C++ 20 translates to
>> it.operator ->(), so I don't use the above header any more but
>> just it.operator ->() to stick with C++17 - although the latest
>> MSVC++-upddate is C++20 feature-complete and I could use to_address().

> This seems unrelated to above, perhaps you inadvertently failed to
> include some more code?

No, if I do ptr = &*vec.end(), depending on _HAS_ITERATOR_DEBUGGING,
the dereferencing of end is runtime-checked and an assertion fails.

> Anyway, it's practically safe to take the address of `*s.end()` for
> a `std::string`, because since C++11 the buffer has a guaranteed
> null-terminator, and (at least as I see it) only a perverse
> implementation would not let `*s.end()` be that final null item.

Interesting, but I'm concerned about vecors in this case.

> But in general, e.g. for a `std::vector`, there is no guaranteed item
> beyond the last one "in" the collection, so that the `collection.end()`
> iterator can even be a nullvalue itself. To get a pointer beyond the
> collection use `collection.data() + collection.size()`.

"ptr = vec.end().operator ->()" works perfectly - and even around
_HAS_ITERATOR_DEBUGGING.

Paavo Helde

unread,

Jun 2, 2021, 9:46:30 AM6/2/21

to

02.06.2021 15:49 Bonita Montero kirjutas:
> Because i needed something like ptr = &*vec.end() and MSVC has iterator
> -debugging by default

Dereferencing vec.end() is UB, so you should not do this even if it
appears to work in some situation.

What's wrong with vec.data()+vec.size() ?

Bonita Montero

unread,

Jun 2, 2021, 10:23:32 AM6/2/21

to

>> Because i needed something like ptr = &*vec.end() and MSVC has iterator
>> -debugging by default

> Dereferencing vec.end() is UB, ...

In theory, but there's for sure no actual vector-implementation
where this doesn't work as long as you don't read from or write
to the reference.
The theory bases on architectures which might check the pointers
inside the references when they're loaded to a register. F.e.
CPUs with segment-registers might load the segment and the seg-
ment might get checked; but such architectures aren't of any
importance today.

Paavo Helde

unread,

Jun 2, 2021, 12:43:18 PM6/2/21

to

02.06.2021 17:23 Bonita Montero kirjutas:
>>> Because i needed something like ptr = &*vec.end() and MSVC has iterator
>>> -debugging by default
>
>> Dereferencing vec.end() is UB, ...
>
> In theory,

Says the person who complains that her code does not work in practice
with a widespread implementation, unless specifically silenced with a
_HAS_ITERATOR_DEBUGGING macro (note: using this macro is deprecated by MS).

Bonita Montero

unread,

Jun 2, 2021, 12:58:08 PM6/2/21

to

> Says the person who complains that her code does not work in practice
> with a widespread implementation, unless specifically silenced with a
> _HAS_ITERATOR_DEBUGGING macro (note: using this macro is deprecated by MS).

That has nothing to do with pseudo-UB.

Bonita Montero

unread,

Jun 2, 2021, 12:59:54 PM6/2/21

to

> ... (note: using this macro is deprecated by MS).

I want the code to work with older compilers.

Paavo Helde

unread,

Jun 2, 2021, 2:13:44 PM6/2/21

to

There is no such thing as pseudo-UB. If your code contains UB, it might
work, it might not work, it might appear to work without really working,
and even if it worked it might cease to work in the next day, next
compiler, or the next compiler version. There are no guarantees.

A term like "pseudo-UB" implies that there is some magic guarantee that
the UB code always works as intended. Alas, UB specifically means no
guarantees, so there is no such thing. Even if you hack your UB code to
(apparently) work with some specific C++ implementation, you have no
guarantee that your code will work with any other C++ implementation, or
with the next version of the same implementation.

The fact that your code does not work with a specific C++ implementation
without hacking already shows that this is not a theoretical issue.

If the C++ committee comes around and says, yes, sorry, we make
&*vec.end() defined behavior after all, then yes, you are welcome to
write this code, now with defined behavior, and MS is obliged to fix
their now non-conforming compiler. But until this has not happened, UB
is UB.

Alf P. Steinbach

unread,

Jun 2, 2021, 2:28:00 PM6/2/21

to

On 2 Jun 2021 15:12, Bonita Montero wrote:
> [snip]

> "ptr = vec.end().operator ->()" works perfectly - and even around
> _HAS_ITERATOR_DEBUGGING.

But why use an expression with formally Undefined Behavior when there is
a perfectly well-defined way to do it?

- Alf

James Kuyper

unread,

Jun 2, 2021, 5:23:22 PM6/2/21

to

On 6/2/21 2:13 PM, Paavo Helde wrote:
> 02.06.2021 19:57 Bonita Montero kirjutas:

...

>> That has nothing to do with pseudo-UB.
>
> There is no such thing as pseudo-UB. If your code contains UB, it might
> work, it might not work, it might appear to work without really working,
> and even if it worked it might cease to work in the next day, next
> compiler, or the next compiler version. There are no guarantees.
>
> A term like "pseudo-UB" implies that there is some magic guarantee that
> the UB code always works as intended. Alas, UB specifically means no
> guarantees,

Not quite. It means "behavior for which this document imposes no
requirements" (3.30). That wording deliberately and explicitly allows
for the possibility that something other than "this document" might in
fact impose requirements on the behavior of the code. This is, in fact,
one of the most common reasons for writing UB code (and the only
legitimate reason for doing so): an implementation provides some
guarantees for the behavior that the standard itself does not. Such code
is only portable to implementations which provide that same guarantee,
but in many contexts that's not a serious problem.

However, if Bonita wants to rely upon that exception, she needs to
identify the document which provides the guarantees she's relying on - and

unread,

Jun 2, 2021, 6:54:57 PM6/2/21

to

>> [snip]
>> "ptr = vec.end().operator ->()" works perfectly - and even around
>> _HAS_ITERATOR_DEBUGGING.

> But why use an expression with formally Undefined Behavior when there is
> a perfectly well-defined way to do it?

It actually works on all machines.

unread,

Jun 3, 2021, 11:06:53 AM6/3/21

to

On 6/3/21 5:21 AM, Juha Nieminen wrote:
> Bonita Montero <Bonita....@gmail.com> wrote:
>>>> Because i needed something like ptr = &*vec.end() and MSVC has iterator
>>>> -debugging by default
>>
>>> Dereferencing vec.end() is UB, ...
>>
>> In theory, but there's for sure no actual vector-implementation
>> where this doesn't work as long as you don't read from or write
>> to the reference.
>
> The problem with relying on undefined behavior is that you have no
> guarantees about what the compiler will do to your code.

The C standard, while describing the unary & operator,says "If the
operand is the result of a unary * operator, neither that operator nor
the & operator is evaluated and the result is as if both were omitted,
except that the constraints on the operators still apply and the result
is not an lvalue." (C 6.5.3.2p3).

I could find no corresponding statement in the C++ standard, but
shouldn't there be one? The C++ version would have to be more
complicated, to deal with issues that can't come up in C, such as
operator overloads. However, is there any good reason why there
shouldn't be a similar clause in the C++ standard? That would make
Bonita's code perfectly safe.

David Brown

unread,

Jun 3, 2021, 1:07:59 PM6/3/21

to

I looked up exactly the same thing myself. In C++, this kind of thing
is complicated by overloaded operators. It is perfectly allowable for
the "*" operator on a class to have side-effects, or do something that
is not normal dereferencing. The same applies to the "&" operator. So
it would not be good to say that "&*" is eliminated in the way it is in
C - you would have to say it only applied to standard types, and then
things get inconsistent.

Paavo Helde

unread,

Jun 3, 2021, 1:18:21 PM6/3/21

to

C does not have user-defined operators. In C++ every iterator has its
own implementation of operator*() and there is no guarantee that the
operator & would exactly undo that operation. Indeed, typically it
cannot and does not. Typically the result of &*iter is a pointer, not an
iterator, and that's exactly why one might use such a construction in
the first place.

So in C++ one just cannot "cancel" &* as the result would be of a wrong
type. It might be possible to make it happen (maybe by adding a special
operator()&* to the standard which iterator classes would use for
converting an iterator into a pointer), but the needed legalese seems
daunting, and the benefits seem pretty meager (satisfying one angry
Bonita Montero somewhere on the internet; everybody else is happy with
vec.data()+vec.size()).

Juha Nieminen

unread,

Jun 3, 2021, 1:49:57 PM6/3/21

to

I honestly cannot understand why you feel the need to behave like such
an asshole. I merely made a remark about the language standard and what
it could allow the compiler to behave like (and gave a practical example
of when this caused a huge issue in the Linux kernel).

So why are you treating it like shit? I don't understand. What exactly
did I do or say that elicits such irrational behavior?

It's not even the first time. You keep doing this again and again,
most often completely unpromted out of the blue, in response to completely
innocuous posts, for completely unknown reasons. Why?

Juha Nieminen

unread,

Jun 3, 2021, 1:53:06 PM6/3/21

to

James Kuyper <james...@alumni.caltech.edu> wrote:
> The C standard, while describing the unary & operator,says "If the
> operand is the result of a unary * operator, neither that operator nor
> the & operator is evaluated and the result is as if both were omitted,
> except that the constraints on the operators still apply and the result
> is not an lvalue." (C 6.5.3.2p3).
>
> I could find no corresponding statement in the C++ standard, but
> shouldn't there be one? The C++ version would have to be more
> complicated, to deal with issues that can't come up in C, such as
> operator overloads. However, is there any good reason why there
> shouldn't be a similar clause in the C++ standard? That would make
> Bonita's code perfectly safe.

It's different when dealing with iterators. I thing some std::vector
implementations in fact use raw pointers as iterators, but that's
not guaranteed.

MrSpo...@3p26kggzpvpn2h.gov.uk

unread,

Jun 4, 2021, 3:43:32 AM6/4/21

to

You're a fine one to talk.

Chris Vine

unread,

Jun 4, 2021, 5:22:28 AM6/4/21

to

On Thu, 3 Jun 2021 20:18:05 +0300
Paavo Helde <myfir...@osa.pri.ee> wrote:
> 03.06.2021 18:06 James Kuyper kirjutas:

[snip]

> > The C standard, while describing the unary & operator,says "If the
> > operand is the result of a unary * operator, neither that operator nor
> > the & operator is evaluated and the result is as if both were omitted,
> > except that the constraints on the operators still apply and the result
> > is not an lvalue." (C 6.5.3.2p3).
> >
> > I could find no corresponding statement in the C++ standard, but
> > shouldn't there be one? The C++ version would have to be more
> > complicated, to deal with issues that can't come up in C, such as
> > operator overloads. However, is there any good reason why there
> > shouldn't be a similar clause in the C++ standard? That would make
> > Bonita's code perfectly safe.
>
> C does not have user-defined operators. In C++ every iterator has its
> own implementation of operator*() and there is no guarantee that the
> operator & would exactly undo that operation. Indeed, typically it
> cannot and does not. Typically the result of &*iter is a pointer, not an
> iterator, and that's exactly why one might use such a construction in
> the first place.
>
> So in C++ one just cannot "cancel" &* as the result would be of a wrong
> type. It might be possible to make it happen (maybe by adding a special
> operator()&* to the standard which iterator classes would use for
> converting an iterator into a pointer), but the needed legalese seems
> daunting, and the benefits seem pretty meager (satisfying one angry
> Bonita Montero somewhere on the internet; everybody else is happy with
> vec.data()+vec.size()).

They shouldn't have been happy with that. It says something about the
state of the C++ language standard that until C++20, although
std::vector::data() was said to return "A pointer such that [data(),
data() + size()) is a valid range", actually applying pointer
arithmetic to the result of std::vector::data() as in 'vec.data() +
vec.size()' gave undefined behaviour - undefined bahaviour arising from
the use of pointer arithmetic on a pointer not pointing within (or one
past the end of) an array object.

No one seems to have noticed until P0593 arrived on the scene, which
also pointed out that use of the standard's std::uninitialized_*()
functions also generated undefined behaviour, for the same reason.

Paavo Helde

unread,

Jun 4, 2021, 7:20:03 AM6/4/21

to

You mean, an array encapsulated by a std::vector might not be an array?

At least https://en.cppreference.com/w/cpp/container/vector/data says:
"Returns pointer to the underlying array serving as element storage."
This wording would make vec.data()+vec.size() legal. Alas, the standard
indeed uses a different wording.

P0593 seems to be more worried about how it is even possible to
implement std::vector, this is a bit different topic.

Chris Vine

unread,

Jun 4, 2021, 8:07:32 AM6/4/21

to

On Fri, 4 Jun 2021 14:19:44 +0300

std::vector does not encapsulate an array. std::vector constructs
objects in memory dynamically allocated by malloc, operator new or some
custom allocator.

[expr.add]/4 on pointer arithmetic is clear:

"When an expression J that has integral type is added to or subtracted
from an expression P of pointer type, the result has the type of P.

— If P evaluates to a null pointer value and J evaluates to 0, the
result is a null pointer value.

— Otherwise, if P points to an array element i of an array object x
with n elements, the expressions P + J and J + P (where J has the
value j) point to the (possibly-hypothetical) array element i + j of
x if 0 ≤ i + j ≤ n and the expression P - J points to the
(possibly-hypothetical) array element i − j of x if 0 ≤ i − j ≤ n.

— Otherwise, the behavior is undefined."

"Array" here means a C-style array, although std::array would do as
that allocates its storage in a C-style array. std::vector does not.

If you compile 'vec.data()+vec.size()' with the -std=c++20 flag, the
expression is valid (but not otherwise), and that is presumably what
cppreference.com is thinking of. In C++20 an array is an implicit
lifetime type which is separate from the sub-objects (the elements) it
contains - the latter may or may not be implicit lifetime types. In
C++20 an array always is, and it will spring into life automatically if
necessary to give the code defined behaviour. 'vec.data()+vec.size()'
creates an array implicitly. Prior to that the allocated storage
concerned is in super-position, being of both array and non-array type
at the same time (Shroedinger's Array).

As you say, in C++17 it is impossible to implement your own equivalent
of std::vector with defined behaviour, and much else besides. In C++20
it is at least now possible to build your own vector type and
associated allocators with something approaching defined behaviour.
You can also used std::uninitialized_copy() with defined behaviour.
To be honest, I regard this as a pretty major fail for C++: it turned
out that the C++ standard committee did not understand its own standard
on fairly fundamental points.

James Kuyper

unread,

Jun 4, 2021, 10:40:28 AM6/4/21

to

On 6/3/21 1:18 PM, Paavo Helde wrote:
...

> So in C++ one just cannot "cancel" &* as the result would be of a wrong
> type.

It could be the right type if the return type of the relevant
operator&() overload is the same as the iterator type.

Paavo Helde

unread,

Jun 4, 2021, 12:28:32 PM6/4/21

to

In general, it cannot be. Imagine:

class A {
std::vector<A>::iterator operator&() {
// magic here???
}
};

An object in C++ does not know if and in which vector it might reside
(unless specifically told so, resulting in some pretty convoluted
design), and so it cannot construct a correct iterator to itself.

Even if it were the "right type" this would not solve the OP's problem
as she specifically wanted to get a pointer, not an iterator.

James Kuyper

unread,

Jun 4, 2021, 1:04:59 PM6/4/21

to

I've never attempted implementing any of the standard containers (not
because I couldn't, but just to avoid reinventing the wheel), so I might
be missing some important detail (possibly one introduced by recent
updates - I'm sill more somewhat shaky in my understanding of some of
the features introduced after C++ 2003). However, it seems to me that it
should be possible to implement std::vector<T> with the iterator type
being T*, in which case that wouldn't be a problem. If that's not
feasible, could you explain what's problematic about it?

Chris M. Thomasson

unread,

Jun 4, 2021, 4:30:34 PM6/4/21

to

On 6/4/2021 10:04 AM, James Kuyper wrote:
> On 6/4/21 12:28 PM, Paavo Helde wrote:
>> 04.06.2021 17:40 James Kuyper kirjutas:
>>> On 6/3/21 1:18 PM, Paavo Helde wrote:
>>> ...
>>>> So in C++ one just cannot "cancel" &* as the result would be of a wrong
>>>> type.
>>>
>>> It could be the right type if the return type of the relevant
>>> operator&() overload is the same as the iterator type.
>>
>> In general, it cannot be. Imagine:
>>
>> class A {
>> std::vector<A>::iterator operator&() {
>> // magic here???
>> }
>> };
>>
>> An object in C++ does not know if and in which vector it might reside
>> (unless specifically told so, resulting in some pretty convoluted
>> design), and so it cannot construct a correct iterator to itself.
>>
>> Even if it were the "right type" this would not solve the OP's problem
>> as she specifically wanted to get a pointer, not an iterator.
>
> I've never attempted implementing any of the standard containers (not
> because I couldn't, but just to avoid reinventing the wheel), so I might
> be missing some important detail (possibly one introduced by recent
> updates - I'm sill more somewhat shaky in my understanding of some of
> the features introduced after C++ 2003).

I always thought that implementing standard containers in a C++ course
would be fun. The teacher says something like: We are going to implement
several std containers under the namespace std_course. Imvvho, it would
be an interesting and worth while exercise.

Paavo Helde

unread,

Jun 4, 2021, 4:46:02 PM6/4/21

to

Indeed, there have been implentations where iterator type was T*.
However, for now the major C++ implementations have abandoned this idea,
both for more type safety (to avoid accidental conversion from pointers
to iterators) and for supporting different levels of iterator debugging
(catching out-of-range iterators, stale iterators, dereferencing
non-dereferencable iterators like vec.end(), etc).

> in which case that wouldn't be a problem.

There is no problem. There is no need to dereference non-dereferencable
iterators.

> If that's not
> feasible, could you explain what's problematic about it?

Apparently there is a perceived problem of misusing iterators and a
desire to provide iterator safety checks and debugging aids. This would
not be possible if iterators were just plain pointers.

A demo: this program reports a run-time error "Error: attempt to advance
a [...] iterator 10 steps, which falls outside its valid range." when
compiled with right options.

> cat test1.cpp
#include <vector>
#include <algorithm>

int main() {
std::vector<int> v = {3,2,1,5,2,5,3};
std::sort(v.begin(), v.begin()+10);
}

> g++ test1.cpp -D_GLIBCXX_DEBUG
> ./a.exe
/usr/lib/gcc/x86_64-pc-cygwin/10/include/c++/debug/safe_iterator.h:906:
In function:
__gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*,
std::__cxx1998::vector<int, std::allocator<int> > >,
std::__debug::vector<int>, std::random_access_iterator_tag>::_Self
__gnu_debug::operator+(const _Self&,
__gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<int*,
std::__cxx1998::vector<int, std::allocator<int> > >,
std::__debug::vector<int>,
std::random_access_iterator_tag>::difference_type)

Error: attempt to advance a dereferenceable (start-of-sequence) iterator 10
steps, which falls outside its valid range.

Objects involved in the operation:
iterator @ 0x0xffffcbc0 {
type = __gnu_cxx::__normal_iterator<int*,
std::__cxx1998::vector<int, std::allocator<int> > > (mutable iterator);
state = dereferenceable (start-of-sequence);
references sequence with type 'std::__debug::vector<int,
std::allocator<int> >' @ 0x0xffffcb30
}
Aborted (core dumped)

MrSpoo...@dggw8.org

unread,

Jun 5, 2021, 5:28:20 AM6/5/21

to

On Fri, 4 Jun 2021 13:30:16 -0700
"Chris M. Thomasson" <chris.m.t...@gmail.com> wrote:
>On 6/4/2021 10:04 AM, James Kuyper wrote:
>> I've never attempted implementing any of the standard containers (not
>> because I couldn't, but just to avoid reinventing the wheel), so I might
>> be missing some important detail (possibly one introduced by recent
>> updates - I'm sill more somewhat shaky in my understanding of some of
>> the features introduced after C++ 2003).
>
>I always thought that implementing standard containers in a C++ course
>would be fun. The teacher says something like: We are going to implement
>several std containers under the namespace std_course. Imvvho, it would
>be an interesting and worth while exercise.

Writing a basic implementation of a doubly linked list used to be a fairly
standard interview test for C programmers back in the day before C++ became
popular. You'd be amazed (or maybe not) how many of them didn't have a clue
what such a construct even was never mind how to implement it.

A way around _HAS_ITERATOR_DEBUGGING

Bonita Montero

Alf P. Steinbach

Bonita Montero

Paavo Helde

Bonita Montero

Paavo Helde

Bonita Montero

Bonita Montero

Paavo Helde

Alf P. Steinbach

James Kuyper

red floyd

Bonita Montero

Bonita Montero

Bonita Montero

Öö Tiib

Juha Nieminen

Bonita Montero

Bo Persson

Bonita Montero

James Kuyper

David Brown

Paavo Helde

Juha Nieminen

Juha Nieminen

MrSpo...@3p26kggzpvpn2h.gov.uk

Chris Vine

Paavo Helde

Chris Vine

James Kuyper

Paavo Helde

James Kuyper

Chris M. Thomasson

Paavo Helde

MrSpoo...@dggw8.org