P0298: A byte type definition: with undefined pointer arithmetic?

[Kazutoshi Satoda]

from P0298R2: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0298r2.pdf
> Motivation and Scope
> Many programs require byte-oriented access to memory. ...

Could you please show examples of such programs?


As I read N4623 (WG21 2016-11 Issaquah Minutes) and saw that the
proposal was about to pass through, I want to understand more concretely
what is going.

I suspect that such programs do something like obtaining unsigned char*
which points to an int object via reinterpret_cast and iterating through
it as if the pointer points to an element of unsigned char[sizeof(int)],
as I personally have seen many such programs.

However, AFAIK, the current standard doesn't (or fails to) give
well-defined behavior to such byte-oriented access on objects of
arbitrary types.
N4618 5.7 [expr.add] p6 says:
> For addition or subtraction, if the expressions P or Q have type
> "pointer to cv T", where T and the array element type are not similar
> (4.5), the behavior is undefined.

Then, if "Many programs require ..." is saying what I suspect, I think
defining such byte-oriented accesses should be done before (or within)
the proposal. It will also resolve CWG issue 1701:
"Array vs sequence in object representation"
http://wg21.cmeerw.net/cwg/issue1701

--
k_satoda

[D. B.]

I'm very interested in the resolution of that issue for the same reasons.

Currently, by my understanding, the idea that any object can be read as a "sequence" of ((un)signed) char* is not massively useful, when one seemingly cannot iterate past the first element in that sequence... It then means even memcpy cannot be implemented without special magic by the compiler.

[Neil MacIntosh]

Hello Kazutoshi,

There are plenty of examples of programs that perform byte-oriented access in the domain of operating systems and communications protocols. Consider reading or writing a wire-protocol, for example. In fact, you seem to go on to give a brief sketch of exactly the sort of byte oriented access people want to perform.

Rather than speculate on core wording myself, and whether or not the pattern you described is undefined behavior, I'll just observe that the Core Working Group reviewed the proposal (twice - once in Oulu, once in Issaquah) and approved it for adoption.

If you have specific questions about array-of-byte access to object representations, then these questions would be completely orthogonal to the std::byte proposal. They would exist today with unsigned char, and they would continue to exist whether or not std::byte is adopted. So modifying the proposal to address that issue is unnecessary. This proposal is about creating a distinct type for accessing bytes of memory - something that can also be done today using unsigned char or char. The semantics of such access should not be changed by the proposal...it's a different issue.

Hope that helps,

Neil.

-----Original Message-----
From: Kazutoshi Satoda [mailto:k_sa...@f2.dion.ne.jp]
Sent: Wednesday, January 4, 2017 2:43 AM
To: Neil MacIntosh <Neil.Ma...@microsoft.com>; std-pr...@isocpp.org
Subject: P0298: A byte type definition: with undefined pointer arithmetic?

from P0298R2: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.open-std.org%2Fjtc1%2Fsc22%2Fwg21%2Fdocs%2Fpapers%2F2016%2Fp0298r2.pdf&data=02%7C01%7CNeil.MacIntosh%40microsoft.com%7C931c7d44cdb941e437b208d4348e8184%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C1%7C636191234041675917&sdata=Cl8GsBUSn8IgERF6qnnr7G3uCHZBeOmF1cFvwaqajVQ%3D&reserved=0

https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwg21.cmeerw.net%2Fcwg%2Fissue1701&data=02%7C01%7CNeil.MacIntosh%40microsoft.com%7C931c7d44cdb941e437b208d4348e8184%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636191234041685924&sdata=RP%2BMlYq7xWlz5BrM13jeaG9fUL6U13iUEpApP5Plgk8%3D&reserved=0

--
k_satoda

[Myriachan]

On Wednesday, January 4, 2017 at 2:54:36 AM UTC-8, D. B. wrote:

I'm very interested in the resolution of that issue for the same reasons.

Currently, by my understanding, the idea that any object can be read as a "sequence" of ((un)signed) char* is not massively useful, when one seemingly cannot iterate past the first element in that sequence... It then means even memcpy cannot be implemented without special magic by the compiler.

I think that the rule ought to be that char/unsigned char pointer arithmetic is valid so long as you don't leave the bounds (or 1 past end) of the dynamic type of the most-derived object.  Arithmetic with pointers of other types would be sketchy at best.

It ought to be legal to do this:

struct Meow
{
    int x;
    float y;
    int z;
};

void Function(Meow *meow)
{
    int *p = &meow.x;
    unsigned char *b = reinterpret_cast<unsigned char *>(p);
    b += offsetof(Meow, z) - (offsetof(Meow, x) + sizeof(meow->x));
    int *q = reinterpret_cast<int *>(b);  // OK because pointer is equal to a valid int * pointer
    *q = *p;   // same as meow->z = meow->x;
}

Melissa

[D. B.]

Indeed, that's pretty much what I would like... not for selfish reasons anymore, since the design where I thought I needed that had worse UB that I had to resolve ;-) but simply because it makes sense, makes the "sequence" concept at all useful, and means that the so-called special properties of char* pointers can actually exist.

But when I (thought I) needed this before, it was a fairly simple situation where an object would have a method invoked on it, taking a pointer to one of its members, and was then supposed to calculate the offset of that member within itself and do stuff with it. That hardly seems unreasonable.

[Bengt Gustafsson]

I can't say that I have checked but isn't an array of enum values equivalent to its underlying type when it comes to size and alignment?

As byte is defined as an enum with a unsigned char as underlying type pointer arithmetic should be defined as expected: to be bytewise. I don't think this requires specific mention in the standard text.

[D. B.]

On Thu, Jan 5, 2017 at 10:45 AM, Bengt Gustafsson <bengt.gu...@beamways.com> wrote:

I can't say that I have checked but isn't an array of enum values equivalent to its underlying type when it comes to size and alignment?

As byte is defined as an enum with a unsigned char as underlying type pointer arithmetic should be defined as expected: to be bytewise. I don't think this requires specific mention in the standard text.

It's worse than that! The problem is that char* itself does not seem to be allowed to have the special properties it supposedly has, for reading other objects as a "sequence" o char*, because it is formally undefined to increment that pointer to the next byte in the "sequence". The linked issue is an attempt to clarify what this nebulous "sequence" really is and whether (or why not) it is equivalent to an array for the purposes of indexing/arithmetic.

So if char* isn't allowed to do these things, std::byte won't be, regardless of any transitive properties it gets from other facets of the Standard.

[Kazutoshi Satoda]

On 2017/01/05 4:21 +0900, Neil MacIntosh wrote:
> There are plenty of examples of programs that perform byte-oriented
> access in the domain of operating systems and communications protocols.
> Consider reading or writing a wire-protocol, for example. In fact, you
> seem to go on to give a brief sketch of exactly the sort of byte
> oriented access people want to perform.

It sounds like that what I suspected is going on. Hmm.

> Rather than speculate on core wording myself, and whether or not the
> pattern you described is undefined behavior, I'll just observe that the
> Core Working Group reviewed the proposal (twice - once in Oulu, once in
> Issaquah) and approved it for adoption.
>
> If you have specific questions about array-of-byte access to object
> representations, then these questions would be completely orthogonal to
> the std::byte proposal. They would exist today with unsigned char, and
> they would continue to exist whether or not std::byte is adopted. So
> modifying the proposal to address that issue is unnecessary. This
> proposal is about creating a distinct type for accessing bytes of memory
> - something that can also be done today using unsigned char or char. The
> semantics of such access should not be changed by the proposal...it's a
> different issue.

Now I see your point of view. But I disagree. I think the defined-ness
of byte-oriented access is not orthogonal to the proposal, rather I see
it as a major factor of the impact of the proposal.

If the byte-oriented access results in undefined behavior, adding a new
byte type which help such accesses is harmful by encouraging people to
write programs with undefined behavior, and approving such a feature
will decrease reliability of the standard and the committee. Such
negative impact may outweigh the benefit of the feature.
(I personally think "the benefit" of a new byte type, even when the
said pointer arithmetic were defined, is not so significant; I can live
without it.)

I'll seek a way to ask CWG whether the approval is stable in sight of
the said problems. Please tell me if you (or someone) know a preferred
way.

--
k_satoda

[Neil MacIntosh]

I am still not sure you are correct about whether that access pattern is actually undefined behavior. I'm afraid I'm not a language lawyer ;-)

I would suggest you ask Core the question about undefined behavior via their mailing reflector.

I'm not sure how adding a new type that is distinct from existing types used to also hold both integers and characters, but has the exact same semantics as those types worsens the situation. It certainly does nothing to encourage people to write the type of code we are discussing. That sort of code is already written - millions of lines of it - and more is written every day. Passing or blocking the byte proposal will have no impact on that. I certainly don't see what sort of evidence you can point to that adding "byte" in addition to "unsigned char", would somehow reduce the reliability of the language.

The intent of the proposal is to enable people to be less ambiguous about their intended use of a pointer (are they using it to access some object storage, or an integer, or a character?). I think it is relatively uncontroversial to say that having distinct types for these different purposes makes programs easier to understand, analyze and maintain.

Neil.

-----Original Message-----
From: Kazutoshi Satoda [mailto:k_sa...@f2.dion.ne.jp]
Sent: Thursday, January 5, 2017 10:21 AM
To: Neil MacIntosh <Neil.Ma...@microsoft.com>; std-pr...@isocpp.org

[Kazutoshi Satoda]

On 2017/01/07 8:27 +0900, Neil MacIntosh wrote:
> I am still not sure you are correct about whether that access pattern is
> actually undefined behavior. I'm afraid I'm not a language lawyer ;-)

For me, it seems somewhat easy. Consider the following concrete example:

int i = 0;
unsigned char* p = reinterpret_cast<unsigned char*>(&i);
p + 1;

"p" points to "i", which is an object of type int.

N4618 5.7 [expr.add] p6 says:
> For addition or subtraction, if the expressions P or Q have type
> "pointer to cv T", where T and the array element type are not similar
> (4.5), the behavior is undefined.

and footnote 86:
> An object that is not an array element is considered to belong to a
> single-element array for this purpose; see 5.3.1.

"T" and "the array element type" in the above example are unsigned char
and int, respectively. Then, because unsigned char and int are not
similar, the behavior is undefined.

Can you assert otherwise?

> I would suggest you ask Core the question about undefined behavior via
> their mailing reflector.

Thank you. But I can't do that by myself now because I'm not a member of
national body, and don't have attended a meeting.
https://stackoverflow.com/questions/9070651/c-standards-committee-reflector-mailing-lists

I'll seek someone who can forward this to the Core mailing reflector.
(Do you know?)

> I'm not sure how adding a new type that is distinct from existing
> types used to also hold both integers and characters, but has the exact
> same semantics as those types worsens the situation. It certainly does
> nothing to encourage people to write the type of code we are discussing.

If the proposal got adopted, someone will write an example code to
demonstrate the new feature. But it will be hard to make such an example
without violating the above explained language rule.

> That sort of code is already written - millions of lines of it - and
> more is written every day. Passing or blocking the byte proposal will
> have no impact on that.

The situation sounds same with singed integer overflow, or type based
aliasing rule. People write that sort of code at their own risk, or with
a guarantee given by the implementation. But it doesn't mean the
committee can ignore the existing language rule to standardize a new
feature.

> I certainly don't see what sort of evidence you can point to that adding
> "byte" in addition to "unsigned char", would somehow reduce the
> reliability of the language.

If something says "Here is the feature to do X, but X results in
undefined behavior", I can't rely on that, can you?

> The intent of the proposal is to enable people to be less ambiguous
> about their intended use of a pointer (are they using it to access some
> object storage, or an integer, or a character?). I think it is
> relatively uncontroversial to say that having distinct types for these
> different purposes makes programs easier to understand, analyze and
> maintain.

It sounds like that it is also good to add [[type-punned]] attribute for
pointers without touching existing type based aliasing rule, or to add
[[overflow-wraps]] attribute for signed integer types without touching
existing rule on overflow. It might be actually good in some points you
mentioned, but it adds the language another inconsistency which the
standard should reduce/avoid.

--
k_satoda

[Chris Hallock]

On Wednesday, January 4, 2017 at 2:43:24 AM UTC-8, Kazutoshi SATODA wrote:

[...] AFAIK, the current standard doesn't (or fails to) give

well-defined behavior to such byte-oriented access on objects of

arbitrary types. [...]

I, for one, would be shocked if the Committee (or anyone, really) actually wants byte-oriented pointer arithmetic within objects to be undefined. I strongly anticipate that CWG will accept issue 1701 as a defect and make it well-defined. Remember, we're not really talking about the byte accesses themselves; such aliasing is already well-defined for char and unsigned char. It is only the charPtr + i (or charPtr[i]) expression that is currently undefined, and that expression can be awkwardly worked around anyway on some platforms (e.g., cast to integer, add, cast back to pointer).

[D. B.]

On Sat, Jan 7, 2017 at 7:52 PM, Chris Hallock <christoph...@gmail.com> wrote:

I, for one, would be shocked if the Committee (or anyone, really) actually wants byte-oriented pointer arithmetic within objects to be undefined. I strongly anticipate that CWG will accept issue 1701 as a defect and make it well-defined.

I hope you're right.

 

Remember, we're not really talking about the byte accesses themselves; such aliasing is already well-defined for char and unsigned char. It is only the charPtr + i (or charPtr[i]) expression that is currently undefined, and that expression can be awkwardly worked around anyway on some platforms (e.g., cast to integer, add, cast back to pointer).

Oh, sure, on most platforms/compilers, even the non-workaround syntax just works, because the compilers don't go to lengths to maliciously subvert people's intuition. That doesn't make it well-defined per the IS.

In fact, I'm pretty sure that what you said still isn't valid. It's not the arithmetic that's the problem: it's the fact that it's performed in such a way as to generate a final pointer that doesn't point into the same contiguous array as the original source pointer (because a "sequence" of unsigned char is not an array, formally). By the same token, subtracting or comparing pointers within such a "sequence" is not valid. See:  https://www.securecoding.cert.org/confluence/display/c/ARR36-C.+Do+not+subtract+or+compare+two+pointers+that+do+not+refer+to+the+same+array These all seem like adjacent facets of the same issue to me.

[D. B.]

That said, if you could elaborate on situations in which you think casting to int, adding, then casting back formally improves things, then I'd be interested.

I had a thread here before, about the subtraction issue, and other workarounds were proposed - which, while ingenious and frankly hilarious in their inventiveness, really made me wish a lot of things were better defined. As is, we have this codified concept of 'sequence of unsigned char' but it doesn't seem useful for anything...

[Chris Hallock]

On Saturday, January 7, 2017 at 12:07:59 PM UTC-8, D. B. wrote:

That said, if you could elaborate on situations in which you think casting to int, adding, then casting back formally improves things, then I'd be interested.

Well, not realistic situations, no :). It's just a thought experiment. That is, there is no point in forbidding this kind of pointer arithmetic, even if the Committee and compiler authors wanted to, because you can acheive the same effect anyway with various, well-formed hacks, such as

• performing the offset in the integer domain
  
• performing the offset in non-C++ code (the ultimate fall-back!)
  
• repeatedly incrementing the pointer by 1 and using std::launder on it, in a loop (Myriachan's idea in an old thread, IIRC)

[D. B.]

Granted, offsetting has plenty of plausible (and slapstick) workarounds, but things like subtraction and comparison aren't nearly so lucky. Giving the object representation sequence the same guarantees as arrays of unsigned char would seem to solve this.

However, I can't recall whether there are dry theoretical reasons that the Committee might prefer to remain vague and noncommittal on this... which would be fine if not for the "sequence" wording, which to me reads like an unfulfilled hint about such allowances.

FWIW: The code where I thought I needed these was completely refactored to use arrays, albeit due to other UB that was revealed; otherwise I was seriously considering retaining the formally disallowed subtraction. So I'm really just an interested observer for now, without current insights into what should or shouldn't be allowed, as others might have.

[Ryou]

§ 1.8 p3 said (p0298r2 add "array of N std::byte" to this wording)

>If a complete object is created (5.3.4) in storage associated with another object e of type “array of N

>unsigned char”, that array provides storage for the created object if: 

So I think this is well-formed.

    unsigned char storage[ sizeof(int) ] ;

    int * p1 = new(storage) int(42) ;

    unsigned char * p2 = reinterpret_cast<unsigned char *>(p1) ;

    p2 + 1 ; // OK

This is also well-formed.

    union { int i ; unsigned char buf[sizeof(int)] ; } u ;

    u.i = 42 ;

    unsigned char * p3 = reinterpret_cast< unsigned char *>( &u ) ;

    p3 + 1 ; // OK

But this one is undefined behaviour?

    int i = 42 ;

    unsigned char * p4 = reinterpret_cast< unsigned char *>( &i ) ;

    p4 + 1 ; // Undefined behaviour?

Because the storage for object i is not associated with array of unsigned char. 

[Chris Hallock]

On Saturday, January 7, 2017 at 3:07:23 PM UTC-8, Ryou wrote:

[...]

So I think this is well-formed.

    unsigned char storage[ sizeof(int) ] ;

    int * p1 = new(storage) int(42) ;

    unsigned char * p2 = reinterpret_cast<unsigned char *>(p1) ;

    p2 + 1 ; // OK

Agreed, except that, since the int object is not pointer-interconvertable to storage[0], you also need to std::launder the result of the cast under C++17.

This is also well-formed.

    union { int i ; unsigned char buf[sizeof(int)] ; } u ;

    u.i = 42 ;

    unsigned char * p3 = reinterpret_cast< unsigned char *>( &u ) ;

    p3 + 1 ; // OK

This example also needs std::launder, but even with that, p3 would only point to the first byte in the object representation of u (or, equivalently, u.i), not u.buf[0]. u.buf[0] doesn't exist (as a live object). It doesn't exist because u.buf itself doesn't exist, and that's because it was never made the active member. There's only u.i. Since there is no live array object, the last line is strictly undefined under the current (but presumably defective) wording.

But this one is undefined behaviour?

    int i = 42 ;

    unsigned char * p4 = reinterpret_cast< unsigned char *>( &i ) ;

    p4 + 1 ; // Undefined behaviour?

Because the storage for object i is not associated with array of unsigned char. 

Yes, under the current (but presumably defective) wording. (Also, this needs std::launder.)

[Nicol Bolas]

On Saturday, January 7, 2017 at 8:13:39 PM UTC-5, Chris Hallock wrote:

On Saturday, January 7, 2017 at 3:07:23 PM UTC-8, Ryou wrote:

[...]

So I think this is well-formed.

    unsigned char storage[ sizeof(int) ] ;

    int * p1 = new(storage) int(42) ;

    unsigned char * p2 = reinterpret_cast<unsigned char *>(p1) ;

    p2 + 1 ; // OK

Agreed, except that, since the int object is not pointer-interconvertable to storage[0], you also need to std::launder the result of the cast under C++17.

I think this should be a defect in the pointer-interconvertibility rules. If an object B provides storage for A (in accord with [intro.object]/3), then they should be pointer-interconvertible, for the exact same reason as for member subobjects of unions and the first subobject of a standard layout struct.

This is also well-formed.

    union { int i ; unsigned char buf[sizeof(int)] ; } u ;

    u.i = 42 ;

    unsigned char * p3 = reinterpret_cast< unsigned char *>( &u ) ;

    p3 + 1 ; // OK

This example also needs std::launder,

A union is pointer-interconvertible with all of its member subobjects, per [basic.compound]/4:

> one is a standard-layout union object and the other is a non-static data member of that object

So laundering is not needed. You're right about the rest, though.

but even with that, p3 would only point to the first byte in the object representation of u (or, equivalently, u.i), not u.buf[0]. u.buf[0] doesn't exist (as a live object). It doesn't exist because u.buf itself doesn't exist, and that's because it was never made the active member. There's only u.i. Since there is no live array object, the last line is strictly undefined under the current (but presumably defective) wording.

But this one is undefined behaviour?

    int i = 42 ;

    unsigned char * p4 = reinterpret_cast< unsigned char *>( &i ) ;

    p4 + 1 ; // Undefined behaviour?

Because the storage for object i is not associated with array of unsigned char. 

Yes, under the current (but presumably defective) wording. (Also, this needs std::launder.)

Personally, I don't think any of those should need `std::launder`. Byte-wise access to an object has nothing to do with the purposes that `launder` was invented to solve.

[Chris Hallock]

On Saturday, January 7, 2017 at 8:31:02 PM UTC-8, Nicol Bolas wrote:

On Saturday, January 7, 2017 at 8:13:39 PM UTC-5, Chris Hallock wrote:

On Saturday, January 7, 2017 at 3:07:23 PM UTC-8, Ryou wrote:

[...]

So I think this is well-formed.

    unsigned char storage[ sizeof(int) ] ;

    int * p1 = new(storage) int(42) ;

    unsigned char * p2 = reinterpret_cast<unsigned char *>(p1) ;

    p2 + 1 ; // OK

Agreed, except that, since the int object is not pointer-interconvertable to storage[0], you also need to std::launder the result of the cast under C++17.

I think this should be a defect in the pointer-interconvertibility rules. If an object B provides storage for A (in accord with [intro.object]/3), then they should be pointer-interconvertible, for the exact same reason as for member subobjects of unions and the first subobject of a standard layout struct.

The rules seem odd to me, too. The note at the end of [basic.compound]/4 suggests that the restrictiveness is intentional, at least in part: "An array object and its first element are not pointer-interconvertible, even though they have the same address." So there's something conceptually more to pointer interconvertibility than just sharing the same address.

A union is pointer-interconvertible with all of its member subobjects, per [basic.compound]/4:
> one is a standard-layout union object and the other is a non-static data member of that object

So laundering is not needed.

reinterpret_cast can't return a pointer to u.buf[0] because it doesn't exist. Instead, the reinterpret_cast (or rather, the static_cast it delegates to) returns a pointer to u as a fall-back ("Otherwise, the pointer value is unchanged by the conversion"). The std::launder is then necessary to convert that to a pointer to the first byte in the object representation of u.

Personally, I don't think any of those should need `std::launder`. Byte-wise access to an object has nothing to do with the purposes that `launder` was invented to solve.

+1

[Kazutoshi Satoda]

On 2017/01/07 21:52 +0900, Kazutoshi Satoda wrote:
> On 2017/01/07 8:27 +0900, Neil MacIntosh wrote:
>> I would suggest you ask Core the question about undefined behavior via
>> their mailing reflector.
>
> Thank you. But I can't do that by myself now because I'm not a member of
> national body, and don't have attended a meeting.
> https://stackoverflow.com/questions/9070651/c-standards-committee-reflector-mailing-lists
>
> I'll seek someone who can forward this to the Core mailing reflector.
> (Do you know?)

I contacted to member of Japanese national body, and the discussion has
been forwarded to core-at-lists.isocpp.org. (Though I can't see the
mails by myself.)

--
k_satoda