endian, Just endian

[Howard Hinnant]

This:

http://howardhinnant.github.io/endian.html

is headed towards the pre-Issaquah mailing.

I’m providing a preview here for review purposes. Constructive criticism will be gratefully read, and if acted upon will be acknowledged.

Howard

[Thiago Macieira]

Yes, please. We desperately need that. Since 1998, as you wrote. As it is,
your proposal is very useful and should be accepted.

The only comment I have is about your explanation, which talks about "most
significant byte placed first". In fact, this applies also to bits and, thus, to
bitfields. Which brings the criticism about macros back into focus. Here's an
example that I added to Qt in 5.8:

struct ShortData {
#if Q_BYTE_ORDER == Q_LITTLE_ENDIAN
quintptr status : 8;
#endif
// note: this is only 24 bits on 32-bit systems...
qintptr msecs : sizeof(void *) * 8 - 8;

#if Q_BYTE_ORDER == Q_BIG_ENDIAN
quintptr status : 8;
#endif
};

This is a "Short DateTime Optimisation", which, like your typical SSO,
requires setting the LSB to indicate that the pointer it's in a union with is
otherwise not valid.

Another example of bitfields and endianness macros is the DNS HEADER structure
you can find in <arpa/nameser_compat.h> (see [1] if you don't have it).

Unless we can use C++17's if constexpr in a class definition...

[1] http://www.scs.stanford.edu/histar/src/pkg/uclibc/include/arpa/
nameser_compat.h

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Rene Rivera]

"A ancient, time-honored" ==> "An ancient, time-honored"

--

-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

[Zhihao Yuan]

On Fri, Aug 19, 2016 at 9:34 PM, Howard Hinnant
<howard....@gmail.com> wrote:
>
>

20.15.9 Endian -> 20.15.9 Endianness

the endian of the execution environment -> the endianness of the
execution environment

--
Zhihao Yuan, ID lichray
The best way to predict the future is to invent it.
___________________________________________________
4BSD -- http://blog.miator.net/

[Vicente J. Botet Escriba]

Wondering if the native value shouldn't go out from the enum

enum class endianness
{
little = see below,
big = see below,
unknown = see below
};

constexpr endianness native_endian = see below;


Here it is clear that endianness could be endianness::little,
endianness::big, or other endianness::unknown value.

I'm not sure of the absolute need for endianness::unknown, but it makes
more explicit, that there are other kinds of endian.

An alternative could be to use a namespace endian like Boost.Endian does

namespace endian
{
enum class order ....

constexpr order native = see below;

// ...
}

where other endian types and functions could be added later on.

Curiously Boost.Endian defines the same enumerations

namespace endian
{
enum class order
{
native = see below,
big = see below,
little = see below,
};


Vicente
[1] http://www.boost.org/doc/libs/1_61_0/libs/endian/doc/conversion.html

[dgutson .]

Of course it's something we needed so long ago.

My 2 cents: what about adding another member, "network" or alike, which will always be equal to big, so people (like me) don't need to go and look for it (e.g. when writing conversion code). My second comment is that maybe it's worth mentioning something about how this relates to the work being done in the networking TS (any relationship at all? Could be used to implement hton/ntoh functioms family, opens the possibility for new functioms or interfaces? Just a paragraph).

Very nice work Howard. Hope this gets into the std soon.

   Daniel.

>
> Howard
>
> --
> You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/E80B5A8A-BAAE-4B69-8F91-8A1836C7456D%40gmail.com.

[Bo Persson]

A great proposal, in all its simplicity.

However, there seems to be a cut and paste problem in the comments of
the example

if (endian::native == endian::big)
// handle big endian
else if (endian::native == endian::little)
// handle big endian
else
// handle little endian

which you might want to fix before publication.


Bo Persson

[Patrice Roy]

This proposal's a ray of sunshine :)

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.

To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org.

To post to this group, send email to std-pr...@isocpp.org.

To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/np9av5%24tit%241%40blaine.gmane.org.

[Howard Hinnant]

On Aug 19, 2016, at 11:00 PM, Thiago Macieira <thi...@macieira.org> wrote:
>
> On sexta-feira, 19 de agosto de 2016 22:34:47 PDT Howard Hinnant wrote:
>> This:
>>
>> http://howardhinnant.github.io/endian.html
>>
>> is headed towards the pre-Issaquah mailing.
>>
>> I’m providing a preview here for review purposes. Constructive criticism
>> will be gratefully read, and if acted upon will be acknowledged.
>
> Yes, please. We desperately need that. Since 1998, as you wrote. As it is,
> your proposal is very useful and should be accepted.
>
> The only comment I have is about your explanation, which talks about "most
> significant byte placed first". In fact, this applies also to bits and, thus, to
> bitfields.

Thanks. Bit-endian is out of scope for this proposal. Its tiny scope is a big feature.

Howard

[Howard Hinnant]

On Aug 19, 2016, at 11:01 PM, Rene Rivera <grafi...@gmail.com> wrote:
>
> "A ancient, time-honored" ==> "An ancient, time-honored"

Thanks fixed.

Howard

[Howard Hinnant]

> On Aug 19, 2016, at 11:24 PM, Zhihao Yuan <z...@miator.net> wrote:
>
> On Fri, Aug 19, 2016 at 9:34 PM, Howard Hinnant
> <howard....@gmail.com> wrote:
>>
>>
>> http://howardhinnant.github.io/endian.html
>>
>> is headed towards the pre-Issaquah mailing.
>>
>> I’m providing a preview here for review purposes. Constructive criticism will be gratefully read, and if acted upon will be acknowledged.
>
> 20.15.9 Endian -> 20.15.9 Endianness
>
> the endian of the execution environment -> the endianness of the
> execution environment

Thanks, fixed the latter.

Howard

[Howard Hinnant]

On Aug 20, 2016, at 5:32 AM, Vicente J. Botet Escriba <vicent...@wanadoo.fr> wrote:
>
> Le 20/08/2016 à 04:34, Howard Hinnant a écrit :
>> This:
>>
>> http://howardhinnant.github.io/endian.html
>>
>> is headed towards the pre-Issaquah mailing.
>>
>> I’m providing a preview here for review purposes. Constructive criticism will be gratefully read, and if acted upon will be acknowledged.
>>
>> Howard
>>
> Wondering if the native value shouldn't go out from the enum
>
> enum class endianness
> {
> little = see below,
> big = see below,
> unknown = see below
> };
>
> constexpr endianness native_endian = see below;
>
>
> Here it is clear that endianness could be endianness::little, endianness::big, or other endianness::unknown value.
>
> I'm not sure of the absolute need for endianness::unknown, but it makes more explicit, that there are other kinds of endian.
>
> An alternative could be to use a namespace endian like Boost.Endian does
>
> namespace endian
> {
> enum class order ....
>
> constexpr order native = see below;
>
> // ...
> }
>
> where other endian types and functions could be added later on.

Thanks, but I am fond of the simplicity of the way I have it.

> Curiously Boost.Endian defines the same enumerations
>
> namespace endian
> {
> enum class order
> {
> native = see below,
> big = see below,
> little = see below,
> };

Beman and I have discussed this area before (a couple of years ago). There has been beneficial cross pollination.

Howard

[Howard Hinnant]

On Aug 20, 2016, at 5:55 AM, dgutson . <daniel...@gmail.com> wrote:
>
> My 2 cents: what about adding another member, "network" or alike, which will always be equal to big, so people (like me) don't need to go and look for it (e.g. when writing conversion code).

Every single name added comes with cost (more to teach if nothing else). I’m aiming for “as simple as possible."

> My second comment is that maybe it's worth mentioning something about how this relates to the work being done in the networking TS (any relationship at all? Could be used to implement hton/ntoh functioms family, opens the possibility for new functioms or interfaces? Just a paragraph).

I’ve added an example hton/ntoh implementation to the description.

Howard

[Howard Hinnant]

Fixed, thanks.

Howard

[Greg Marr]

"And they can more easily be built on top of class endian once it exists. At a minimum we need class endian ..."

Should these, and a few others like them, be "enum class endian" instead?

[Howard Hinnant]

Seems reasonable, thanks.

Howard

[Bjorn Reese]

[ Quoting from the proposal ]

> An ancient, time-honored tradition among C++ programmers is detecting
> the endian of their execution environment:
>
> Detecting endianness programmatically in a C++ program

Better references may be:

https://en.wikipedia.org/wiki/Endianness
https://sourceforge.net/p/predef/wiki/Endianness/

Furthermore, endianness (byte order) is also used in Networking TS N4588
(for instance [internet.address.v4.members].)

> Objection #3: I am quite sure that PDP endian is on the way back in.
> PDP is neither big nor little endian. This proposal doesn't handle
> that!
>
> This proposal handles "mixed endian" today by ensuring that
> endian::native equals neither endian::big nor endian::little.

PDP endian is more commonly referred to as "middle endian".

[Nicol Bolas]

The functionality is interesting, but I think it is a bit under-specified.

For example, I would change this sentence:

This subclause describes the endianness of the execution environment.

to:

This subclause describes the byte ordering of all scalar types within the execution environment.

This makes it fully clear that the endian ordering applies to every scalar type, rather than focusing on the "execution environment". Similarly, statements of the form "the execution environment is big-endian" might be better as "scalar types are stored in big-endian ordering".

Furthermore, I think an example or two of the guarantees that you get would be good. Something like:

uint32_t val = 0x12345678;
unsigned char arr[sizeof(uint32_t)];
memcpy(arr, val, sizeof(uint32_t));
if constexpr (endian::native == endian::little)
    assert(arr[0] == 0x78);
else if constexpr (endian::native == endian::big)
    assert(arr[0] == 0x12);

Also, I think it would be a good idea for `endian` to use `unsigned int` as its underlying type rather than `int`. But that may just be personal preference.

[Howard Hinnant]

On Aug 21, 2016, at 4:29 PM, Nicol Bolas <jmck...@gmail.com> wrote:
>
> The functionality is interesting, but I think it is a bit under-specified.
>
> For example, I would change this sentence:
>
> This subclause describes the endianness of the execution environment.
>
> to:
>
> This subclause describes the byte ordering of all scalar types within the execution environment.
>
> This makes it fully clear that the endian ordering applies to every scalar type, rather than focusing on the "execution environment". Similarly, statements of the form "the execution environment is big-endian" might be better as "scalar types are stored in big-endian ordering”.

I’m not touching the encoding of member pointers with a ten foot pole. ;-)

I thought about pointers and arithmetic types, but decided against it. If the EWG wants to define endian, more power to them. In the meantime these words were borrowed and embellished from the POSIX spec, which is good enough for now. As always, the committee will massage these words as they may.

>
> Furthermore, I think an example or two of the guarantees that you get would be good. Something like:
>
> uint32_t val = 0x12345678;
> unsigned char arr[sizeof(uint32_t)];
> memcpy(arr, val, sizeof(uint32_t));
> if constexpr (endian::native == endian::little)
> assert(arr[0] == 0x78);
> else if constexpr (endian::native == endian::big)
> assert(arr[0] == 0x12);

The more said here, the more opportunity for error.

>
> Also, I think it would be a good idea for `endian` to use `unsigned int` as its underlying type rather than `int`. But that may just be personal preference.

I’m not against this but haven’t come up with any rationale to favor any underlying type.

Howard

[Myriachan]

What about a system for which sizeof(long long) == 1?  In other words, a system in which all standard types are the same size, with an address space addressable in words instead of what we'd more typically call "bytes"?  Not that I know of such a system.

In such a system, there is no such thing as endianness (as in byte order).  I guess that it would be handled similarly to PDP endianness?

Finally, I've heard that some systems have unusual forms of endianness for floating-point numbers but not for integers, or possibly having opposite endianness.  (Wikipedia lists older ARM systems' doubles--perhaps before hardware floating-point in ARM?)  How would those systems be handled?

On Friday, August 19, 2016 at 8:01:01 PM UTC-7, Thiago Macieira wrote:

The only comment I have is about your explanation, which talks about "most
significant byte placed first". In fact, this applies also to bits and, thus, to

bitfields. Which brings the criticism about macros back into focus. Here's an
example that I added to Qt in 5.8:

As far as I know, there is also no requirement that endianness of the system control the order of bits in bitfields...?  In practice, that is the case, though.

Melissa

[Viacheslav Usov]

On Tue, Aug 30, 2016 at 9:56 AM, Myriachan <myri...@gmail.com> wrote:

> Finally, I've heard that some systems have unusual forms of endianness for floating-point numbers but not for integers, or possibly having opposite endianness.  (Wikipedia lists older ARM systems' doubles--perhaps before hardware floating-point in ARM?)  How would those systems be handled?

Good point. Is there anything in the C++ standard that would require that even different integral types have the same endianness? If not, then the class should be a template like numeric_limits. And perhaps this could just be an extension of numeric_limits then.

Cheers,

V.

[Avi Kivity]

On 08/30/2016 10:56 AM, Myriachan wrote:

On Friday, August 19, 2016 at 7:34:52 PM UTC-7, Howard Hinnant wrote:

http://howardhinnant.github.io/endian.html

is headed towards the pre-Issaquah mailing.

I’m providing a preview here for review purposes.  Constructive criticism will be gratefully read, and if acted upon will be acknowledged.

What about a system for which sizeof(long long) == 1?  In other words, a system in which all standard types are the same size, with an address space addressable in words instead of what we'd more typically call "bytes"?  Not that I know of such a system.

In such a system, there is no such thing as endianness (as in byte order).  I guess that it would be handled similarly to PDP endianness?

Finally, I've heard that some systems have unusual forms of endianness for floating-point numbers but not for integers, or possibly having opposite endianness.  (Wikipedia lists older ARM systems' doubles--perhaps before hardware floating-point in ARM?)  How would those systems be handled?

Perhaps the standard should specify overloads for

  void to_big_endian(const T&, char*);
  T from_big_endian(const char*, void_t<T>);
  void to_little_endian(const T&, char*);
  T from_big_endian(const char*, void_t<T>);

for all the arithmetic types (except bool).

On Friday, August 19, 2016 at 8:01:01 PM UTC-7, Thiago Macieira wrote:

The only comment I have is about your explanation, which talks about "most
significant byte placed first". In fact, this applies also to bits and, thus, to
bitfields. Which brings the criticism about macros back into focus. Here's an
example that I added to Qt in 5.8:

As far as I know, there is also no requirement that endianness of the system control the order of bits in bitfields...?  In practice, that is the case, though.

Melissa

--
You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group.

To unsubscribe from this group and stop receiving emails from it, send an email to std-proposal...@isocpp.org.

To post to this group, send email to std-pr...@isocpp.org.

To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/2d31b898-9277-496d-97da-fcb19645b221%40isocpp.org.

[Howard Hinnant]

On Aug 30, 2016, at 3:56 AM, Myriachan <myri...@gmail.com> wrote:
>
> On Friday, August 19, 2016 at 7:34:52 PM UTC-7, Howard Hinnant wrote:
>> http://howardhinnant.github.io/endian.html
>>
>> is headed towards the pre-Issaquah mailing.
>>
>> I’m providing a preview here for review purposes. Constructive criticism will be gratefully read, and if acted upon will be acknowledged.
>>
>
> What about a system for which sizeof(long long) == 1? In other words, a system in which all standard types are the same size, with an address space addressable in words instead of what we'd more typically call "bytes"? Not that I know of such a system.

About 16 years ago I worked on a DSP where this was true. Everything was 24 bits: from char to long long to double. In practice it won’t matter how native is set on that platform because in typical use (as shown in the example hton implementation, code needs to check for sizeof == 1 (which is true for all types on the system you describe). I.e. on such a system, even though the hton code says you’re checking endian::native, you’re always short-circuiting the logic before you get to that check because sizeof(Integral) == 1.

>
> In such a system, there is no such thing as endianness (as in byte order). I guess that it would be handled similarly to PDP endianness?

Exactly.

>
> Finally, I've heard that some systems have unusual forms of endianness for floating-point numbers but not for integers, or possibly having opposite endianness. (Wikipedia lists older ARM systems' doubles--perhaps before hardware floating-point in ARM?) How would those systems be handled?

One of the tried-and-true techniques of standardization is to:

Standardize existing practice.

__ORDER_LITTLE_ENDIAN__, __ORDER_BIG_ENDIAN__ and __BYTE_ORDER__ are existing practice with at least two major compilers (probably more but I don’t have them available to check). This proposal simply packages this existing practice using modern C++ tools.

Howard

[Nicol Bolas]

What good is standardizing existing practice if you can't actually say what that existing practice means? That's the part that doesn't make sense here.

The standard specifies behavior. Unless these values are tied into genuinely observable behavior, then standardizing them doesn't really accomplish anything. And if they are tied into genuinely observable behavior, then we need to know what that behavior is.

I don't understand the point of standardizing three numbers without giving any meaning to these numbers.

[Howard Hinnant]

I have no doubt that the LWG (or LEWG) will offer improvements to the wording. But note that we have been living with the concept of endian in the C++ standard for half a decade and the sky hasn’t fallen (yet). See [locale.stdcvt] for our current use of the terms little endian and big endian.

The proposal’s current description of endian is borrowed from what the POSIX standard says about it (http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html), which seems like a pretty good straw man to get the process started. To the best of my knowledge, no one has complained that the POSIX description of endian is meaningless.

I’ve also scanned the LWG issues list for issues in [locale.stdcvt]. There are two:

http://cplusplus.github.io/LWG/lwg-active.html#2507
http://cplusplus.github.io/LWG/lwg-defects.html#2229

Neither is concerned with this section’s use of the word “endian”. Though if you feel that this is an issue that needs to be addressed in the current standard, here are instructions on how to get that process started:

http://cplusplus.github.io/LWG/lwg-active.html#submit_issue

Howard

[Jeffrey Yasskin]

On Tue, Aug 30, 2016 at 11:40 AM, Howard Hinnant <howard....@gmail.com> wrote:

On Aug 30, 2016, at 2:21 PM, Nicol Bolas <jmck...@gmail.com> wrote:
>
> On Tuesday, August 30, 2016 at 10:39:10 AM UTC-4, Howard Hinnant wrote:
>> On Aug 30, 2016, at 3:56 AM, Myriachan <myri...@gmail.com> wrote:
>>> >
>>> >
>>> >
>>> > Finally, I've heard that some systems have unusual forms of endianness for floating-point numbers but not for integers, or possibly having opposite endianness.  (Wikipedia lists older ARM systems' doubles--perhaps before hardware floating-point in ARM?)  How would those systems be handled?
>>>
>> One of the tried-and-true techniques of standardization is to:
>>
>>    Standardize existing practice.
>>
>> __ORDER_LITTLE_ENDIAN__, __ORDER_BIG_ENDIAN__ and __BYTE_ORDER__ are existing practice with at least two major compilers (probably more but I don’t have them available to check).  This proposal simply packages this existing practice using modern C++ tools.
>>
> What good is standardizing existing practice if you can't actually say what that existing practice means? That's the part that doesn't make sense here.
>
> The standard specifies behavior. Unless these values are tied into genuinely observable behavior, then standardizing them doesn't really accomplish anything. And if they are tied into genuinely observable behavior, then we need to know what that behavior is.
>
> I don't understand the point of standardizing three numbers without giving any meaning to these numbers.

I have no doubt that the LWG (or LEWG) will offer improvements to the wording.  But note that we have been living with the concept of endian in the C++ standard for half a decade and the sky hasn’t fallen (yet).  See [locale.stdcvt] for our current use of the terms little endian and big endian.

The proposal’s current description of endian is borrowed from what the POSIX standard says about it (http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html), which seems like a pretty good straw man to get the process started.  To the best of my knowledge, no one has complained that the POSIX description of endian is meaningless.

POSIX only has to deal with 8-bit bytes and, I think, integer types without padding, so it has an easier job than C++. That said, your definition seems clear enough, and LWG can tighten it if they want.

Jeffrey

[Myriachan]

How about the following definition?

A system uses big-endian value representation for an integer type T if, for any valid value n of type T, after executing the following code, m contains the same value as n:

unsigned char bytes[sizeof(T)];
std::memcpy(bytes, &n, sizeof(T));
T m = 0;
for (std::size_t i = 0; i < sizeof(T); ++i)
{
    m = static_cast<T>((m << std::numeric_limits<unsigned char>::digits) + bytes[i]);
}

This should work, even with signed types, and non-two's-complement representation.  Remember that it's well-defined to shift 1 bits into the sign bits of a signed integer (even though any further is undefined).

This would not work in some systems using "nails", but whether such systems could really be defined as having a particular standard endianness is itself a question.

Floating-point types could have a definition if std::numeric_limits<T>::is_iec559 is true.  If not, the answer is basically meaningless from a standardization point of view.

Melissa

[Howard Hinnant]

On Aug 30, 2016, at 5:42 PM, Myriachan <myri...@gmail.com> wrote:
>
> How about the following definition?
>
> A system uses big-endian value representation for an integer type T if, for any valid value n of type T, after executing the following code, m contains the same value as n:
>
> unsigned char bytes[sizeof(T)];
> std::memcpy(bytes, &n, sizeof(T));
> T m = 0;
> for (std::size_t i = 0; i < sizeof(T); ++i)
> {
> m = static_cast<T>((m << std::numeric_limits<unsigned char>::digits) + bytes[i]);
> }
>
>
> This should work, even with signed types, and non-two's-complement representation. Remember that it's well-defined to shift 1 bits into the sign bits of a signed integer (even though any further is undefined).
>
> This would not work in some systems using "nails", but whether such systems could really be defined as having a particular standard endianness is itself a question.
>
> Floating-point types could have a definition if std::numeric_limits<T>::is_iec559 is true. If not, the answer is basically meaningless from a standardization point of view.

That looks reasonable to me. I would want the language lawyers over in SG12 (Undefined and Unspecified Behavior) to look at it though. I keep loosing track of what’s UB and what’s not.

Howard