code review 4595047: encoding/binary: provide a NativeEndian ByteOrder. (issue4595047)

[dand...@google.com]

Reviewers: golang-dev_googlegroups.com,

Message:
Hello golan...@googlegroups.com (cc: golan...@googlegroups.com),

I'd like you to review this change to
https://go.googlecode.com/hg/

Description:
encoding/binary: provide a NativeEndian ByteOrder.

This is very useful when serializing C structs to communicate
with the local kernel.

Please review this at http://codereview.appspot.com/4595047/

Affected files:
M src/pkg/encoding/binary/binary.go

Index: src/pkg/encoding/binary/binary.go
===================================================================
--- a/src/pkg/encoding/binary/binary.go
+++ b/src/pkg/encoding/binary/binary.go
@@ -12,6 +12,7 @@ import (
"io"
"os"
"reflect"
+ "unsafe"
)

// A ByteOrder specifies how to convert byte sequences into
@@ -36,6 +37,21 @@ var LittleEndian littleEndian
// BigEndian is the big-endian implementation of ByteOrder.
var BigEndian bigEndian

+// NativeEndian is the ByteOrder of the current system.
+var NativeEndian ByteOrder
+
+func init() {
+ // Examine the memory layout of an int16 to determine system
+ // endianness.
+ var one int16 = 1
+ b := (*byte)(unsafe.Pointer(&one))
+ if *b == 0 {
+ NativeEndian = BigEndian
+ } else {
+ NativeEndian = LittleEndian
+ }
+}
+
type littleEndian unused

func (littleEndian) Uint16(b []byte) uint16 { return uint16(b[0]) |
uint16(b[1])<<8 }

[r...@golang.org]

This is in the wrong place. If the need is to serialize C structs, put
it in C code. Go shouldn't need to care about byte order.

Also, it's a shame to put package unsafe in here.

http://codereview.appspot.com/4595047/

[David Anderson]

On Sat, Jun 11, 2011 at 14:47, <r...@golang.org> wrote:

This is in the wrong place. If the need is to serialize C structs, put
it in C code.  Go shouldn't need to care about byte order.

It would be nice to be able to be byte-agnostic in all situations. My programs of choice however use strange linux kernel APIs (notably Netlink). These APIs look like network protocols, but mandate that the fields shall be encoded with the system's native endianness. In practice, on the kernel side a packed C struct is used to deserialize the packets, which is why I talked about serializing C structs. That was a misnomer, I want to serialize data to binary protocols that require native-endian encoding.

Also, it's a shame to put package unsafe in here.

I agree. The unsafe C-style cast is the only mechanism I know of to reliably determine endianness of the current machine. Any suggestions on how to avoid the use of unsafe?

 

http://codereview.appspot.com/4595047/

--
Google Switzerland GmbH
Brandschenkestrasse 110,
Zurich, Switzerland
8002 Zurich

Identifikationsnummer:
CH-020.4.028.116-1

[Rob 'Commander' Pike]

On 12/06/2011, at 8:29 AM, David Anderson wrote:

On Sat, Jun 11, 2011 at 14:47, <r...@golang.org> wrote:

This is in the wrong place. If the need is to serialize C structs, put
it in C code.  Go shouldn't need to care about byte order.

It would be nice to be able to be byte-agnostic in all situations. My programs of choice however use strange linux kernel APIs (notably Netlink). These APIs look like network protocols, but mandate that the fields shall be encoded with the system's native endianness. In practice, on the kernel side a packed C struct is used to deserialize the packets, which is why I talked about serializing C structs. That was a misnomer, I want to serialize data to binary protocols that require native-endian encoding.

Also, it's a shame to put package unsafe in here.

I agree. The unsafe C-style cast is the only mechanism I know of to reliably determine endianness of the current machine. Any suggestions on how to avoid the use of unsafe?

If you don't have C code that you're talking to that can provide the information, unsafe is the only reasonable way I can think of that will do the job. That does not mean, however, that the test belongs in a public package. Put it in your own code.

-rob

[David Anderson]

Fair enough.

 

-rob

[Russ Cox]

> That was a misnomer, I want to serialize data
> to binary protocols that require native-endian encoding.

That's not a serialization. That's a pointer to a struct.
Look at how the netlink code in package syscall
(IfAddrmsg, for example) works.

Russ

[Martin Capitanio]

IIRC I was forced to parse some native-endian encoded data coming

from a virtual char device ...

Martin 

[David Anderson]

RFC 3549 defines netlink as a packet protocol. As it happens, the initial C userspace implementation that led to that RFC used packed structs to construct those packets, because in C that's the most straightforward and kernelly way of doing it. But from other languages, netlink can (and should imho, but that's just personal religion) be considered purely as a network protocol, rather than as an arcane struct-based kernel API that happens to be bolted onto a socket.

- Dave

[Martin Capitanio]

On Sunday, June 12, 2011 9:30:23 PM UTC+2, David Anderson wrote:

RFC 3549 defines netlink as a packet protocol. As it happens, the initial C userspace implementation that led to that RFC used packed structs to construct those packets, because in C that's the most straightforward and kernelly way of doing it. But from other languages, netlink can (and should imho, but that's just personal religion) be considered purely as a network protocol, rather than as an arcane struct-based kernel API that happens to be bolted onto a socket.

A religion is obviously not helping with the real problem.

For some kernel data you basically need to know the CPU

byte order. But I can think of Russ's pragmatic answer ;-)

It doesn't matter, the war is over, all big-endians are dead.

Martin

[Rob 'Commander' Pike]

Except for ARM processors, which are big-endian I believe, and PC
graphics, which is half big-endian, and ....

-rob

[Martin Capitanio]

On Monday, June 13, 2011 2:21:53 AM UTC+2, r wrote:

Except for ARM processors, which are big-endian I believe, and PC
graphics, which is half big-endian, and ....

AFAIK ARM is only in the theory BE capable

and the dying rest (PowerPC, ...) doesn't seems to be

supported by gc anyway.

Martin

[Florian Weimer]

* Rob Pike:

> Except for ARM processors, which are big-endian I believe, and PC
> graphics, which is half big-endian, and ....

ARM can be both, some implementations are even switchable (similar to
PowerPC). Early ARM ABIs for Linux were big-endian, but current ones
are not.

[Martin Capitanio]

In the theory -- yes, in the praxis -- no. If you build the SoC or wire

the peripherals, you have to take the choice and then you can't switch

any more. There are probably still BE capable network processors

around, but (in my opinion) it seems to be increasingly difficult to find them.

Martin