Re: [go-nuts] how to tell endian-ness of machine

[David Symonds]

On Tue, Sep 18, 2012 at 7:11 PM, ris <ris.w...@gmail.com> wrote:

> of course I can't cast an integer pointer to a byte pointer in go

Not true. You can if you use the unsafe package and cast via unsafe.Pointer.

[Rémy Oudompheng]

On 2012/9/18 ris <ris.w...@gmail.com> wrote:
> using c I would create a union of a 32bit integer and a four byte array,
> initialize the integer with an eight character hex constant and then check
> the first byte of the array.

I'm not sure what you are trying to do. Is your result supposed to be
endianness-dependent?
You are either trying to convert the integer to an uint8 (which is
straightforward), or trying to introspect the memory representation of
the integer (which you can do using unsafe).

Rémy.

[Rémy Oudompheng]

On 2012/9/18 Rémy Oudompheng <remyoud...@gmail.com> wrote:
> On 2012/9/18 ris <ris.w...@gmail.com> wrote:
>> using c I would create a union of a 32bit integer and a four byte array,
>> initialize the integer with an eight character hex constant and then check
>> the first byte of the array.
>
> I'm not sure what you are trying to do. Is your result supposed to be
> endianness-dependent?

Sorry I think I simply misread the message.

Rémy.

[Jan Mercl]

On Tue, Sep 18, 2012 at 11:11 AM, ris <ris.w...@gmail.com> wrote:
> has anybody got a neat trick to determine endian-ness in go without
> resorting to incorporating c code?

Recommended reading
http://commandcenter.blogspot.de/2012/04/byte-order-fallacy.html

-j

[ris]

Unhelpfully, this kinda assumes I am an idiot. If I wish to display the contents of binary files that have been produced on any one of a number of different machines I might need to analyze 32 bits of data from the file as four EBCDIC characters, an IEEE floating point number or z80 assembler code. If I want a utility to tell me the endian-ness of the current machine then I can write it very neatly in C. I'd be interested if anybody knows how I can determine the endian-ness of a machine using go.

[Rémy Oudompheng]

On 2012/9/18 ris <ris.w...@gmail.com> wrote:

> Unhelpfully, this kinda assumes I am an idiot. If I wish to display the
> contents of binary files that have been produced on any one of a number of
> different machines I might need to analyze 32 bits of data from the file as
> four EBCDIC characters, an IEEE floating point number or z80 assembler code.
> If I want a utility to tell me the endian-ness of the current machine then I
> can write it very neatly in C. I'd be interested if anybody knows how I can
> determine the endian-ness of a machine using go.

Can you give an example of what you would like to write in C? It
should make answers more helpful.

Rémy.

[ris]

C code to determine endian-ness

typedef union {long n; char b[sizeof(long)];} number;

int main(int argc, char ** argv) {

    int i;

    number x;

    char * type = "unknown";

    x.n = 0x0123456789abcdef;

    if (0x01 == x.b[0]) {

        type = "big";

    } else if ((0xef & 0xff) == (x.b[0] & 0xff)) {

        type = "little";

    } else {

        printf("long number = %016lx hex\n", x.n);

        for (i=0; i<sizeof(long); i++) {

            printf("byte[%d] = %02x hex\n", i, x.b[i] & 0xff);

        }

    }

    printf("%s endian (long is %d bits)\n", type, sizeof(long) * 8);

    return 0;

}

and that code nearly translated to go

func main() {

    var x int = 0x012345678

    var p unsafe.Pointer = unsafe.Pointer(&x)

    var bp *byte = unsafe.Pointer(p)

    var endian string = "unknown"

    if 0x01 == *(*byte)(p) {

        endian = "big"

    } else if (0x78 & 0xff) == ((*(*byte)(p)) & 0xff) {

        endian = "little"

    } else {

        var i uintptr = 0

        for i = 0; i < unsafe.Sizeof(x); i++ {

            fmt.Printf("byte[%d] = %d\n", i, *(bp+1))

        }

    }

    fmt.Printf("%s endian (int is %d bits)\n", endian, unsafe.Sizeof(x)*8)

}

Casting to byte pointer and dereferencing the first byte works fine but how can I display each byte of the integer without integer arithmetic.

[Jan Mercl]

On Tue, Sep 18, 2012 at 11:59 AM, ris <ris.w...@gmail.com> wrote:
> Unhelpfully, this kinda assumes I am an idiot.

No.

> If I wish to display the
> contents of binary files that have been produced on any one of a number of
> different machines I might need to analyze 32 bits of data from the file as
> four EBCDIC characters, an IEEE floating point number or z80 assembler code.

Which has nothing to do with endianess of your machine(*) - as the
article explains.

> If I want a utility to tell me the endian-ness of the current machine then I
> can write it very neatly in C. I'd be interested if anybody knows how I can
> determine the endian-ness of a machine using go.

Writing Go in C usually gets as hard as is the opposite way.

-j

(*) Assuming you're not doing e.g. fbdev DMA or simillar things ;-)

[Rémy Oudompheng]

On 2012/9/18 ris <ris.w...@gmail.com> wrote:
> C code to determine endian-ness

I thought your goal was to "analyze 32 bits of data from binary files
produced by different machines", and that determining endianess was
only an intermediate step (for what?).

> and that code nearly translated to go
>
> func main() {
> var x int = 0x012345678
> var p unsafe.Pointer = unsafe.Pointer(&x)
> var bp *byte = unsafe.Pointer(p)
>
> var endian string = "unknown"
>
> if 0x01 == *(*byte)(p) {
> endian = "big"
> } else if (0x78 & 0xff) == ((*(*byte)(p)) & 0xff) {
> endian = "little"
> } else {
> var i uintptr = 0
> for i = 0; i < unsafe.Sizeof(x); i++ {
> fmt.Printf("byte[%d] = %d\n", i, *(bp+1))
> }
> }
> fmt.Printf("%s endian (int is %d bits)\n", endian, unsafe.Sizeof(x)*8)
> }
>
> Casting to byte pointer and dereferencing the first byte works fine but how
> can I display each byte of the integer without integer arithmetic.

Ints are not 32-bit large. Use uint32.
Don't use pointer arithmetic which is even uglier. Cast the
unsafe.Pointer to a *[4]byte instead.

Rémy.

[ris]

It's really just a lack of imagination that makes you say that. If I want to run a data analysis utility on some data from a machine with a known endian-ness I might want to check the endian-ness of the current machine first. I'd need a utility to tell me that. Simples.

[Rémy Oudompheng]

On 2012/9/18 ris <ris.w...@gmail.com> wrote:

> It's really just a lack of imagination that makes you say that. If I want to
> run a data analysis utility on some data from a machine with a known
> endian-ness I might want to check the endian-ness of the current machine
> first. I'd need a utility to tell me that. Simples.

Reading binary data encoded with a known endianness is usually done
using the methods from the standard library:
binary.LittleEndian.Uint32 or binary.BigEndian.Uint32 for example.

Rémy.

[Rémy Oudompheng]

On 2012/9/18 Rémy Oudompheng <remyoud...@gmail.com> wrote:

In the encoding/binary package: http://golang.org/pkg/encoding/binary/

[minux]

On Tue, Sep 18, 2012 at 6:49 PM, ris <ris.w...@gmail.com> wrote:

It's really just a lack of imagination that makes you say that. If I want to run a data analysis utility on some data from a machine with a known endian-ness I might want to check the endian-ness of the current machine first. I'd need a utility to tell me that. Simples.

If you already know the endianness of the data file, why don't use what Rob's article suggests?

If you are using Go, encoding/binary could help you.

[ris]

Like this

            fmt.Printf("byte[%d] = %d\n", i, (*[4]byte(p))[i])

but that doesn't compile

[Jan Mercl]

On Tue, Sep 18, 2012 at 12:49 PM, ris <ris.w...@gmail.com> wrote:
> It's really just a lack of imagination that makes you say that. If I want to
> run a data analysis utility on some data from a machine with a known
> endian-ness I might want to check the endian-ness of the current machine
> first. I'd need a utility to tell me that. Simples.

If you know the endianness of data in a file then you (have to)
process that data endianessaware. Processing external data sources
(like files, network packets, ...) in a
endianess-of-that-external-data-aware-way doesn't need to know the
endianess of the machine executing the processing program.

You might want to check the endianness of the current machine first.
But you don't have to and doing that is (most often) a mistake.
Simples.

-j

[ris]

Fair enough. Incidentally, does anyone know how to tell the endian-ness of a machine using go?

[David Symonds]

On Tue, Sep 18, 2012 at 9:14 PM, ris <ris.w...@gmail.com> wrote:

> Fair enough. Incidentally, does anyone know how to tell the endian-ness of a
> machine using go?

We've told you multiple times: cast an integer to a byte array via
unsafe.Pointer.

[ris]

See earlier posting of my code. Doing so allows me to get first byte but not subsequent ones.

[Dan Kortschak]

*(*[4]byte)(unsafe.Pointer(i))

--
 
 

[Francisco Souza]

On Tue, Sep 18, 2012 at 8:59 AM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> *(*[4]byte)(unsafe.Pointer(i))
>

Just (*[4]byte)(unsafe.Pointer(i)) is enough.

--
~f

[ris]

I don't understand what to make of this as I have an unsafe.Pointer "p" pointing to an int and I have a variable "i" which is a byte offset from the first byte of the int.

The problem is go doesn't allow me to add the offset to the pointer and I just don't know how to otherwise get at the internal bytes of the int.

[Dan Kortschak]

why do you need to add an offset when you can index into the array?

[ris]

Are you suggesting I can do something like:

            fmt.Printf("byte[%d] = %d\n", i, (*[4]byte(p)) + (*[4]byte(unsafe.Pointer(i))))

because that gives compile time erros

./ndn.go:21: cannot convert p (type unsafe.Pointer) to type [4]byte

./ndn.go:21: invalid indirect of ([4]byte)(p) (type [4]byte)

./ndn.go:21: cannot convert unsafe.Pointer(i) (type unsafe.Pointer) to type [4]byte

./ndn.go:21: invalid indirect of ([4]byte)(unsafe.Pointer(i)) (type [4]byte)

[Larry Clapp]

Is this what you're looking for?

func main() {

  var i uint32 = 0x12345678;

  x := (*[4]byte)(unsafe.Pointer(&i))

  for _, xn := range *x {

    fmt.Printf("%x\n", xn)

  } 

} 

=>

78

56

34

12

-- L

[ris]

By index index into the array, do you mean like this:

                        fmt.Printf("byte[%d] = %d\n", i, (*[4]byte(p))[i])

because that gives compile time errors

./ndn.go:21: cannot convert p (type unsafe.Pointer) to type [4]byte

./ndn.go:21: invalid indirect of ([4]byte)(p) (type [4]byte)

[ris]

You are receiving long and heartfelt applause *thumbsup*

[Rémy Oudompheng]

The star operator has low priority, just like in C. You must enclose
the pointer type in parentheses.

Rémy.


2012/9/18, ris <ris.w...@gmail.com>:

> By index index into the array, do you mean like this:
>
> fmt.Printf("byte[%d] = %d\n", i, (*[4]byte(p))[i])
>
> because that gives compile time errors
>
> ./ndn.go:21: cannot convert p (type unsafe.Pointer) to type [4]byte
> ./ndn.go:21: invalid indirect of ([4]byte)(p) (type [4]byte)
>
> On Tuesday, September 18, 2012 1:13:12 PM UTC+1, kortschak wrote:
>>
>> why do you need to add an offset when you can index into the array?
>>

>> On 18/09/2012, at 9:38 PM, "ris" <ris.w...@gmail.com <javascript:>>

>> wrote:
>>
>> > I don't understand what to make of this as I have an unsafe.Pointer "p"
>> >
>> pointing to an int and I have a variable "i" which is a byte offset from
>> the first byte of the int.
>> >
>> > The problem is go doesn't allow me to add the offset to the pointer and
>> >
>> I just don't know how to otherwise get at the internal bytes of the int.
>>
>

> --
>
>
>

[Francisco Souza]

func main() {
var x int = 0x012345678
var p unsafe.Pointer = unsafe.Pointer(&x)

var bp *[4]byte = (*[4]byte)(p)

var endian string = "unknown"

if 0x01 == bp[0] {
endian = "big"
} else if (0x78 & 0xff) == (bp[0] & 0xff) {

endian = "little"
} else {
var i uintptr = 0
for i = 0; i < unsafe.Sizeof(x); i++ {

fmt.Printf("byte[%d] = %d\n", i, bp[i])

}
}
fmt.Printf("%s endian (int is %d bits)\n", endian, unsafe.Sizeof(x)*8)
}

> --
>
>



--
~f

[Francisco Souza]

sorry, looks like my email was out of sync :(

> --
>
>



--
~f

[Larry Clapp]

Cool, glad to help out.

Having established some "cred", let me just say that I guess I lack imagination, 'cause I find Rob's article pretty convincing.  If you know the encoding of the source data, why would you care what the current machine is?

I guess you're trying to do something like "If endian-ness of incoming data is same as endian-ness of current machine, read the data byte for byte; else byte-swap incoming data".  But why not take the portable approach and just do as Rob suggested?

Let's say your data stream has a little-endian-encoded 32-bit integer. Here's how to extract it (assuming unsigned bytes):
i = (data[0]<<0) | (data[1]<<8) | (data[2]<<16) | (data[3]<<24);
If it's big-endian, here's how to extract it:
i = (data[3]<<0) | (data[2]<<8) | (data[1]<<16) | (data[0]<<24);

Again, I'm not saying you're wrong, just that I don't understand.

-- L

[ris]

So, I have some binary data.

I know which machine it came from but I don't know the endian-ness of
that machine.

I run a little utility on that machine to tell me which endian-ness it
is.

Now I can run a data analysis program on another machine (the endian-
ness of which I don't need to know) with a command line argument or
environment variable telling the code whether to take bytes front to
back or back to front as in Rob's article.

I am hoping not to be inundated with questions about why I cannot run
the data analysis program on the same machine as the data file came
from because I'm not saying :-P

Also, I'm not answering any questions about why I keep putting a dash
in endian-ness :-)

On Sep 18, 1:36 pm, Larry Clapp <la...@theclapp.org> wrote:
> Cool, glad to help out.
>
> Having established some "cred", let me just say that I guess I lack
> imagination, 'cause I find Rob's article pretty convincing.  If you know
> the encoding of the source data, why would you care what the current
> machine is?
>

> I *guess *you're trying to do something like "If endian-ness of incoming

[roger peppe]

On 18 September 2012 13:48, ris <ris.w...@gmail.com> wrote:
> I run a little utility on that machine to tell me which endian-ness it
> is.

Why not just write it in C? It's only two lines of code. :-)

[ris]

Yes, I see. That works nicely too. The different style of declaration
between c and go is going to mess with my chi for a while.

[ris]

Grrrrrrrrrrr

On Sep 18, 1:57 pm, roger peppe <rogpe...@gmail.com> wrote:

[Rémy Oudompheng]

On 2012/9/18 ris <ris.w...@gmail.com> wrote:

> So, I have some binary data.
>
> I know which machine it came from but I don't know the endian-ness of
> that machine.
>
> I run a little utility on that machine to tell me which endian-ness it
> is.

Are you using gccgo? I think the architectures supported by "the main
Go distribution" are all little-endian.

Rémy.

[ris]

Please, let us never talk of endian-ness again!

I have a nice little snippet of go code with examples of casting and array ranging which will do nicely as I explore the ramifications of this language.

func main() {

    var x int = 0x012345678

    var bp *[4]byte = (*[4]byte)(unsafe.Pointer(&x))

    var endian string = "unknown"

    if 0x01 == bp[0] {

        endian = "big"

    } else if (0x78 & 0xff) == (bp[0] & 0xff) {

        endian = "little"

    } else {

        for _, b := range bp {

            fmt.Printf("%x\n", b)

        }

    }

    fmt.Printf("%s endian (int is %d bits)\n", endian, unsafe.Sizeof(x)*8)

}

[roger peppe]

On 18 September 2012 14:25, ris <ris.w...@gmail.com> wrote:
> Please, let us never talk of endian-ness again!
>
> I have a nice little snippet of go code with examples of casting and array
> ranging which will do nicely as I explore the ramifications of this
> language.
>
> func main() {
> var x int = 0x012345678
> var bp *[4]byte = (*[4]byte)(unsafe.Pointer(&x))
> var endian string = "unknown"

FWIW you can use := above, which is a little
more idiomatic. Also, it's worth not assuming
that int is 4 bytes.

x := 0x012345678
bp := (*[unsafe.Sizeof(x)]byte)(unsafe.Pointer(&x))
endian := "unknown"

[ris]

Yep. Even better.

[John Asmuth]

Not true. Once you get the pointer, you can do arithmetic on it for whatever you like. Also what kortschak suggests.

[Rob Pike]

var x uint32 = 0x01020304
switch *(*byte)(unsafe.Pointer(&x)) {
case 0x01:
println("big")
case 0x04:
println("little")
}

-rob

[Jan Mercl]

big or not: http://play.golang.org/p/CWGVSpnANx

-j

[ohir]

Beware, it is very old thread (2012). I'm posting an answer for others who will search for this topic.

On Tuesday, September 18, 2012 at 1:14:52 PM UTC+2, Ris Wright wrote:

does anyone know how to tell the endian-ness of a machine using go?

import "runtime"

    switch runtime.GOARCH {
    case "mips64", "ppc64":
        ourOrderIsBE = true
    }

// case clause will need maintenace with more BE archs coming
// list such architectures with:
// grep -rn 'BigEndian\s\+=\s\+1' $GOROOT/src/runtime/*|cut -sf2 -d_|cut -d. -f1

http://play.golang.org/p/t6QDLNV-jk

--
Wojciech S. Czarnecki
       ^oo^ OHIR-RIPE