Fun with reflect.SliceHeader
Eleanor McHugh
My use case is simple. I want to take an existing slice of an arbitrary type, and recast it as a []byte with adjusted length and capacity. At a later date I'll also be wanting to do the reverse. One essential restriction if at all possible is that the underlying allocated data memory should be identical.
Entries on a postcard to...
Ellie
Eleanor McHugh
Games With Brains
http://feyeleanor.tel
----
raise ArgumentError unless @reality.responds_to? :reason
Rob 'Commander' Pike
"unsafe". Using reflection alone it shouldn't be possible to cheat
like that. If it is, I'd like to know so it can be fixed.
On the other hand, with unsafe plus reflection just about anything is
possible. Have a look at src/pkg/gob/decode.go (if you have a strong
stomach) and you'll see things like
hdrp := (*reflect.SliceHeader)(unsafe.Pointer(p))
hdrp.Data = uintptr(unsafe.NewArray(atyp.Elem(), n))
hdrp.Len = n
hdrp.Cap = n
and other code that knows too much.
-rob
roger peppe
> reflect.SliceHeader isn't useful unless you're also using package
> "unsafe". Using reflection alone it shouldn't be possible to cheat
> like that. If it is, I'd like to know so it can be fixed.
i wonder if SliceHeader wouldn't be better defined in unsafe for that reason.
roger peppe
<ele...@games-with-brains.com> wrote:
> I seem to recall recently seeing a code snippet which changed the SliceHeader for an existing slice, but I don't seem to be able to find it in the mailing-list archives or via Google. Ignoring the possibility I imagined this in a fit of wishful thinking, has anyone successfully teased a slice from one type to another this way?
>
> My use case is simple. I want to take an existing slice of an arbitrary type, and recast it as a []byte with adjusted length and capacity. At a later date I'll also be wanting to do the reverse. One essential restriction if at all possible is that the underlying allocated data memory should be identical.
FWIW, i used SliceHeader in some recent code to do deep copying of structures
while retaining cycles and self-referential pointers:
rog-go.googlecode.com/hg/deepcopy
BTW, you'll have to be careful with such shenanigans when the garbage
collector gets better,
as the pointers could be freed if you only keep a pointer to the []byte form.
Eleanor McHugh
> reflect.SliceHeader isn't useful unless you're also using package
> "unsafe". Using reflection alone it shouldn't be possible to cheat
> like that. If it is, I'd like to know so it can be fixed.
It doesn't seem to be, although I may not be being creative enough ;)
> On the other hand, with unsafe plus reflection just about anything is
> possible. Have a look at src/pkg/gob/decode.go (if you have a strong
> stomach) and you'll see things like
>
> hdrp := (*reflect.SliceHeader)(unsafe.Pointer(p))
> hdrp.Data = uintptr(unsafe.NewArray(atyp.Elem(), n))
> hdrp.Len = n
> hdrp.Cap = n
>
> and other code that knows too much.
It took me several hours of unladylike language last night but I ended up with:
var _BYTE = reflect.Typeof(byte(0))
var _BYTE_SLICE = reflect.Typeof([]byte{})
var _SLICE_HEADER = reflect.Typeof(reflect.SliceHeader{})
func AsByteSlice(b interface{}) []byte {
if v, ok := reflect.NewValue(b).(*reflect.SliceValue); ok {
typesize := int(v.Type().(*reflect.SliceType).Elem().Size())
h := unsafe.Unreflect(_SLICE_HEADER, unsafe.Pointer(v.Addr())).(reflect.SliceHeader)
h.Len = typesize * v.Len()
h.Cap = typesize * v.Cap()
return unsafe.Unreflect(_BYTE_SLICE, unsafe.Pointer(&h)).([]byte)
}
panic(b)
}
It's not pretty, but it seems to do the job I want done in GoLightly. The more interesting case for me, and the one I'll be working on over the next couple of days, is int{n}<->float{n} as once the header's created it will allow the underlying data stored to be accessed as either type without incurring any additional costs. Whilst not something I actively want to encourage I can think of situations where it might be useful: the classic one is implementing Forth where being able to intermingle floats and ints on the runtime stack is an obvious (if not necessarily sensible) design choice.
Eleanor McHugh
> On 19 September 2010 20:30, Eleanor McHugh
> <ele...@games-with-brains.com> wrote:
>> I seem to recall recently seeing a code snippet which changed the SliceHeader for an existing slice, but I don't seem to be able to find it in the mailing-list archives or via Google. Ignoring the possibility I imagined this in a fit of wishful thinking, has anyone successfully teased a slice from one type to another this way?
>>
>> My use case is simple. I want to take an existing slice of an arbitrary type, and recast it as a []byte with adjusted length and capacity. At a later date I'll also be wanting to do the reverse. One essential restriction if at all possible is that the underlying allocated data memory should be identical.
>
> FWIW, i used SliceHeader in some recent code to do deep copying of structures
> while retaining cycles and self-referential pointers:
> rog-go.googlecode.com/hg/deepcopy
Thanks, that was precisely the piece of code I was thinking of. Until I saw that SliceHeader hadn't registered on my radar.
> BTW, you'll have to be careful with such shenanigans when the garbage
> collector gets better,
> as the pointers could be freed if you only keep a pointer to the []byte form.
I'm trying to isolate these bits of nastiness to as few places as possible in my code precisely because of that. However I expect I'll still fall foul of my own 'cleverness' at some point lol
Eleanor McHugh
> It took me several hours of unladylike language last night but I ended up with:
>
> var _BYTE = reflect.Typeof(byte(0))
> var _BYTE_SLICE = reflect.Typeof([]byte{})
> var _SLICE_HEADER = reflect.Typeof(reflect.SliceHeader{})
>
> func AsByteSlice(b interface{}) []byte {
> if v, ok := reflect.NewValue(b).(*reflect.SliceValue); ok {
> typesize := int(v.Type().(*reflect.SliceType).Elem().Size())
> h := unsafe.Unreflect(_SLICE_HEADER, unsafe.Pointer(v.Addr())).(reflect.SliceHeader)
> h.Len = typesize * v.Len()
> h.Cap = typesize * v.Cap()
> return unsafe.Unreflect(_BYTE_SLICE, unsafe.Pointer(&h)).([]byte)
> }
> panic(b)
> }
And if I ever end up interviewing other developers for jobs coding Go, you can bet this is just the sort of code I'll be asking them to deconstruct...
roger peppe
i think you can be a little more direct than that.
how about this (only one Unreflect necessary):
var byteSliceType = reflect.Typeof(([]byte)(nil))
func AsByteSlice(x interface{}) []byte {
v := reflect.NewValue(x).(*reflect.SliceValue)
h := *(*reflect.SliceHeader)(unsafe.Pointer(v.Addr()))
size := int(v.Type().(*reflect.SliceType).Elem().Size())
h.Len *= size
h.Cap *= size
return unsafe.Unreflect(byteSliceType, unsafe.Pointer(&h)).([]byte)
}
or, if you prefer:
func AsByteSlice(x interface{}) []byte {
v := reflect.NewValue(x).(*reflect.SliceValue)
h0 := *reflect.SliceHeader)(unsafe.Pointer(v.Addr())
size := int(v.Type().(*reflect.SliceType).Elem().Size())
h := reflect.SliceHeader{h0.Data, h0.Len * size, h0.Cap * size}
return unsafe.Unreflect(byteSliceType, unsafe.Pointer(&h)).([]byte)
}
note that the explicit panic is unnecessary, as you'll get an appropriate
panic anyway if it's passed a non-slice type.
Eleanor McHugh
> i think you can be a little more direct than that.
> how about this (only one Unreflect necessary):
>
> var byteSliceType = reflect.Typeof(([]byte)(nil))
> func AsByteSlice(x interface{}) []byte {
> v := reflect.NewValue(x).(*reflect.SliceValue)
> h := *(*reflect.SliceHeader)(unsafe.Pointer(v.Addr()))
> size := int(v.Type().(*reflect.SliceType).Elem().Size())
> h.Len *= size
> h.Cap *= size
> return unsafe.Unreflect(byteSliceType, unsafe.Pointer(&h)).([]byte)
> }
>
> or, if you prefer:
>
> func AsByteSlice(x interface{}) []byte {
> v := reflect.NewValue(x).(*reflect.SliceValue)
> h0 := *reflect.SliceHeader)(unsafe.Pointer(v.Addr())
> size := int(v.Type().(*reflect.SliceType).Elem().Size())
> h := reflect.SliceHeader{h0.Data, h0.Len * size, h0.Cap * size}
> return unsafe.Unreflect(byteSliceType, unsafe.Pointer(&h)).([]byte)
> }
Thanks :) I suspected there was some fat that could be trimmed, but you know how it it is: the longer you look at a half-dozen lines of code, the more they read like a Lovecraftian novel lol