UnmarshalJSON on non string map question

[Christopher Helck]

I have:

type Foo struct {

    x map[int]string

}

I can't encode this to JSON because encoder only works with maps with String keys. Is it possible to override this by providing my own Marshal and Unmarshal methods?

If yes, is there an example somewhere?

Thanks,

C. Helck

[Monnand]

Well, if I were you, I would rather write some function convert map[int]string to map[string]string and then use the existing json pkg.

[Arlen Cuss]

On Thu, 2011-06-09 at 23:09 -0400, Christopher Helck wrote:
> I have:
>
>
> type Foo struct {
> x map[int]string
> }
>
>
> I can't encode this to JSON because encoder only works with maps with
> String keys. Is it possible to override this by providing my own
> Marshal and Unmarshal methods?

You could, but then you'd no longer be encoding to JSON:

http://www.json.org/

JSON only supports string keys in objects. JavaScript's object literal
syntax, OTOH, supports a far wider range of key types, and syntaxes for
that (e.g. unquoted strings, single-quotes, etc.).

--
Arlen Cuss
Software Engineer

Phone: +61 3 9877 9921
Email: ar...@noblesamurai.com

Noble Samurai Pty Ltd
Level 1, 234 Whitehorse Rd
Nunawading, Victoria, 3131, Australia

noblesamurai.com | arlen.co

[Christopher Helck]

Sorry if I've been unclear. It seems to me that an encoder for a specific field in an object could convert it to a form compatible with json. So integer keys just get atoi() called on them when encoding, and itoa() when decoding. -Chris

[Russ Cox]

Yes. The simplest example is json.RawMessage:

// RawMessage is a raw encoded JSON object.
// It implements Marshaler and Unmarshaler and can
// be used to delay JSON decoding or precompute a JSON encoding.
type RawMessage []byte

// MarshalJSON returns *m as the JSON encoding of m.
func (m *RawMessage) MarshalJSON() ([]byte, os.Error) {
return *m, nil
}

// UnmarshalJSON sets *m to a copy of data.
func (m *RawMessage) UnmarshalJSON(data []byte) os.Error {
if m == nil {
return os.NewError("json.RawMessage: UnmarshalJSON on nil pointer")
}
*m = append((*m)[0:0], data...)
return nil
}

but you can use any type of your own as long as you
implement the MarshalJSON and UnmarshalJSON methods.

Russ

[Christopher Helck]

Good timing, I was just about to post my own example. Here it is. 

%<--------------------------------------------------

package main

import (

"fmt"

"json"

"os"

"reflect"

"strconv"

"testing"

)

// Struct we want to encode/decode to JSON

type Numbers struct {

StringToInt    map[string]int

IntToString    map[int]string     // JSON will have a fit over this, need custom encoder/decoder.

}

func NewNumbers() Numbers {

return Numbers{make(map[string]int, 0), make(map[int]string, 0)}

}

func (self Numbers) add(n int, s string) {

self.StringToInt[s]=n

self.IntToString[n]=s

}

// This struct is acceptable to JSON. Map keys are strings.

type memento struct {

F1    map[string]int

F2    map[string]string

}

// Give a Numbers object, create the memento

func NewMemento(n *Numbers) *memento {

f2 := make(map[string]string, len(n.IntToString))

for k,v := range n.IntToString {

f2[strconv.Itoa(k)]=v

}

return &memento{n.StringToInt, f2}

}

// Create a Numbers object from its memento

func (m memento) newNumbers() (*Numbers, os.Error) {

f2 := make(map[int]string, len(m.F2))

for k,v := range m.F2 {

ik,err := strconv.Atoi(k)

if err != nil {

return nil, err

}

f2[ik]=v

}

return &Numbers{m.F1, f2}, nil

}

// To Marshal a Numbers object, just convert it to its

// memento and marshal it instead.

func (self *Numbers) MarshalJSON() ([]byte, os.Error) {

return json.Marshal(NewMemento(self))

}

// To Unmarshal into a Numbers object, first unmarshal

// into a memento, then convert the memento into Numbers.

func (self *Numbers) UnmarshalJSON(bytes []byte) os.Error {

var m memento

json.Unmarshal(bytes, &m)

n, err := m.newNumbers()

if err == nil {

*self = *n

}

return err

}

// Test normal use case

func TestNumbers(t *testing.T) {

n1 := NewNumbers()

n1.add(1, "I")

n1.add(2, "II")

n1.add(3, "III")

b,err := json.Marshal(&n1)

if err != nil {

t.Fatalf("Error: %v on Marshal(n1). n1 = %v\n", err, n1)

}

fmt.Println(string(b))

n2 := NewNumbers()

err = json.Unmarshal(b, &n2)

if err != nil {

t.Fatalf("Error: %v on Unmarshal(n1). n2 = %v\n", err, n2)

}

if !reflect.DeepEqual(n1, n2) {

t.Fatalf("n1 and n2 should be the same, but are not: %v, %v\n", n1, n2)

}

}

%<------------------------------------------------------------

There are other ways of doing it. I chose to use a memento because it was simple, but its memory foot print it pretty large. If I had tens of thousands of items in the IntToString map I might choose to marshal/unmarshal directly instead of going through the memento. In fact I played around with this a bit, and I got the feeling that the json package could be friendlier in this aspect. There are lots of private methods in the package (such as "func (e *encodeState) string(s string)") that would be handy to have when creating a custom encoder/decoder. 

I do have a question. I tried adding MarshalJSON and UnmarshalJSON to map[int]string like this:

type is map[int]string

func (self *is) MarshalJSON(...)...

func (self *is) UnmarshalJSON(...)...

I figured it be better to write customized decodes/encoders at the lowest level possible. Getting the Marshaller to work was no big problem, but I couldn't get the Unmarshaller to work. The json package would not recognize that type "is" implemented the Unmarshaller interface. Would you expect this to work? I have the code saved if it is of interest.

Thanks,

Chris Helck

[ancientlore]

From my experience, UnmarshalJSON doesn't seem to fire unless your
structure holds a pointer to the object. i.e. if T has an
UnmarshalJSON method, it is used for A but not for B:

type A struct {
Foo *T
}
type B struct {
Foo T
}

These issues are related and interesting, but I am not sure that I
understand the issue completely.

http://code.google.com/p/go/issues/detail?id=1039
http://code.google.com/p/go/issues/detail?id=1260

Mike

On Jun 12, 1:17 pm, Christopher Helck <christopher.he...@gmail.com>
wrote:

[Kyle Lemons]

For interface purposes, the method set of the type in question must explicitly satisfy the interface, whether or not you can call the method on a variable of that type.

http://golang.org/doc/go_spec.html#Method_sets

That is:

func (t *Type) UnmarshalJSON(...)

will only ever be satisfied by a var x *Type, and 

func (t Type) UnmarshalJSON(...)

will can be satisfied either by a var x *Type or a var x Type.

Note that whether you have a var x *Type or var x Type, you can always say x.UnmarshalJSON in the above example.

For the types-match-receiver case, see:

http://golang.org/doc/play/#package%20main%0A%0Aimport%20%22fmt%22%0A%0Atype%20Iface%20interface%20%7B%0A%20%20%20%20Method()%0A%7D%0A%0Atype%20PointerObject%20struct%7B%7D%0Afunc%20(p%20*PointerObject)%20Method()%20%7B%7D%0A%0Atype%20ValueObject%20struct%7B%7D%0Afunc%20(v%20ValueObject)%20Method()%20%7B%7D%0A%0Afunc%20main()%20%7B%0A%09p%20%3A%3D%20%26PointerObject%7B%7D%0A%09v%20%3A%3D%20ValueObject%7B%7D%0A%0A%09%2F%2F%20Both%20work%0A%09p.Method()%0A%09v.Method()%0A%0A%09_%2C%20pIsIface%20%3A%3D%20interface%7B%7D(p).(Iface)%0A%09_%2C%20vIsIface%20%3A%3D%20interface%7B%7D(v).(Iface)%0A%09fmt.Printf(%22pIsIface%3A%20%25v%5CnvIsIface%3A%20%25v%5Cn%22%2C%0A%09%09pIsIface%2C%20vIsIface)%0A%7D%0A

for the type-receiver-mismatch case, see:

http://golang.org/doc/play/#package%20main%0A%0Aimport%20%22fmt%22%0A%0Atype%20Iface%20interface%20%7B%0A%20%20%20%20Method()%0A%7D%0A%0Atype%20PointerObject%20struct%7B%7D%0Afunc%20(p%20*PointerObject)%20Method()%20%7B%7D%0A%0Atype%20ValueObject%20struct%7B%7D%0Afunc%20(v%20ValueObject)%20Method()%20%7B%7D%0A%0Afunc%20main()%20%7B%0A%09p%20%3A%3D%20PointerObject%7B%7D%0A%09v%20%3A%3D%20%26ValueObject%7B%7D%0A%0A%09%2F%2F%20Both%20work%0A%09p.Method()%0A%09v.Method()%0A%0A%09_%2C%20pIsIface%20%3A%3D%20interface%7B%7D(p).(Iface)%0A%09_%2C%20vIsIface%20%3A%3D%20interface%7B%7D(v).(Iface)%0A%09fmt.Printf(%22pIsIface%3A%20%25v%5CnvIsIface%3A%20%25v%5Cn%22%2C%0A%09%09pIsIface%2C%20vIsIface)%0A%7D%0A

[Michael Lore]

I get that - but when using json unmarshal, you still have to use a pointer. In this example, it doesn't matter which way you declare UnmarshalJSON, it still requires a pointer when you use it or UnmarshalJSON isn't called.

package main

import (

"json"

"log"

"os"

)

type Bar struct {

X string

}

type T Bar // doing this prevents recursive call in UnmarshalJSON since we're only injecting a print

type A struct {

Foo *T // UnmarshalJSON will be called

}

type B struct {

Foo T // UnmarshalJSON will not be called

}

var s = "{ \"Foo\": { \"X\": \"Hello, World\" } }"

func (self *T) UnmarshalJSON(x []byte) os.Error {

log.Print("inside UnmarshalJSON - ", string(x))

return json.Unmarshal(x, (*Bar)(self))

}

/* 

func (self T) UnmarshalJSON(x []byte) os.Error {

log.Print("inside UnmarshalJSON - ", string(x))

return json.Unmarshal(x, (Bar)(self))

}

*/

func main() {

var a A

json.Unmarshal([]byte(s), &a)

log.Print(a.Foo)

buf, err := json.Marshal(a)

if err != nil {

log.Print(err)

}

log.Print(string(buf))

var b B

json.Unmarshal([]byte(s), &b)

log.Print(b.Foo)

buf, err = json.Marshal(b)

if err != nil {

log.Print(err)

}

log.Print(string(buf))

[Russ Cox]

It's true: UnmarshalJSON requires a pointer on the theory
that if you don't have a pointer receiver how can you possibly
save the result of unmarshalling?

Russ

[Michael Lore]

That makes sense in a call to json.Unmarshal(), but not when I implement UnmarshalJSON() on my object which already has space allocated. In the example below, the json library is dictating the design of my structures. I am not sure why it can't just run on a pointer to the structure member I declared, if it's not already a pointer. But I haven't had time to look at the json code yet - maybe I'll understand better when I take a look.

http://golang.org/doc/play/#package%20main%0A%0Aimport%20(%0A%09%22json%22%0A%09%22log%22%0A%09%22os%22%0A)%0A%0Atype%20Bar%20struct%20%7B%0A%09X%20string%0A%7D%0A%0Atype%20T%20Bar%20%2F%2F%20doing%20this%20prevents%20recursive%20call%20in%20UnmarshalJSON%20since%20we're%20only%20injecting%20a%20print%0A%0Atype%20A%20struct%20%7B%0A%09Foo%20*T%20%2F%2F%20UnmarshalJSON%20will%20be%20called%0A%7D%0A%0Atype%20B%20struct%20%7B%0A%09Foo%20T%20%2F%2F%20UnmarshalJSON%20will%20not%20be%20called%0A%7D%0A%0Avar%20s%20%3D%20%22%7B%20%5C%22Foo%5C%22%3A%20%7B%20%5C%22X%5C%22%3A%20%5C%22Hello%2C%20World%5C%22%20%7D%20%7D%22%0A%0Afunc%20(self%20*T)%20UnmarshalJSON(x%20%5B%5Dbyte)%20os.Error%20%7B%0A%09log.Print(%22inside%20UnmarshalJSON%20-%20%22%2C%20string(x))%0A%09return%20json.Unmarshal(x%2C%20(*Bar)(self))%0A%7D%0A%0A%2F*%20%0Afunc%20(self%20T)%20UnmarshalJSON(x%20%5B%5Dbyte)%20os.Error%20%7B%0A%09log.Print(%22inside%20UnmarshalJSON%20-%20%22%2C%20string(x))%0A%09return%20json.Unmarshal(x%2C%20(Bar)(self))%0A%7D%0A*%2F%0A%0Afunc%20main()%20%7B%0A%09var%20a%20A%0A%09json.Unmarshal(%5B%5Dbyte(s)%2C%20%26a)%0A%09log.Print(a.Foo)%0A%09buf%2C%20err%20%3A%3D%20json.Marshal(a)%0A%09if%20err%20!%3D%20nil%20%7B%0A%09%09log.Print(err)%0A%09%7D%0A%09log.Print(string(buf))%0A%0A%09var%20b%20B%0A%09json.Unmarshal(%5B%5Dbyte(s)%2C%20%26b)%0A%09log.Print(b.Foo)%0A%09buf%2C%20err%20%3D%20json.Marshal(b)%0A%09if%20err%20!%3D%20nil%20%7B%0A%09%09log.Print(err)%0A%09%7D%0A%09log.Print(string(buf))%0A%7D

Thanks,

Mike

[Michael Lore]

I understand that. My question is, why can't it use reflection to determine it's not a pointer and use (&NotAPointer).UnmarshalJSON() ? That's why I was going to look at the json library to see what's going on. I haven't used the reflection API yet either; something else to look into.

Cheers,

Mike

On Tue, Jun 14, 2011 at 2:23 PM, Corey Thomasson <cthom...@gmail.com> wrote:

On 14 June 2011 13:32, Michael Lore <mi...@skyegg.com> wrote:
> That makes sense in a call to json.Unmarshal(), but not when I implement
> UnmarshalJSON() on my object which already has space allocated. In the
> example below, the json library is dictating the design of my structures. I
> am not sure why it can't just run on a pointer to the structure member I
> declared, if it's not already a pointer.

Because even if it did call NotAPointer.UnmarshalJSON() it's not
really going to unmarshal anything, because the method will be passed
a _copy_ of NotAPointer which will no longer exist once the call
returns. Consider

type Foo struct {
   X int
}

func (f Foo) Add(y int) {
   f.X += y
}

func main() {
   f := Foo{3}
   f.Add(4)
   println(f.X)
}


This code will print 3, not 7. Because in Foo.Add f is a copy of the
receiver. Likewise if Foo.Add was called Foo.UnmarshalJSON

[chris dollin]

On 14 June 2011 19:38, Michael Lore <mi...@skyegg.com> wrote:
> I understand that. My question is, why can't it use reflection to determine
> it's not a pointer and use (&NotAPointer).UnmarshalJSON() ?

Because NotAPointer is a copy of whatever you passed in and
however hard you write to the copy it can't affect the original. So
it doesn't do that.

Chris

--
Chris "allusive" Dollin

[Corey Thomasson]

Because &NotAPointer would be the address of the copy, and still not
accomplish anything.

[Corey Thomasson]

On 14 June 2011 13:32, Michael Lore <mi...@skyegg.com> wrote:

> That makes sense in a call to json.Unmarshal(), but not when I implement
> UnmarshalJSON() on my object which already has space allocated. In the
> example below, the json library is dictating the design of my structures. I
> am not sure why it can't just run on a pointer to the structure member I
> declared, if it's not already a pointer.

Because even if it did call NotAPointer.UnmarshalJSON() it's not

[Michael Lore]

So in my example &(b.Foo) is a copy? I think not. I am talking about taking the address of the object already in the structure and invoking UnmarshalJSON() on that. It's not a copy. Look at the example.

-Mike

[roger peppe]

i think the unmarshal in that example should invoke UnmarshalJSON
as you expect.

i think that the reason that it does not is that reflect.AddressOf
was not available until recently, so the JSON package was
unable to create a pointer from a value, even when the value
was addressable.