JSON Marshal/Unmarshal Extra Data

[jammi...@gmail.com]

Hello Gophers!

I'm looking to find out if what I want to do is possible with the standard json package. I created a small example to demonstrate: http://play.golang.org/p/pnLAi2WZXF. What I'm trying to do is unmarshal only the fields that I care about into my struct and any extra info will get added into the map[string]interface{}. So for example Name and Weight would be unmarshalled to their fields while Age and Loc would be entries in the map. And the struct could finally be marshalled back into the same json as before. 

What would be the correct way to do this?

[Staven]

On Wed, Jul 29, 2015 at 05:10:09AM -0700, jammi...@gmail.com wrote:
> http://play.golang.org/p/pnLAi2WZXF. What I'm trying to do is unmarshal
> only the fields that I care about into my struct and any extra info will

Is it that you don't care about those fields, or you don't know what
they're going to be in advance?

If you don't care, you can include them in the struct and just not do
anything with them.

If you don't know what they're going to be, then I don't think this is
possible to do at all with the standard json package.

If you can change the protocol to have the miscalleneous parts under
a different field, like this:

{"Name":"Alex", "Weight": 180, "Extra": { "Age": 24, "Loc": "USA"} }

it becomes trivial. You just use map[string]interface{} for Extra.

[Dustin]

If its for a specific struct, you can write a MarshalJSON and UnmarshalJSON on it which will accomplish this.

A more general solution I've seen so far is still pretty cumbersome (using the bson package):
http://devel.io/2013/08/19/go-handling-arbitrary-json/

This is an open issue: https://github.com/golang/go/issues/6213

[jammi...@gmail.com]

So my use case is that I ingest some JSON data that has lots and lots of fields, but I care about crunching some numbers and changing only on a few of these fields. For the rest of the unused fields I would like to be able to pass them through without having to create struct fields for each one. I'd like to keep around the extra info in a map b/c I need to serialize it back to json and send it down the pipeline where another program could process some of the fields that I didn't care about at this stage. I came up with a very hacky solution that isn't general at all. 

A solution I saw in your link to the issues page was to add a 'catch-all' tag (could look like Extra map[string]interface{} `json:"*"`) in the json package that would be applied only to a map[string]interface{} at the top-level of the struct. This way the json package could unmarshal in its normal way and after it has processed all the available tags it would look for a catch-all tag and unmarshal any remaining fields into their. That is exactly what I'm looking for. 

[Dustin]

You could try something like this: https://play.golang.org/p/qruF3obd7y

It's not a perfect soln. but I hope it meets your needs.

[jammi...@gmail.com]

Very neat solution. Thanks Dustin! 

[Matt Harden]

Why not just unmarshal into a map[string]interface{}, manipulate the map entries that you care about, and marshal that map back to JSON when done? JSON numbers will be stored in the map as float64 values and JSON strings as string values.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jammi...@gmail.com]

Thats was my initial way of doing it, but wanted to see if the json package had a general way of doing this with structs w/ a catch-all map. Dustin's solution meets my needs although the t.Main fields are still in the map after unmarshalling. In the Unmarshal method I think I can use reflection to loop over t.Main's fields getting each name and delete those map entries from t.Extra. I'm going to give it a shot and post something back here later. 

[Caleb Doxsey]

With Dustin's approach you could use a temporary map:

    func (t *MyType) MarshalJSON() ([]byte, error) {

    tmp := make(map[string]interface{})

    for k, v := range t.Extra {

    tmp[k] = v

    }

    // overwrite fields in Main into tmp

    data, err := json.Marshal(t.Main)

    if err != nil {

    return nil, err

    }

    json.Unmarshal(data, &tmp)

    return json.Marshal(&tmp)

    }

If you only needed read access you could use a json.RawMessage: https://play.golang.org/p/UG39tmmQ90

    type MyType struct {

    Main

    Extra json.RawMessage

    }

    func (t *MyType) MarshalJSON() ([]byte, error) {

    return t.Extra, nil

    }

    func (t *MyType) UnmarshalJSON(p []byte) error {

    json.Unmarshal(p, &t.Main)

    t.Extra = json.RawMessage(p)

    return nil