Coercion of JSON null to magic value per golang type

[Jon Cooper]

I'm parsing some rather wide JSON objects (~300 fields) which contains many attributes that derive from nullable SQL numeric types.

Each of these attributes may take on values in the range [0, <max>] or null.

I do not control the input JSON.

What I would like to do is parse these into struct fields of primitive type, coercing JSON null to a magic value per golang type.

For example, given a struct like:

type Record struct {

        A int

        B float64

}

Parsing JSON like:

{"A": null, "B": null}

Should yield:

{-1, -1.0}

Options that I've considered:

* typedef each numeric type to my own type, use custom UnmarshalJSON to do coercion, have to cast all over the place later

* json.Decoder with UseNumber() set, decode into map[string]interface{}, manually type-assert and copy each field into instance of struct

* several increasingly janky riffs using reflection

I'd really appreciate any ideas. Have been unable to come up with an approach that isn't fug. Perhaps the low-hanging fruit is to modify the encoding/json package, but it would be nice to find an idiomatic-ish approach.

Best,

Jon

[Nick Craig-Wood]

On 01/03/14 01:10, Jon Cooper wrote:
> I'm parsing some rather wide JSON objects (~300 fields) which contains
> many attributes that derive from nullable SQL numeric types.
>
> Each of these attributes may take on values in the range [0, <max>] or null.
>
> I do not control the input JSON.
>
> What I would like to do is parse these into struct fields of primitive
> type, coercing JSON null to a magic value per golang type.
>
> For example, given a struct like:
>
> type Record struct {
> A int
> B float64
> }
> Parsing JSON like:
>
> {"A": null, "B": null}
>
> Should yield:
>
> {-1, -1.0}
>
> Options that I've considered:
>
> * typedef each numeric type to my own type, use custom UnmarshalJSON to
> do coercion, have to cast all over the place later
> * json.Decoder with UseNumber() set, decode into map[string]interface{},
> manually type-assert and copy each field into instance of struct

Instead of manually copying into each field you could update the value
in the map[string]interface{}, encode it back into json, then decode it
into your struct which might be slow, but is relatively straight forward.

> * several increasingly janky riffs using reflection

Did you consider using

type Record struct {
A *int
B *float64
}

Then you can find the nil values and update them afterwards?

> I'd really appreciate any ideas. Have been unable to come up with an
> approach that isn't fug. Perhaps the low-hanging fruit is to modify the
> encoding/json package, but it would be nice to find an idiomatic-ish
> approach.

There was a discussion here quite recently about setting default values
for json decoding. Unfortunately my search foo is too low to find it at
the moment!

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick

[egon]

* Idiomatic way would be to use pointers to the integers.

* If you have only ints/strings etc. simple values then marshaling into a map[string]interface{} is a good approach as well. http://blog.golang.org/json-and-go

* If that style causes problems... (e.g. memory allocation) then the custom Unmarshaler/Marshaler would be the next suggestion.

* If casting back-and-forth is very inconvenient (too much places) then there are two options make your own json package or pre-process the json.

* Preprocessing adds another step to your pipeline.

* Maintaining your own json package could become cumbersome and you might some optimizations that the go package get. (the place where null value decoding is done is: http://golang.org/src/pkg/encoding/json/decode.go?s=16863:17037#L662)

In that order, see which one works out for you... 

+ egon

Best,

Jon

[Jon Cooper]

Thanks, Nick and egon.

I don't want nil in my struct, so using pointers is out. The whole goal here is to avoid having to null-check elsewhere in the program.

What I would really like, I believe, is a tag that lets me use a custom Marshaler/Unmarshaler for the struct field corresponding to a JSON key without having to change that field's type.

In the short run I suppose that I'll use a typedef to apply the custom Marshaler/Unmarshaler to fields which need coercion. But it feels philosophically wrong; the fact that an int field came from a coerced nullable JSON number doesn't make it not an int field for my code. I don't need to track the provenance of that int and in fact explicitly wish not to.

(As an aside - http://godoc.org/github.com/mitchellh/mapstructure#DecoderConfig has some interesting ideas although I would prefer to configure via tags rather than a struct.)

[Carlos Castillo]

If a json null is parsed, the field in the struct is not updated, so any value it had before the parse is untouched. For newly allocated struct values you are looking at the zero value, but you don't have to keep it that way before unmarshalling: http://play.golang.org/p/7320M1gxng

[egon]

On Monday, March 3, 2014 8:18:38 PM UTC+2, Jon Cooper wrote:

Thanks, Nick and egon.

I don't want nil in my struct, so using pointers is out. The whole goal here is to avoid having to null-check elsewhere in the program.

Instead of checking against -1? Which for me could be more problematic, because it's easy to forget that and get subtle bugs... e.g.

type Payment { V float64 }

total := 0.0

payments := []Payment{Payment{-1}}

for _, v := range payments {

   total += v

}

But, I guess it depends what the struct actually is and how exactly it's used... I guess you have your reasons.

[Jon Cooper]

If a json null is parsed, the field in the struct is not updated, so any value it had before the parse is untouched. For newly allocated struct values you are looking at the zero value, but you don't have to keep it that way before unmarshalling: http://play.golang.org/p/7320M1gxng

This is a reasonable compromise. I have been resistant to the notion of pre-filling the nullable fields with a default value because there are ~300 such fields and the giant initializer doesn't look very pretty. But for smaller structs it seems this would clearly be the way to go. I've opted to just go with the typedef approach.

Thanks for the input everybody.

[Matthew Zimmerman]

On Mon, Mar 3, 2014 at 2:34 PM, Jon Cooper <jon.c...@gmail.com> wrote:
> I have been resistant to the notion of pre-filling the nullable fields with
> a default value because there are ~300 such fields and the giant

I imagine wherever those default values exist will be giant since
there's 300 of them.

If you really don't want it in your face, stick it in elsewhere like
in constants.go and have a NewDefaultObject() Object function create
it and set defaults.