Deterministic json.Marshal result

[Jérôme LAFORGE]

Hello,
Can I consider that output of json.Marshal is deterministic for the same struct with the identical values?

For example :
http://play.golang.org/p/Sb1rTfGkFj

In the above example, can I consider that result will always the same into result string (with the same sha1sum for example)?
There is no inversion of field into json result?

Thx in advance.
Regard
Jérôme

[Konstantin Khomoutov]

Sorry for asking, but why would you need that?

The JSON standard says nothing about explicit ordering of fields in the
representations of objects because, naturally, those fields refer to
*named* fields in the source/destination objects, so a JSON object has
essentially the same semantics as a hash map which binds keys to values
but does not impose any ordering. To cite the json.org's title page:

| JSON is built on two structures:
|
| * A collection of name/value pairs. In various languages, this is
| realized as an object, record, struct, dictionary, hash table, keyed
| list, or associative array.
| * An ordered list of values. In most
| languages, this is realized as an array, vector, list, or sequence.
|
| These are universal data structures. Virtually all modern
| programming languages support them in one form or another. It makes
| sense that a data format that is interchangeable with programming
| languages also be based on these structures.
|
| In JSON, they take on these forms:
|
| An object is an unordered set of name/value pairs. An object begins
| with { (left brace) and ends with } (right brace). Each name is
| followed by : (colon) and the name/value pairs are separated by ,
| (comma).
| An array is an ordered collection of values. An array begins with
| [ (left bracket) and ends with ] (right bracket). Values are
| separated by , (comma).
| ...

Notice the word "unordered" used when describing objects.

I hence think that making any guarantees about how encoding/json works
beyond those specified in the JSON spec would be stange.

[Jérôme LAFORGE]

Thx for your reply.

Sorry for asking, but why would you need that?

I use gokit, and I write Publisher (https://github.com/go-kit/kit/blob/master/loadbalancer/publisher.go) for netflix/eureka (with the client https://github.com/hudl/fargo) for service discovery.
I also use EndpointCache (https://github.com/go-kit/kit/blob/master/loadbalancer/endpoint_cache.go) for managing the cache of the endpoints.

This EndpointCache uses map[string]endpointCloser (https://github.com/go-kit/kit/blob/master/loadbalancer/endpoint_cache.go#L24) for caching all endpoints.
For updating this cache, I use Replace(instances []string) (https://github.com/go-kit/kit/blob/master/loadbalancer/endpoint_cache.go#L47) that takes []string for each instance of endpoint.
And this instance's string have to be equal as it use as key of the map (https://github.com/go-kit/kit/blob/master/loadbalancer/endpoint_cache.go#L55)

Anyway, I write my own marshaller, and use the standard json unmarshaller.

Thx again.

[Matt Harden]

Maybe https://github.com/tent/canonical-json-go

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Harden]

"github.com/ugorji/go/codec" also has support for canonical representations: https://godoc.org/github.com/ugorji/go/codec#EncodeOptions

[Konstantin Khomoutov]

On Fri, 13 Nov 2015 04:18:06 +0000
Matt Harden <matt....@gmail.com> wrote:

> "github.com/ugorji/go/codec" also has support for canonical
> representations:
> https://godoc.org/github.com/ugorji/go/codec#EncodeOptions

This one appears to only canonicalize maps.

[...]
> > Maybe https://github.com/tent/canonical-json-go

From cursory look, this one appears to do the right thing for objects
as well.

Interesting projects; thanks for sharing these pointers!

[...]

[Matt Harden]

Did you mean "structs" rather than objects?

[Konstantin Khomoutov]

On Fri, 13 Nov 2015 17:38:44 +0000
Matt Harden <matt....@gmail.com> wrote:

> Did you mean "structs" rather than objects?

Yes, I meant structs. Sorry for confusion.

[Ugorji Nwoke]

go-codec canonicalizes all values.

Maps are the only thing that can cause differences, due to its random iteration order, so I spelt that out in the docs.

Try go-codec. You will love it (shameless plug).

[Ben Kehoe]

While structs have an ordering defined for their fields, I'd say for a canonical representation, since both structs and maps can represent JSON objects, a struct and a map representing equivalent JSON objects should encode to the same JSON text (i.e., the struct fields should be sorted too). This means that, give text in canonical format, encode(decode(text)) would return the same bytes regardless of whether a struct or map was used in the decoding.

[Ugorji Nwoke]

And that is exactly what happens.

Encoding always uses the natural consistent ordering (iteration for arrays, slices, fields of a struct). However, there is no consistent ordering for maps, so that's where we have to do work, to sort keys first and then encode the corresponding key value pairs.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/e-y8vQS1Yj8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Konstantin Khomoutov]

On Mon, 16 Nov 2015 07:37:35 -0500
Ugorji Nwoke <ugo...@gmail.com> wrote:

> > While structs have an ordering defined for their fields, I'd say
> > for a canonical representation, since both structs and maps can
> > represent JSON objects, a struct and a map representing equivalent
> > JSON objects should encode to the same JSON text (i.e., the struct
> > fields should be sorted too). This means that, give text in
> > canonical format, encode(decode(text)) would return the same bytes
> > regardless of whether a struct or map was used in the decoding.
>

> And that is exactly what happens.
>
> Encoding always uses the natural consistent ordering (iteration for
> arrays, slices, fields of a struct). However, there is no consistent
> ordering for maps, so that's where we have to do work, to sort keys
> first and then encode the corresponding key value pairs.

Does your library sorts the struct fields by names before serializing?
I mean, if I swap places of two fields of a struct type, objects of
this type should continue to be encoded as before.

Sorry, I did not read the source code; just want to make things clear.

[Ben Kehoe]

I just did a quick check on this. I assumed "natural order" for structs meant you used the ordering from reflect (as encoding/json uses). It might help clear up the confusing to state in the docs that the struct fields are sorted (even, apparently, when canonical is set to false)

[Konstantin Khomoutov]

On Mon, 16 Nov 2015 05:07:28 -0800 (PST)
Ben Kehoe <bke...@irobot.com> wrote:

> I just did a quick check on this. I assumed "natural order" for
> structs meant you used the ordering from reflect (as encoding/json
> uses). It might help clear up the confusing to state in the docs that
> the struct fields are sorted (even, apparently, when canonical is set
> to false)

I'd say that since JSON explicitly defines "objects" (those { ... }
things) as being unordered -- as opposed to arrays, -- canonical
encoders *must* sort the contents of stuct-typed values and maps
by their "keys", which, for structs, are their fields.

The logic behind this is quite simple:

1) Given a struct type

type T struct {
A int
B int
}

the code

var t T
t.A = 1
t.B = 2

and the code

var t T
t.B = 2
t.A = 1

yield exactly the same two values.
That is, the ordering of assignments of the fields does not matter.

2) For a map type

type T map[string]int

the code

var t T
t["A"] = 1
t["B"] = 2

and the code

var t T
t["B"] = 2
t["A"] = 1

produce the two semantically equivalent maps.
That is, again, the ordering of assignments does not matter.

[Ben Kehoe]

I agree—and go-codec does, in fact, do this. When encoding, it sorts the fields of a struct by name whether or not it has been directed to produce canonical output. It only sorts map keys, however, when canonical is set to true. I think the go-codec documentation could be more clear that "natural consistent ordering" for struct fields means sorted by name, rather than the ordering defined by the reflect package.

[Matt Harden]

It should probably be stated explicitly that the sorting is by the binary [UTF-8] encoding of the strings, as opposed to some other sequence based on normalization, language, etc. This is described in the language spec as "lexically byte-wise".

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.