Proposal: Add tool for generating Go struct definitions from JSON

[Aditya Mukerjee]

Gophers,

Context: Go 1.4 adds the `go generate` command, which facilitates (limited) code generation to solve certain problems when developing in Go, such as parser generation (goyacc).

Problem: Go, like most statically-typed languages, requires developers to explicitly list fields and structures when unmarshaling JSON. In Go, this takes the form of enumerating all the fields in a struct definition, and potentially adding JSON struct tags for good measures. However, much JSON "in the wild" is effectively strongly typed, which means that these struct definitions can easily be generated automatically from a single sample input.

Approach: A few years ago, I wrote gojson, a tool to automate this process, and I've been thrilled to see that so many other Gophers have found it a useful tool in their toolbox as well. In December, `go generate` was shipped with Go 1.4, which has made it even more seamless to generate Go structs than I had originally foreseen.

Proposal: The gojson tool is analogous to goyacc. Both generate working Go code which should be checked into source control, and are usually invoked from `go generate`. goyacc is already one of the tools distributed with Go. Given the prevalence of JSON (and especially Brad's recent proposal for dependency/vendoring in Go, which involves JSON configuration) it may make sense to consider shipping a tool for facilitating the use of JSON in Go.

Example:

$ curl -s https://api.github.com/repos/chimeracoder/gojson | wc -l
89

Rather than writing out 89 fields manually, we can do

$ curl -s https://api.github.com/repos/chimeracoder/gojson | gojson -name=Repository

type Repository struct {
    ArchiveURL       string      `json:"archive_url"`
    AssigneesURL     string      `json:"assignees_url"`
    BlobsURL         string      `json:"blobs_url"`
    ...
}

(fields omitted for brevity)


Which hooks in nicely to go generate:

//go:generate gojson -o repository.go -name "Repository" -pkg "github" -input json/repository.json

In many cases, the "json/repository.json" file is already present as it is also used by tests and/or examples, making this even more convenient for the developer.



Aditya

[Rob Pike]

Didn't Francesc already describe such a tool at a conference this year?

-rob

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aditya Mukerjee]

Not sure; I know I spoke with a number of people about it at GopherCon last summer, and I wouldn't be surprised if either the idea made its way around or someone else stumbled across the same use case independently.

Though if you're referring to jsonenums, that tool is related, but serves a different purpose. As I understand it, jsonenums is specifically for enum types. The gojson tool is more like goyacc, in that it takes a parsing specification (to use the term loosely) as input and generates Go code based on the spec. This is very handy when writing either end (server/client) of a REST service. In gojson, the example JSON input fills the role that parser.y fills in goyacc.

Aditya

[Josh Bleecher Snyder]

> Not sure; I know I spoke with a number of people about it at GopherCon last
> summer, and I wouldn't be surprised if either the idea made its way around
> or someone else stumbled across the same use case independently.

There are a few implementations of this floating around.

https://github.com/ChimeraCoder/gojson
https://github.com/bashtian/jsonutils
https://github.com/tmc/json-to-struct/

I'm sure that there are others. I recall seeing them announced on
golang-nuts, and I didn't search very hard to build this list.

There are also websites that do this:

https://mervine.net/json2struct
http://mholt.github.io/json-to-go/

The question is why this would need to be part of the Go distribution
instead of existing as it already does as a very useful third party
tool. The fact that JSON is popular doesn't seem like a sufficient
reason. Note that goyacc is used to generate code for the compiler
itself.

-josh

[Aditya Mukerjee]

Well, at least three of those links are based on gojson. :) Travis and Josh have actually both contributed to gojson.

My reasoning for proposing this now is not simply that JSON is popular (XML is - regrettably - still widely used!). JSON is being used increasingly for configuration within Go applications already, and was also just included in the recent proposal for dependency and vendoring within Go. If JSON is being used as a first-class data interchange format both within the Go community and within the Go toolchain, it makes sense for the toolchain to facilitate the use of JSON within Go, doesn't it?


Aditya

[dele...@gmail.com]

The talk by Francesc "JSON, interfaces and go generate" covers marshaling/unmarshaling when the Go type is known.
I think it does not talk about generating a struct definition from a sample JSON.

[Russ Cox]

Hi. I like gojson a lot (I tweeted to advertise it last June) but I don't think that means it should go into the standard repos or into the Go distribution. In general we want to build up an ecosystem where people exchange and install tools outside those places, because that scales much better. That's the main reason for 'go get'. Even stringer is not in the standard distribution, for example. If you are planning to abandon gojson and looking for a new home for it, we should advertise to see if anyone wants to officially take it over (again, probably not in a golang-managed repo), but otherwise I think you should keep doing what you're doing.

Probably there should be a wiki page for Go code generators that links to things like stringer, gojson, and probably others that exist that I don't know about.

Russ

[Hein Meling]

I like Russ's idea. What about making a curated list of quality packages/tools that are recommended by some quorum of reasonable people. I would also love it if the go tool had a command "getable" or something to display the list of those recommended packages. I'm not suggesting to make the go tool a fully fledged package manager...

Hein :)

[Jeremy Jackins]

On Thu, 9 Apr 2015 at 09:48 Hein Meling <hein....@gmail.com> wrote:

I like Russ's idea. What about making a curated list of quality packages/tools that are recommended by some quorum of reasonable people. I would also love it if the go tool had a command "getable" or something to display the list of those recommended packages. I'm not suggesting to make the go tool a fully fledged package manager...

You seem to be suggesting that we add some sort of class stratification to separate officially blessed packages from the plebs. That hardly seems fair to high quality packages that simply haven't been discovered or aren't deemed interesting by such a list-keeper.

[Russ Cox]

Let's start with a wiki page.

[Aditya Mukerjee]

Russ,

I am definitely not looking at abandoning gojson; I merely felt that it would be beneficial to Go programmers to facilitate the use of JSON in Go. But you may be right, in that this could be a discoverability issue for third-party tools in general, rather than a matter of the tooling in the standard repo. I had considered this analogous to go yacc, but your point about stringer is well noted.

In that case, let's create a wiki page, then, and go from there.

Aditya