proposal: go generate

[Rob Pike]

New go tool subcommand proposed for Go 1.4. Please see the design
document and comment in this thread.

http://golang.org/s/go1.4-generate

-rob

[Rob Pike]

Either works, but they mean different things, consistent with existing
go subcommands.

-rob


On Tue, Jul 1, 2014 at 1:22 PM, jimmy frasche <soapbo...@gmail.com> wrote:
> Why is it
>
> go generate -run yacc all
>
> instead of
>
> go generate -run yacc ./...
>
> or the like, for consistency with the rest of the go subcommands?

[Ingo Oeser]

Love it!

But having a way to (only) print the exact commands that will be run after all substitutions are done would be awesome.

Is this intended to be covered by -x and -n flags?

[Rob Pike]

Yes, that's what -x and -n are for.

-rob

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[bsiegert]

The design sounds great. Just a few points:

Will the ... syntax for subdirectories be supported? That is, can I run the regenerator scripts in all subdirectories by running "go generate ..."?

How about respecting build tags in the file? If the file containing the generate tag is, say, Windows specific, skip execution when run from Linux.

There should be a way to provide a working directory for the command, as you note in your proto example.

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+unsubscribe@googlegroups.com.

[jimmy frasche]

[Rob Pike]

Yes, package syntax and semantics and the handling of build tags will
work exactly as they do for other go subcommands.

The need for a working directory is easy handled by a cd in a command directive.

-rob

>> email to golang-dev+...@googlegroups.com.

[Michael Hudson-Doyle]

Rob Pike <r...@golang.org> writes:

> Yes, package syntax and semantics and the handling of build tags will
> work exactly as they do for other go subcommands.
>
> The need for a working directory is easy handled by a cd in a command directive.

So something like "//go:generate cd ../scratch; some-generator arg"? I
was thinking on reading the document that it's not really clear if

//go:generate thing1 arg1; thing2 arg2

will invoke both thing1 and thing2 or does it invoke thing1 with three
arguments?

Cheers,
mwh

[Rob Pike]

No, that won't work because the generator syntax does not parse shell
metacharacters (like semicolons), and shouldn't.

Instead you would define a tool that does the cd for you and runs
protoc from there.

-rob

[andrewc...@gmail.com]

It would be interesting if there was a standard tool for generating code from a stamdard library of datastructures/algorithms for specific types. A simpler solution than generics anyway.

[Kyle Lemons]

I am continually impressed by the simplicity of the solutions you guys propose.  +1 for this.

My only suggestion, I think, is to use "// +generate" instead of "//go:generate" or, at a minimum, allow "// go:generate".  It seems like "// +generate" would be an obvious contender though, so there's probably a reason you opted not to use it.

-rob

[Stephen Gutekanst]

This is beautiful! Simplistic and powerful. I love the idea for generating custom sort types as well as the idea for embedding binary data.

I could imagine another use-case for this as well: SWIG.

• SWIG would not have to be installed by anyone downloading a package.
• SWIG API's could be viewed in godoc, etc.

Of course, this would mean SWIG would have to generate code which can be compiled on every platform, but AFAIK it is already pretty close to that.

Stephen

[Russ Cox]

We started out a bit ad hoc but we have been standardizing on //go: for comments with special meaning to the go toolchain.

[minux]

On Tue, Jul 1, 2014 at 8:22 PM, Stephen Gutekanst <stephen....@gmail.com> wrote:

This is beautiful! Simplistic and powerful. I love the idea for generating custom sort types as well as the idea for embedding binary data.

I could imagine another use-case for this as well: SWIG.

• SWIG would not have to be installed by anyone downloading a package.
• SWIG API's could be viewed in godoc, etc.

Of course, this would mean SWIG would have to generate code which can be compiled on every platform, but AFAIK it is already pretty close to that.

I think both cgo and SWIG are still platform dependent.

That said, using go generate for cgo/SWIG so that people do not need gcc/swig to install a cgo/swig package is

still pretty cool.

This should also enable the various Go static analyzers to function correctly on cgo/swig packages.

and also help reducing cgo/swig package build time.

[andrewc...@gmail.com]

So if the generated source needs to be different per arch or OS for whatever reason, is the correct way to have one file for each with the appropriate build flags in each file?

[Gustavo Niemeyer]

Can we standardize around the existing convention instead, or at least
support go:build and obsolete +build so we can eventually converge?

Another unrelated detail that caught my attention is that

//go:generate command

and

//go:generate yacc

Look exactly the same, although they mean very different things. Also,
this convention prevents future extensibility as any other name might
conflict with words in use.

So, what about standardizing around flags for these comments, and using

//go:generate -command

instead? Or perhaps -define?

On Tue, Jul 1, 2014 at 9:22 PM, Russ Cox <r...@golang.org> wrote:
> We started out a bit ad hoc but we have been standardizing on //go: for
> comments with special meaning to the go toolchain.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[Sameer Ajmani]

Regarding the run flag:

The -run flag takes a regular expression, analogous to that of the go test subcommand, that restricts generation to those directives whose command (see below) matches the regular expression.

If one has a file with several generate comments:

//go:generate protoc foo.proto

//go:generate protoc bar.proto

//go:generate protoc baz.proto

The -run constrains you to running all the protoc commands or none of them.  If the user wants finer control, they would need to split those generate comments across multiple files (or use a different command name for each).

Would it make sense to match -run against the command's args as well?  That would allow -run "protoc baz" or simply -run baz (which is interesting when there are multiple "baz" generators).

S

-rob

[Michael Hudson-Doyle]

Rob Pike <r...@golang.org> writes:

> No, that won't work because the generator syntax does not parse shell
> metacharacters (like semicolons), and shouldn't.

Makes sense. I think it was only the mention of environment variables
that made me think it might....

Cheers,
mwh

[Daniel Theophanes]

+1 for also converging, even if +build stays around, I'd rather have a single "go:directive" syntax that can be used.

-Daniel

[Kevin Gillette]

Regarding Kyle's comment: that was the first thought I had as well (use of "// +generate" instead of "//go:generate". So far, the "//go:" comments I've seen have been internal or expert-use, such as "//go:inline", and may become noops as the compiler matures. In contrast, "// +whatever" felt like a much warmer convention suitable for all gophers to use. Anyway, I'm not trying to argue the larger point -- if +build is the exception to the [new] rule, so be it.

[Rob Pike]

I originally meant to have a minus sign on there but dropped it for no
good reason. It should probably be there.

-rob

[Rob Pike]

I thought about that, but I believe it introduces more problems than
it solves. It shouldn't be an issue to rerun all the proto files at
once. If it is, something else needs rethinking.

-rob

[Rob Pike]

If we're really arguing about the exact way to spell the generator
directive's introductory phrase, I think we've reached the end of the
discussion.

As Russ said, '"// +build" should have been "//go:build" but we
weren't thinking far enough ahead. On the other hand, changing it now
is certainly not worthwhile.

-rob

[Carlos Castillo]

Would it make sense to have a "-package" option that would specify a package path that generates the tool (if it's written in go). Benefits:

1. Only the command generated by the package in question would be allowed to process the code, and not an unfortunately named alternative program
   
2. A useful error message when the package/command isn't found (eg: please run "go get github.com/jteeuwen/go-bindata" to install the the go-bindata tool)
   
3. It would encourage the use of tools written in go
4. 2+3 would help others on platforms different from the original dev, as the tools would be portable both to run and install
   

I'm not suggesting that the tool be fetched or built automatically, but just that to get off the ground, "go generate" could help the user install the appropriate tool, and ensure that no obvious mistakes with poorly named tools or forks happen.

On Tuesday, July 1, 2014 12:57:43 PM UTC-7, Rob Pike wrote:

[Jan Mercl]

I like the simple and clean design. However:

"It is hoped, however, that it may replace many existing uses
of make(1) in the Go repo at least."

I concur that this mechanism may also replace Makefiles of my own. The
question is: Why? If this tooling is aimed at authors only (no
generators are run for go {build,get,install} and the results are the
same as when the (old) Makefile is run - what's the gain for the
effort of rewriting the Makefile into the annotations? I cannot find
the answer in the design document.

-j

[yiyus]

Something I would find quite natural and is not discussed in the proposal is to include the generation comment in the generated file.

In the first example, if I go get the package my/own/gopher and the file gopher.go does not make any reference to the comment in main.go, I would not even think I have to look for it (or for gopher.y). If the comment was at the top of gopher.go, I would instantly know how it has been generated. Same happens with the strmeth example, I find the top of the day_string.go file a better place for the go:generate comment than somewhere inside main.go.

I think this could be done with the proposal as it is, it is just a matter of convention (which would have to be followed by generators, to add the top comment). For generators that write their output to stdout, it could be useful to have a (trivial to write) helper program that reads the comment, executes the command, and concatenates the comment with the output of the command. It could also be interesting that go generate did this with go:generate comments at the top of the file.

Has something along these lines been considered?

[Egon Elbre]

On Tuesday, 1 July 2014 22:57:43 UTC+3, Rob Pike wrote:

New go tool subcommand proposed for Go 1.4. Please see the design
document and comment in this thread.

http://golang.org/s/go1.4-generate

-rob

I expect that one common usage would be to run a local "maketables.go". I assume something like this would work:

// -- maketables.go --

//+build ignore

//go:generate go run maketables.go -o tables.go

package main

...

So, you would have a "maketables.go" file that contains the directive to run itself.

+ egon

[roger peppe]

I like the proposal.

One possibility that occurs to me is that people
might want to write a generator that generates whole
packages, rather than just go files within a package.

This could be particularly useful in the light of
the internal packages proposal. It also allows
programs to keep generated code strictly separate
from user-written code.

Has this been considered? It could be restricted to
creation of packages below the current package
directory.

[Han-Wen Nienhuys]

I like it. It could also be a very unconventional but practical answer
to "Go doesn't support generic programming"

On Tue, Jul 1, 2014 at 9:57 PM, Rob Pike <r...@golang.org> wrote:
> New go tool subcommand proposed for Go 1.4. Please see the design
> document and comment in this thread.
>
> http://golang.org/s/go1.4-generate
>
> -rob
>

> --
> You received this message because you are subscribed to the Google Groups "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Han-Wen Nienhuys
Google Munich
han...@google.com

[yiyus]

On Wednesday, 2 July 2014 10:00:36 UTC+2, Jan Mercl wrote:

what's the gain for the
effort of rewriting the Makefile into the annotations? I cannot find
the answer in the design document.

An advantage (there may be more) is that go generate will take care of regenerating everything if asked, possibly in different packages, without having to invoke make manually.

If avoiding make supposes an extra effort, I think the most obvious solution would be to just add a comment:

// go:generate make

so that make is run when you run go generate. When the Makefile is trivial, you can avoid it, but else you do not have to.

[tomwilde]

Can we standardize around the existing convention instead, or at least
support go:build and obsolete +build so we can eventually converge?

I'd go one step further and propose the introduction of some kind of formal structure for all the semantics we're pushing into (import) strings and comments in the various distinct recent proposals.

[Sameer Ajmani]

Agreed.  And the user can always create whatever groupings they want by putting generate directives that should run together in the same go source file.

[Steve McCoy]

If your generators or generated code have some platform-specific aspect, you can have go:generate tags in platform-specific files (e.g. frob_win.go and frob_linux.go), saving you from the associated makefile choreography. But I don't see that as a particularly big benefit, because most of the motivating examples (and most other examples I can think of) are platform-agnostic.

[fge...@gmail.com]

On Wed, Jul 2, 2014 at 6:58 AM, Rob Pike <r...@golang.org> wrote:
...

> As Russ said, '"// +build" should have been "//go:build" but we
> weren't thinking far enough ahead. On the other hand, changing it now
> is certainly not worthwhile.
>
> -rob

[I understand this is only marginally relevant to the proposal.]
Most probably I am missing something, although wouldn't this work?
1. rewrite pkg/go/build/* to understand //go:build
2. create a tool to automatically rewrite files
If i understand correctly neither gofix nor gofmt would help here,
though with an utf-8 aware awk something like this might be good
enough:
awk '{if ($0 ~ /^\/\/ \+build/) $1="//go:build"; $2=""; print $0}'

3. run the tool on the standard library
4. ask everybody to do the same with their sources after 1.4

What did I miss or which part of the process is not worthwhile?

1.3 source grep show this:
$ # .go files:
; grep '^// +build' `du -a |grep '\.go$'|awk '{print $2}'` |wc
213 1104 13132
; # all the rest:
; grep '^// +build' `du -a |grep -v '\.go$'|awk '{print $2}'` 2> /dev/null| wc
79 320 3634

[Russ Cox]

Please stop this tangent on "// +build" vs "//go:build". It is not going to change.

Changing the spelling would incur one or more of these costs: (1) there would be two ways to write exactly the same thing, (2) people would have to update their code, (3) using the new way would make a program not compatible with older versions of Go, (4) other tools looking for the old way would have to be updated.

Go is intended for programming at scale, with large code bases. These are real costs in large code bases. If at all possible, the rule is to avoid incurring them.

Occasionally there is an important justification for breaking the rule. For example, Go 1.3 introduced some effective restrictions on unsafe.Pointer (mainly, they have to hold pointers) that might trigger cost (2). However, making those restrictions do not incur any of the other costs, and they make possible the significant benefit of being able to grow the stack by moving it, which has much better performance characteristics than the old stack growth implementation.

Changing the spelling of " +" to "go:" has real costs and no real benefit. It is not going to change. Let's keep this thread about 'go generate' and not about other comments. If you really must continue this conversation, start a new thread with a different subject.

Thanks.

Russ

[Rob Pike]

That puts the focus on the client running the generator, which is
against the purpose of this proposal. A comment should be sufficient
to stimulate the memory of the package author. No feature is required.

-rob

[Rob Pike]

1) Make isn't always available, especially on Windows.
2) Make is disliked by many programmers.
3) The generate command can use features of the go command not
available to Make, such as ... as a package path.

-rob

[Rob Pike]

The generators are computer programs. They are capable of anything
computer programs can do.

The point of this proposal is to encourage experimentation. Experiment!

-rob

[mattn]

The generators will run in each times for go build?

Overwrite the file? Or generate another files? The file that contains generator pragma has suffix .go ?

I sugguest to give another file extension. ex: .gx , .gox, etc.

- mattn