Generating code with go/ast

[Carl Mastrangelo]

In a toy project I am working on, I am trying to generate Go code automatically to cut down on the amount of boiler plate I have to write.  I was looking at the packages under go/ in the standard library, but a lot of them seem specific to taking existing code and interpreting it rather than the other direction.  Specifically, a lot of the API depends on having a token.FileSet, which assumes the file already exist.  Since I am making it from scratch, I don't have a file or line number or token positions.

Is there a proper way to generate code?  Looking at golang/protobuf, it seems to print out raw strings currently too.  Other projects like protobuf would likely benefit from being able to build code using standard packages.

[Jan Mercl]

On Sat, May 7, 2016 at 7:30 PM Carl Mastrangelo <carl.mas...@gmail.com> wrote:

> Is there a proper way to generate code?

Go source code is just text, so the fmt package and common string operations is all you need. The go/format package could be used to make the result look good.

--

-j

[Carl Mastrangelo]

That is what I ended up doing.  My first attempt did use templates, but it was very difficult to get the formatting right.  For example, getting indentation and new lines in the right place does work with templates, even with the new {{- foo -}} directives.  I ended up copying protobuf's approach of using a tab writer, which makes for a more visually pleasing generated indentation, but uglier generator.  

One of the reasons I was looking at go/ast in the first place is that it is hard to tell if an import is used or not until the whole source is generated.  Writing code out with fmt has problems too, since now I have strconv.Quote all over the place.  Using "fmt" also has problems when used with bytes.Buffer as the temporary destination since you can't really "unwrite" a rune or set of bytes.  Having a full syntax tree would make these problems easier to solve.

[Caleb Spare]

I have a few different code gen projects and I generally use
text/template but don't worry about formatting/indentation, just
getting valid output. Then I run the result through go/format as Jan
suggested (which is the library equivalent of gofmt).

> One of the reasons I was looking at go/ast in the first place is that it is hard to tell if an import is used or not until the whole source is generated.

For generated code, I think it's reasonable not to worry about this.
You can just add dummy usage so the code compiles.

import (
"bufio"
"fmt"
"strconv"
)

var (
_ = bufio.Scanner
_ = fmt.Sprint
_ = strconv.Atoi
)

This isn't acceptable in hand-written code, but as r and others like
to say, we hold generated code to a lower standard. (Protobuf and lots
of other projects take this approach.)

-Caleb

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[jimmy frasche]

Using templates is the easiest way to generate code. go/ast isn't bad
for analyzing go files, but it's hard to generate code from an AST or
even to transform an existing AST (at least while preserving
comments). Even if you manage to pull it off the template is going to
be much shorter and easier to debug and read.

[Nigel Tao]

On Sun, May 8, 2016 at 5:20 AM, Carl Mastrangelo
<carl.mas...@gmail.com> wrote:
> I ended up copying protobuf's approach
> of using a tab writer, which makes for a more visually pleasing generated
> indentation, but uglier generator.

I'm not sure which protobuf you're referring to, but I wouldn't use a
tabwriter. Instead, as Jan suggested, just let go/format auto-format
your generated source code. For example,
https://go.googlesource.com/image/+/master/font/basicfont/gen.go
generates
https://go.googlesource.com/image/+/master/font/basicfont/data.go

> One of the reasons I was looking at go/ast in the first place is that it is
> hard to tell if an import is used or not until the whole source is
> generated. Writing code out with fmt has problems too, since now I have
> strconv.Quote all over the place.

I'm not sure what you're trying to do exactly, but you might not need
strconv.Quote all over the place if you use the %q verb instead:
https://play.golang.org/p/7b_pZI2rL1

> Using "fmt" also has problems when used
> with bytes.Buffer as the temporary destination since you can't really
> "unwrite" a rune or set of bytes.

Can you give an example where you need to unwrite a rune?

[Mihai B]

templates is the only sane way(AFAIK) to go right now .  I've created a library for the  most commonly used constructs (e.g. package declaration,  type declaration, struct literal declaration & assignment and some forms of function calls) to keep the code generation manageable and avoid redundancy but it is far from being a robust solution because the lib was created as a workaround rather than a holistic approach based on the language specifications. Initially I used go/ast as the IR which worked fine for simple type declarations but it proved quite challenging on more complex constructs so I've moved to a mix of templates (to write the source code) and an internal representation/types. 

   Recently I've moved many parts of the "generated" code into a reflect based solution due the high cost of maintenance of the code generation tools(i.e. my library) so I would suggest you to consider a reflect based solution mixed with code generation(strictly only where reflect can't help) and if the performance is an issue move from reflect only when the API is stable. This will save you many weeks/months of work if the project is large.

Mihai.