Generating code into module cache?

[Matt Mueller]

Hey folks, 

I'm working on a project with a lot of code generation. I'd love to:

- get the code generation out of the way when you don't care about it

- be able to vendor the generated code when you do care about it

That led me to an idea about generating code into the go module cache. Do you see any downside in doing this? 

Thanks!

Matt

[seank...@gmail.com]

That doesn't really make sense? 

The module cache is for immutable versions of modules,

ie. publish your (generated) code as a package/module, import that

it will then get cached

[Matt Mueller]

Sorry, maybe cache is the wrong word.

Wherever "go get -u ..." downloads modules, I'd like to stick my generated code. 

Conceptually you could think of it as a virtual module used only during development. If you're familiar with Node.js, You could also think of it as creating a directory in node_modules/ and then importing it with require(...).

I'm just wondering if I'll run into issues doing this or mess up other people's modules.

[Axel Wagner]

Technically, I think this would be possible. Your code wouldn't be usable together with code that doesn't use modules. And you'd need to have a server that takes the code from the git repository and generates a .zip file for any interesting version (possibly on demand), which you would probably have to build yourself. It also means there is no reliable provenance from your git repository to your published module, so reproducing/forking what you do might be more difficult. Code analysis running from the repository (like sourcegraph or githubs hover info) might not understand the code anymore.

But, yeah. In general, you can put *any* go code in an appropriate .zip-file and serve it up as a module. You don't even have to publish the repository itself at all, if you so wish.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a26c4aca-bbe3-4dc5-9f66-09d1db521517n%40googlegroups.com.

[Axel Wagner]

On a second read, I probably misunderstood what you intend to do.

I still think what you intend to do might technically be possible - I'm not entirely sure, though. The module cache is by default read-only, so you'd want a tool that force-replaces a directory. And then, the go build cache might get confused, if the contents of the cache change. I'd also predict that you regularly run into issues where things don't build the same on your machine as for other people, leading to hard to debug and confusing errors. And, lastly, I'm not sure what the advantages would be over doing the normal approach of generating the files and committing them? It seems strictly more work, with no added benefits.

[Paul Jolly]

I would personally steer clear of writing anything to the module cache.

If you are only code generating into the module cache and not
publishing this generated code, by definition that generated content
will not be available remotely. Hence any code you publish that
depends on this generated content will fail to resolve. So we are
talking local development only.

In which case replace directives with a directory target is most
likely your best solution:

https://golang.org/ref/mod#go-mod-file-replace

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfF36cBaAhuxPqTK3dbF6tja3yVAYrcNaWkMsw%3D8LR_-Pw%40mail.gmail.com.

[Matt Mueller]

Thanks for your thoughts!

> I'm not sure what the advantages would be over doing the normal approach of generating the files and committing them?

Mostly trying to improve the UX by avoiding generated code clutter during development. I concede that storing all your generated code in one directory inside your project isn't that big of a deal. Another solution could be to configure your editor to hide the generated directory. 

> So we are talking local development only.

Yah, exactly. The CLI wraps "go run" anyway, so there's a place to generate code on new machines before starting the program.

> In which case replace directives with a directory target is most likely your best solution:

Replace is an interesting idea. I think replace would work well if there was a standard place to put generated code outside of your project. I guess that's what I'm looking for: a standard place to hide away generated code until you care about it and want to vendor it. And when you do want to vendor it, you can use the same code paths as your third-party dependencies.

[Paul Jolly]

> Mostly trying to improve the UX by avoiding generated code clutter during development. I concede that storing all your generated code in one directory inside your project isn't that big of a deal. Another solution could be to configure your editor to hide the generated directory.

Either that or, assuming you have some sort of visual tree view, to
show generated files in italics or some such. Or to have generated
files not be included in fuzzy finder results. There are many
approaches.

> Yah, exactly. The CLI wraps "go run" anyway, so there's a place to generate code on new machines before starting the program.

So this is a tool you are writing for other developers? I'm further
assuming the generated code is critical to code they are then writing.
If so, this surely must be committed with whatever code they are
writing? In which case, generating this to anywhere other than the
place where they are writing code is a problem. But then I might have
totally misunderstood what you are doing here, in which case can you
explain a bit more?

> > In which case replace directives with a directory target is most likely your best solution:
>
> Replace is an interesting idea. I think replace would work well if there was a standard place to put generated code outside of your project. I guess that's what I'm looking for: a standard place to hide away generated code until you care about it and want to vendor it. And when you do want to vendor it, you can use the same code paths as your third-party dependencies.

There is nothing stopping you coming up with your own convention here,
or indeed retaining the convention of using
GOPATH/src/github.com/my/repo etc. Tools like gohack
(https://github.com/rogpeppe/gohack) do just that: establish a
convention.


Paul