How to setup your development environment using Go modules

[Victor]

Hello everyone. I'm looking for best practices on how to setup your development environment using Go modules.

Let assume a situation when you work on a project where you write a go package (call it Pkg) and an application (App) that uses that package Pkg and keep writing code in both. Using GOPATH, the process is pretty straightforward: you create your workspace, then you open $GOPATH/src/App project in your IDE and just continue editing .go files in both folders: $GOPATH/src/App and $GOPATH/src/Pkg where Pkg can be in a form "github.com/author/pkg" and that way in your App you can import it by the same "github.com/author/pkg" name and everything works since when you build App, it looks for that package in $GOPATH/src/github.com/author/pkg folder and always uses your latest changes. And since you have direct access to that folder in your IDE, you can continue editing Pkg while at the same time editing App.

Now assume a situation where I want to get rid of GOPATH completely and start using modules. That way I'd put my App and Pkg folders in ~/Projects/App and ~/Projects/Pkg (since I don't have to put them in GOPATH any more). But then once App starts using go.mod file (assuming GO111MODULE is auto), it will be driven by "github.com/author/package" and store it in a local cache and not be using ~/Projects/Pkg where I make changes at the same time while working on ~/Projects/App.

What would be the best practice to setup your development environment using Go modules?

[Seth Hoenig]

You'll want to add a replace directive  to the go.mod file to reference the copy of your library Pkg on disk, e.g. 

replace github.com/author/package => ../<wherever>/package 

[Victor]

Seth, thanks! It looks great but what about production vs development builds? I assume I can add that line to my go.mod when I develop it on my Mac, but can I reuse the same go.mod file when I do a production build ? Certainly during a production build I would like NOT to use "replace". Or should I keep two go.mod files?

[Dan Kortschak]

While this seems to be the correct way to do it, it also seems like a
very obvious point of entry for human error.

We have tooling to prevent incorrect commits of files using gitignore
and hgingnore and yet people still accidentally commit files that
should not end up in repositories. Now the situation exists where there
is a file that is intended to be committed to the repository, but
during development may need to have specific lines of it altered for
development builds.

The absence of good documentation for the modules workflow and apparent
warts like this make the change to modules an unwelcoming experience
for Go developers.

[Wojciech S. Czarnecki]

On Sat, 01 Sep 2018 06:57:21 +0930
Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:

> While this seems to be the correct way to do it, it also seems like a
> very obvious point of entry for human error.

Looks like we need `go.mod.local`
ASAP.

In many workplaces a mishap leak of internal paths to the outside, even with
opensourced stuff can end someone's career. With current go.mod one would
need to **rewrite** go.mod on CI commit.

> We have tooling to prevent incorrect commits of files using gitignore
> and hgingnore and yet people still accidentally commit files that
> should not end up in repositories. Now the situation exists where there
> is a file that is intended to be committed to the repository, but
> during development may need to have specific lines of it altered for
> development builds.
>
> The absence of good documentation for the modules workflow and apparent
> warts like this make the change to modules an unwelcoming experience
> for Go developers.

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

[Shulhan]

I am also stumbled on this issues, thanks for opening a topic on this.

>
> You'll want to add a replace directive to the go.mod file to
> reference the copy of your library Pkg on disk, e.g.
>
> replace github.com/author/package => ../<wherever>/package
>

This method does not work if someone clone the application repository
manually using VCS. I am not sure is this by design or an issue. For
example, here is the application with "replace" directive [1],

master ms 0 % git clone https://github.com/shuLhan/rescached-go
Cloning into 'rescached-go'...
remote: Counting objects: 277, done.
remote: Compressing objects: 100% (158/158), done.
remote: Total 277 (delta 141), reused 234 (delta 98), pack-reused 0
Receiving objects: 100% (277/277), 264.64 KiB | 544.00 KiB/s, done.
Resolving deltas: 100% (141/141), done.

6:19 ~/tmp
master ms 0 % cd rescached-go

6:19 ~/tmp/rescached-go
master ms 0 % git reset --hard
47c04cb4542505173a0216edffb5b8388838966f
HEAD is now at 47c04cb make: fix typo on sudo

6:28 ~/tmp/rescached-go
master ms 0 % cat go.mod
module github.com/shuLhan/rescached-go

require github.com/shuLhan/share v0.0.0-20180827221555-192454617153

replace github.com/shuLhan/share => ../share


When I run go build, it will report error about missing module,

6:28 ~/tmp/rescached-go
master ms 0 % go build ./...
go: parsing ../share/go.mod: open /home/ms/tmp/share/go.mod: no such
file or directory
go: error loading module requirements


If I remove the "replace" directive, Go can build the package.

6:31 ~/tmp/rescached-go
master ms 0 % cat go.mod
module github.com/shuLhan/rescached-go

require github.com/shuLhan/share v0.0.0-20180827221555-192454617153

6:31 ~/tmp/rescached-go
master ms 0 % go build ./...


I think my problem is thinking that "replace" directive is optional, as
in,

If "replace" path not exist, use module from cache, otherwise get the
required package from remote into cache.--
Shulhan

because the syntax use relative import, not absolute import path.

[1]
https://github.com/shuLhan/rescached-go/commit/47c04cb4542505173a0216edffb5b8388838966f

[Seth Hoenig]

`go.mod.local` was one idea. Another idea is to have a sort of "publish local" semantics, where the go tool has support for something like <version>-devel tags, which override the <version> defined in go.mod.  So then you would "publish" the next version to your local mod cache (just creating/updating the module of the special version), and the go tool would then make use of that.

[Mirko Friedenhagen]

Publishing to the local cache is how Maven, a build tool for Java, does it. There is the concept of snapshot versions. For Golang maybe stating master (or any other branch) as version would be fitting. Then your CI could use a master checkout as well.

Regards Mirko

[Jim]

I would suggest that developers get used to using one or more separate branch for releases. Periodically, master will be merged into these branches as part of the release process.

It's likely that the source for your package (including go.mod) won't be the only thing that's different between the development and release branches. Your clients will want reproducible builds so you'll want to freeze go.mod for a release with the dependencies used to build that release.

Changes from the master/development go.mod will need to be carefully merged into release go.mod.

You'll only need to do this merge when you decide to update your release dependencies.

You should plan on developing in a different branch (with a different go.mod) from the one you use for releases. I don't think go.mod complicates this process. In fact, I would argue it simplifies it.

[Dan Kortschak]

And here we have the problem. Low probability of error events do not
scale.

[thepud...@gmail.com]

Hi all,

This thread is hitting on a fairly important set of topics.

It certainly seems some additional brainstorming and community discussion about how things could or should work regarding multi-module workspaces would be beneficial.

A file tree with a `go.mod` has been described as something like "a little GOPATH", but as has been observed, if you want to edit multiple modules simultaneously, you have to decide how you want to wire together those multiple "little GOPATHs".

Manually adding `replace` directives is one approach to using local copies of related modules that you might be simultaneously editing.

However, that can translate to many manual steps, especially if they are frequently added and removed.

If you haven't already, it is definitely worthwhile to look at https://github.com/rogpeppe/gohack, which is a new community tool to automate and greatly simplify multi-module workflows and related `replace` directive workflows. It allows you to very easily modify one of your dependencies. For example, `gohack github.com/some/dependency` automatically clones the appropriate repository to your local disk and adds the necessary `replace` directives to your go.mod so that your build picks up the local copy that you can start editing. You could do this as part of a one-off investigatory workflow, or you could do `gohack example.com/foo  example.com/bar  example.com/baz` as part of your normal development setup if that makes sense for your workflow. When done, `gohack -u` removes all gohack replace statements (which can be done prior to committing).

A related note is that community-based tooling for modules in general is starting to emerge. You can see an initial list here:

  

  https://github.com/golang/go/wiki/Modules#what-community-tooling-exists-for-working-with-modules

  

Overall, it seems clear one of the intents of the modules system is to enable one-off or community-based tooling to be built on top of the core modules functionality, and often it can be done in a relatively straightforward manner. For example, there is whole section of the official documentation titled "Edit go.mod from tools or scripts" (https://tip.golang.org/cmd/go/#hdr-Edit_go_mod_from_tools_or_scripts).

Some example capabilities for inspecting module configuration (which might be used to encode a sanity check, or to automate further tooling):

  *  `go mod edit -json` outputs the current module's `go.mod` file in JSON format.

  *  `go mod edit -json <path/to/go.mod>` outputs a particular `go.mod` file in JSON format.

  *  `go list -m -json all` gives a JSON representation for the full set of modules available to a build, including the path to each `go.mod` file. (You can then feed each `go.mod` path to `go mod edit -json <path/to/go.mod>` to get the additional details for that `go.mod`).

Some example capabilities for programmatically editing module configuration (which for example could support automating adding/removing `replace` directives to wire together multiple related modules for simultaneous editing of multiple modules):

  *  `go mod edit -replace github.com/my/foo=../foo` adds a `replace` statement to use a copy of `foo` on the local disk.

  *  `go mod edit -dropreplace=github.com/my/foo` removes all replacements for `foo` from the current `go.mod`.

One of the concerns raised in this thread is that if you use a `replace` directive to point a module to a development copy of another module, someone might accidentally check that the `go.mod` containing the `replace` directive. 

To help guard against that (and as a simple example of one-off tooling that could be built on top of 1.11 modules), you could consider some type of pre-commit VCS check or CI check to validate that a go.mod does not have any development replace directives, such as something along the lines of:

    go mod edit -json | python2.7 -c 'import json, sys; assert json.load(sys.stdin)["Replace"] == None'

(Sorry for slipping in a python one-liner; that was just to convey a simplified example. Obviously could instead be in Go, bash, jq, or whatever your preferred method might be).

There are admittedly many different opinions on how to use git, but another approach to add and remove development-specific `replace` directives could be via something like the git smudge/clean workflow that lets you tweak things on checkout (e.g., add `replace` directives) but then automatically reverse the tweak when checking back in (to strip out development `replace` directives). Or perhaps something similar could be automated however might work best for your current workflows...

A completely different approach could be to make the on-disk organization of your multiple modules always be the same in their development/environment/test/build/CI environment, and wire that together via replace directives that you do check in (in contrast to adding and removing replace directives for specifically for development). That is not something that would work for all projects, but perhaps something to consider. A small number of modules could be manually wired together like that via a one-time hand edit of the various `go.mod` files, or the actual edits to the `go.mod` files could be automated if there are many modules that would need to be touched. 

Overall, and setting aside any particular solution -- there are probably a few different timescales to think about regarding the topic of this thread and how multi-module workspaces should be set up:

  1. What do people do immediately? (For example, based on manual workflows, or via small amounts of automation on top of 1.11 modules).

  2. What might be done in community tools like `gohack` on top of 1.11? (For example, hopefully people can contribute ideas or pull requests to `gohack` or other existing efforts, or perhaps some people might start their own experimental module tooling repo, which might help flesh out a few different independent perspectives).

  3. How could and should this all work in a 1.12 time frame?

Part of the nature of the current on-going modules experiment is such that hopefully that near-term experience and community experimentation will help shape what happens in a 1.12 time frame. 

   

Stepping backing even further, though -- when it comes to multi-module projects, one key initial question to ask is how many modules do you want to create for a given project, including asking yourself what do you really need to version separately (given modules are defined as "the unit of source code interchange and versioning"). Especially to start as people ramp up on modules and as the community is working out different workflows, larger modules are likely easier than finer-grained modules, and one module per repo is definitely easier than multiple modules per repo. The official modules proposal is predicting "most projects will adopt a workflow in which a version-control repository corresponds exactly to a single module" (from https://golang.org/design/24301-versioned-go). That said, different people will have different preferences.

Sorry for the long-ish comments, but I guess my broadest point would be that the topics raised in this thread do seem worthy of some thinking and discussion by the overall community...

--thepudds

[thepud...@gmail.com]

Seth and/or Mirko,

What would you think about one of you opening a new issue that covers the possible approach you were outlining earlier in this thread:

   "Another idea is to have a sort of "publish local" semantics, where the go tool has support for something like <version>-devel tags, which override the <version> defined in go.mod.  So then you would "publish" the next version to your local mod cache (just creating/updating the module of the special version), and the go tool would then make use of that."

and

   "Publishing to the local cache is how Maven, a build tool for Java, does it. There is the concept of snapshot versions. For Golang maybe stating master (or any other branch) as version would be fitting. Then your CI could use a master checkout as well."

At least, I don't see an issue suggesting that approach... 

It could be worth it to jot down some quick thoughts on how you would envision that working.

 

--thepudds

[thepud...@gmail.com]

Hi all,

Also, for reference, here are three other related issues:

  #26640 "cmd/go: allow go.mod.local to contain replace/exclude lines"

  https://github.com/golang/go/issues/26640

  #26377 "proposal: cmd/go: module semantics in $GOPATH/src"

  https://github.com/golang/go/issues/26377

  

  #27542 "cmd/go: support simultaneous edits of interdependent modules"

  https://github.com/golang/go/issues/27542

--thepudds

[Scott Cotton]

Hi all,

wanted to follow up to @thepudds summary below.   

I work with multiple modules.  For me, I have found the following helpful

1. keeping dependencies acyclic 

2. organising the granularity of what is the smallest module to be as coarse as possible, so there aren't too many modules and

release versions can refer to a project with legitimate interest all by itself.

3. making sure that each module has a clear goal distinct from the others.

4. instead of replace/local modifications and extra tooling I have

    a) used lots of micro-granularity semver tags using prerelease notation, allowing to bump pre-releases as much as I want without worrying about releases.    

    b) when nonetheless in a bind for local multi-module edits (which is rare given 2,3 above) simply follow dependency order.  

    c) Once, instead of editing go.mod and using tooling trickery, I just made a copy of one package in one module in another temporarily

and changed the import path. Simple cp -r did the trick, no need to worry about how it effected go.mod or versions.

The advantages to me compared to other options presented by @thepudds below are:  no multi module setup to need to think about, no extra tools to learn.  

The disadvantages:  probably less flexible.  less opportunity to contribute to shared tooling.

Hope that helps anyone considering workflows with multiple modules

Scott