$GOPATH/src/mod

[Matt Joiner]

Is there any reason the module stuff has to go here? It breaks a lot of existing tools, and I can't determine the benefit. Why not just put it in $GOPATH/mod? This avoids having to update the entire world of tooling out there.

[Russ Cox]

On Wed, Jul 25, 2018 at 4:21 AM, Matt Joiner <anac...@gmail.com> wrote:

Is there any reason the module stuff has to go here? It breaks a lot of existing tools, and I can't determine the benefit. Why not just put it in $GOPATH/mod? This avoids having to update the entire world of tooling out there.

A bunch of people set GOPATH=$HOME, so I didn't want to create yet another top-level directory in GOPATH. 

The choices are bin, pkg, and src. 

GOPATH/bin is clearly wrong. 

GOPATH/pkg might be OK except that we want to retire that directory entirely (#4719).

GOPATH/src remains.

Also, tools should in general not be walking over all of GOPATH/src, even before modules - that simply doesn't scale. If this GOPATH/src/mod helps us find and fix those tools, that's not an entirely bad thing.

Best,
Russ

[Dave Cheney]

> Also, tools should in general not be walking over all of GOPATH/src

A common command, go build .../cmd/unused

Does this, as far as a I understand.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Russ Cox]

On Wed, Jul 25, 2018 at 10:39 AM, Dave Cheney <da...@cheney.net> wrote:

> Also, tools should in general not be walking over all of GOPATH/src

A common command, go build .../cmd/unused

Indeed. That's golang.org/issue/26401 (as you know) and we will probably backport it to a Go 1.10 point release too.

Russ

[Matt Joiner]

What happens to gorename? It's not going to work as intended "across
GOPATH" anymore. Renaming within a single module is helpful, but not
its real power.

[Vladimir Varankin]

I think the question is why mod stuff goes to GOPATH, why doesn't it use GOCACHE or XDG_CACHE_HOME, or even creates its own location? A potential problem with putting them into GOPATH is that tools that scan over GOPATH, e.g. IDEs or godoc, could go crazy because of the increased number of new random stuff in the workspace.

[Mark Pulford]

On Thu, Jul 26, 2018 at 04:27:38AM -0700, Vladimir Varankin wrote:
> GOCACHE or XDG_CACHE_HOME, or even creates its own location? A potential

I was just about to ask if that might be reasonable :). src/mod is
effectively a cache that can typically be replaced (especially when
Athens or something with that functionality is available).

If using XDG_CACHE_HOME isn't always suitable a GOMOD variable could
specify the location.

In a post-GOPATH world, using $XDG_CACHE_HOME/go-mod or similar would
leave only "go install" using $GOPATH/bin. GOPATH could then be
deprecated in favour of GOBIN/GOMOD.

-m

[Carolyn Van Slyck]

Agreed, I was hoping that the contents of mod/ would be placed in GOCACHE as well.

[Russ Cox]

> Agreed, I was hoping that the contents of mod/ would be placed in GOCACHE as well.

I understand why this would seem surprising at first, but cached compilation resuilts and cached downloaded source code are a bit different.

The build cache ($GOCACHE, defaulting to $HOME/.cache/go-build) is for storing recent compilation results, so that if you need to do that exact compilation again, you can just reuse the file. The build cache holds entries that are like "if you run this exact compiler on these exact inputs. this is the output you'd get." If the answer is not in the cache, your build uses a little more CPU to run the compiler nstead of reusing the output. But you are guaranteed to be able to run the compiler instead, since you have the exact inputs and the compiler binary (or else you couldn't even look up the answer in the cache).

The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is for storing downloaded source code, so that every build does not redownload the same code and does not require the network or the original code to be available. The module cache holds entries that are like "if you need to download mymo...@v1.2.3, here are the files you'd get." If the answer is not in the cache, you have to go out to the network. Maybe you don't have a network right now. Maybe the code has been deleted. It's not anywhere near guaranteed that you can redownload the sources and also get the same result. Hopefully you can, but it's not an absolute certainty like for the build cache. (The go.sum file will detect if you get a different answer on re-download, but knowing you got the wrong bits doesn't help you make progress on actually building your code. Also these paths end up in file-line information in binaries, so they show up in stack traces, and the like and feed into tools like text editors or debuggers that don't necessarily know how to trigger the right cache refresh.)

I expect there are cron jobs or other tools that clean $HOME/.cache periodically. If part of the build cache got deleted, it would be no big deal, so it's fine to store the build cache there. But if downloaded source code got deleted unasked, I think that would potentially be quite surprising and problematic in various ways. That's why we store the source code in $GOPATH/src/mod, to keep it away from more expendable data.

Russ

[Carolyn Van Slyck]

I’ve been telling people who are learning about Go Modules that it’s okay to nuke the mod directory whenever things get weird (mostly because are playing around with Athens and other stuff and corrupting it).

 

Are there situations where deleting the mod directory would have a bad effect other than causing the module cache to be repopulated?

[Ralph Corderoy]

Hi Russ,

> The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is
> for storing downloaded source code, so that every build does not

> redownload the same code and does not require... the original code to
> be available.

So it has two roles, and the most important seems to be as an archive of
downloaded source code, not as a cache of it; that's a bonus.

I think the common accepted usage of `cache' in computing terms is a
duplicate copy conveniently located for speed that can be nuked with
*only* speed taking the hit. Perhaps if we swung more towards `module
archive' for referring to src/mod then it would lessen the confusion
with the build cache, which is a cache in the accepted sense.

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

[Russ Cox]

On Thu, Jul 26, 2018 at 10:12 AM, Carolyn Van Slyck <Carolyn.VanSlyck@microsoft.com> wrote:

I’ve been telling people who are learning about Go Modules that it’s okay to nuke the mod directory whenever things get weird (mostly because are playing around with Athens and other stuff and corrupting it).

 

Are there situations where deleting the mod directory would have a bad effect other than causing the module cache to be repopulated?

If the bits you need are still available for re-downloading from the network, there's no bad effect. That's just a big if.

Russ

[Vladimir Varankin]

I see your point about the need for the source code to be downloaded to be able to run "go build" / "go test". But pointing specifically on the fact that the origin of the dependencies might not be available after the cache was flushed, is a bit strange (I'm really sorry in if I misunderstood something, and sorry in advance if this goes outside of the original question). 

In real-world projects, where many people work on the same codebase, sharing their changes, running tests, and builds on the CI, one can't simply say "If the bits you need are still available for re-downloading from the network <..>. That's just a big if." I really don't see how modules solve this particular issue without vendoring? I mean how would they fix an integration and delivery processes that run in a clean environment without pre-populated stuff in src/mod, which might not be available?

And if we go back to the point about cache cleaning, what would be your suggestion about cleaning src/mod from dependencies that are not required by any project? I can picture a situation when, after some long period of time, a hard drive will be full of unused things in src/mod, and it will be very tempting to have the very same cronjob you mentioned that flushes it.

On Thu, 26 Jul 2018 at 18:24, Russ Cox <r...@golang.org> wrote:

On Thu, Jul 26, 2018 at 10:12 AM, Carolyn Van Slyck <Carolyn....@microsoft.com> wrote:

I’ve been telling people who are learning about Go Modules that it’s okay to nuke the mod directory whenever things get weird (mostly because are playing around with Athens and other stuff and corrupting it).

 

Are there situations where deleting the mod directory would have a bad effect other than causing the module cache to be repopulated?

If the bits you need are still available for re-downloading from the network, there's no bad effect. That's just a big if.

Russ

--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/RjSj4bGSmsw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Vladimir Varankin

vlad...@varank.in

[Carolyn Van Slyck]

This is why I’m looking forward to the Athens project which implements a proxy and registry for go! It’s not quite ready to try out yet, but once it is I can post back on this list with info on how to kick the tires.

 

https://github.com/gomods/athens

 

 

 

From: Vladimir Varankin <vlad...@varank.in>
Sent: July 26, 2018 3:03 PM
To: r...@golang.org
Cc: Carolyn Van Slyck <Carolyn....@microsoft.com>; golan...@googlegroups.com
Subject: Re: [golang-dev] $GOPATH/src/mod

 

I see your point about the need for the source code to be downloaded to be able to run "go build" / "go test". But pointing specifically on the fact that the origin of the dependencies might not be available after the cache was flushed, is a bit strange (I'm really sorry in if I misunderstood something, and sorry in advance if this goes outside of the original question). 

[Russ Cox]

After thinking more about the points raised here (and on the issue tracker),

I've sent CL 126755 to move the module archive/cache to $GOPATH/pkg/mod,

next to dep's $GOPATH/pkg/dep it turns out.

Thanks as always for the helpful feedback.

Best,

Russ

[Dave Cheney]

Excellent idea. Thanks Russ.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an

[Florin Pățan]

If the goal is to remove $GOPATH/pkg then isn't moving this under $GOPATH/pkg/mod creating more work for us in the future when GOPATH will be removed/not mandatory anymore? I know we have now a default path for it but will this be there in the future? Shouldn't we move this under $HOME/.go/mod or something like that so that other tools can also use this in the future as well rather than inventing their own spaces?

[Henrik Johansson]

I agree with Florin why don't we put it in $HOME/....? Not only does it make the removal of GOPATH easier but it is well known from other languages such as Mavens .m2/repository for example.

tis 31 juli 2018 kl 11:52 skrev Florin Pățan <flori...@gmail.com>:

If the goal is to remove $GOPATH/pkg then isn't moving this under $GOPATH/pkg/mod creating more work for us in the future when GOPATH will be removed/not mandatory anymore? I know we have now a default path for it but will this be there in the future? Shouldn't we move this under $HOME/.go/mod or something like that so that other tools can also use this in the future as well rather than inventing their own spaces?

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.

[ra...@develer.com]

Il giorno giovedì 26 luglio 2018 18:24:55 UTC+2, rsc ha scritto:

On Thu, Jul 26, 2018 at 10:12 AM, Carolyn Van Slyck <Carolyn....@microsoft.com> wrote:

I’ve been telling people who are learning about Go Modules that it’s okay to nuke the mod directory whenever things get weird (mostly because are playing around with Athens and other stuff and corrupting it).

 

Are there situations where deleting the mod directory would have a bad effect other than causing the module cache to be repopulated?

If the bits you need are still available for re-downloading from the network, there's no bad effect. That's just a big if.

Module support has been integrated in "go build" so that it is transparent, goes to the network and downloads stuff. This also mirrors the experience of all other package managers out there that, through a dedicated command, expect to be able to download packages/modules and use them. 

People are generally aware of the risk that a repository or a version might go away, or be push-forced to be something else, and it's a reasonable assumption given the mental model of downloading stuff off GitHub. Obviously, we do need a solution to solve this problem for people that cannot (or do not want) to run this risk. The solution, it seems, is to vendor dependencies through "go mod -vendor".

If that is the one and official solution, I don't see what giving more "prominence" to the module archive wins us. It is an intermediate solution that just saves the day for people that happened to download the same version before, that didn't reinstall their computer or cleaned up the directory for other reasons; it creates additional differences in behavior in the already-confusing case where a remote repository disappears or is push-forced (different people with different module archives will have different behaviors). I think it should just be a simple download cache, subject to the usual cache rules. We already have a solution for people that cannot run the "network risk" (vendoring), we don't need a half-backed second solution.

Giovanni

[Russ Cox]

On Tue, Jul 31, 2018 at 5:52 AM, Florin Pățan <flori...@gmail.com> wrote:

If the goal is to remove $GOPATH/pkg then isn't moving this under $GOPATH/pkg/mod creating more work for us in the future when GOPATH will be removed/not mandatory anymore? I know we have now a default path for it but will this be there in the future? Shouldn't we move this under $HOME/.go/mod or something like that so that other tools can also use this in the future as well rather than inventing their own spaces?

When I do stack traces I don't want to see source code in dot-directories, and I don't want to invent yet another environment variable today.

GOPATH is already not mandatory and defaults to $HOME/go, so this directory defaults to $HOME/go/pkg/mod, which seems like a completely reasonable name. Issue 4719 is about getting rid of all the .a files. If $HOME/go/pkg/mod is left behind that won't be the end of the world. Also it's very little work to move - nothing should be hard-coding that path.

Russ