"Multi-Module Workspaces in cmd/go" draft proposal

[Michael Matloob]

Hello!

I've put up a draft design for a new workspace mode in the go command to be used when working on multiple modules at the same time. It's here: https://go.googlesource.com/proposal/+/refs/heads/master/design/draft-workspace.md

I introduced it at the tools call earlier today, but didn't say anything that's not in the doc. (and the doc's gotten better editing than my remarks on the call!)

It's not quite ready for a proposal yet, but I wanted to see what y'all think about it, so please feel free to discuss and leave any comments here!

Thanks!

Michael

[Daniel Martí]

Thanks for working on this, Michael! I'm sure you've seen that multiple
threads on GitHub are advancing towards multi-module development in
different directions. I hope your proposal, when filed soon, can help
unify the conversations and get us to a solution.

After a second read of the draft, here are some questions:

What kind of semantic changes do we imagine will happen with go.work's
"go X.Y" directive? Since workspaces are just for local development,
could one not simply get the new semantics by upgrading the Go version
being used?

I'm a bit worried about replace directives in workspace modules applying
to the entire build list. I get why that's likely the only reasonable
option here, but it could easily get confusing when one has more than a
handful of workspace modules. Maybe we want to encourage go.work users
to use go.work replace directives instead, where possible. Or,
alternatively, some way of seeing all the replace directives in effect.

I seem to understand that if workspace mode is on and I'm also inside a
module, that module *must* be listed in go.work's directory. Does that
mean that, if it's not, the go tool will just error? If the module does
not belong in the workspace, is our solution here "move the module
outside the workspace directory"?

If my understanding above is correct, then perhaps I'm missing why the
directory is necessary, if all modules found under the workspace
directory must be part of it. We could reasonably cheaply walk the
directory tree to find all go.mod files. If that's too expensive to do
each time, commands like "go mod initwork" could record them as a form
of cache.

I'm still unconvinced that we need to be able to use the build list in
workspace mode but outside any of its modules. This seems like a feature
that we could keep out of the initial proposal, so if you want to keep
it there, I'd add a section to the draft explaining the use case.

I think the proposal should be opinionated as to whether go.work files
should be published in VCS. Some people initially thought that go.sum
files shouldn't be published, for example. My assumption is that, since
go.work files are for local development, they should generally not be
published. But perhaps it's reasonable to do for monorepos containing
many modules?

If I have two worspace modules A and B, and A depends on B, would
running "go list all" in B list all of A's packages? This is how I
understand the proposal. It does make the proposal more consistent, as
there is just one build list for the entire workspace, no matter what
module one is in. But, from the user's point of view, I'd find it a bit
surprising if the "go list all" command in B listed packages which are
not transitive dependencies of the current module.

This is enough of a wall of text, so I'll stop here :)

On Wed, Apr 14, 2021 at 22:30:05 -0400, Michael Matloob wrote:
> Hello!
>
> I've put up a draft design for a new workspace mode in the go command to be
> used when working on multiple modules at the same time. It's here:
> https://go.googlesource.com/proposal/+/refs/heads/master/design/draft-workspace.md
>
> I introduced it at the tools call

> <https://github.com/golang/go/wiki/golang-tools#calls> earlier today, but

[Michael Matloob]

On Thursday, April 15, 2021 at 10:22:35 AM UTC-4 ... wrote:

Thanks for working on this, Michael! I'm sure you've seen that multiple
threads on GitHub are advancing towards multi-module development in
different directions. I hope your proposal, when filed soon, can help
unify the conversations and get us to a solution.

After a second read of the draft, here are some questions:

What kind of semantic changes do we imagine will happen with go.work's
"go X.Y" directive? Since workspaces are just for local development,
could one not simply get the new semantics by upgrading the Go version
being used?

Even though I think it's fair game as far as our compatibility promises go to

change the behavior across go versions, I think most users would be unhappy

to see their workspaces behaving differently without opting into it themselves.

I hope we don't have to make many or any breaking changes to the semantics,

but our experience with the go directive in modules tells us it's better to be

safe here.

 

I'm a bit worried about replace directives in workspace modules applying
to the entire build list. I get why that's likely the only reasonable
option here, but it could easily get confusing when one has more than a
handful of workspace modules. Maybe we want to encourage go.work users
to use go.work replace directives instead, where possible. Or,
alternatively, some way of seeing all the replace directives in effect.

Yeah, I think the design of replace is very constrained by the way replace

directives work in go.mod files.

I definitely want to encourage Go users to use replace directives in go.work

where possible. Hopefully, if this proposal is accepted, replaces in go.mod files will

 become rare and only used temporarily to work around bugs in dependencies.

But I think it would be useful to see the set of replace directives in effect.

I added a 'future work' section to the doc about listing the set of module roots

and replace directives across the set of modules to make it more clear

to users what their set of root dependencies actually are. What do you think?

 

I seem to understand that if workspace mode is on and I'm also inside a
module, that module *must* be listed in go.work's directory. Does that
mean that, if it's not, the go tool will just error? If the module does
not belong in the workspace, is our solution here "move the module
outside the workspace directory"?

I expect go.work files in repos will be added deliberately

so and only if all the modules under that file's directory belong in the

workspace. Outside of a repo I'd expect a user would only add a go.work

file to a directory containing modules if they expect all the modules under

that directory belong to the workspace (or they could add gitignored workspace files

to the modules to cut them out of the parent workspace).

If a user does end up with a workspace directory that contains a module

that doesn't belong in the workspace they could disable workspaces with

-workfile=off. I don't want the recommendation to users to be to "move the

module outside the workspace directory", but I can see that that ends up

possible.

If my understanding above is correct, then perhaps I'm missing why the
directory is necessary, if all modules found under the workspace
directory must be part of it. We could reasonably cheaply walk the
directory tree to find all go.mod files. If that's too expensive to do
each time, commands like "go mod initwork" could record them as a form
of cache.

Do you mean to have the go command implicitly create a workspace

consisting of all the module the current directory is contained in and

all modules in subdirectories?

 

I'm still unconvinced that we need to be able to use the build list in
workspace mode but outside any of its modules. This seems like a feature
that we could keep out of the initial proposal, so if you want to keep
it there, I'd add a section to the draft explaining the use case.

Added a section about this to the rationale in the latest revision.

There are two reasons for this: keeping the workspace scope model similar to go.mod

makes it easier for Go users to start working with workspaces. It also

has the nice property of allowing users to create a GOPATH like setup

by putting a go.work file in their home directory that contains the set

of modules they work on.

I think the proposal should be opinionated as to whether go.work files
should be published in VCS. Some people initially thought that go.sum
files shouldn't be published, for example. My assumption is that, since
go.work files are for local development, they should generally not be
published. But perhaps it's reasonable to do for monorepos containing
many modules?

That's exactly my position. Go.work files should only be in a multi-module

repo where the modules are tightly-coupled together. Added a sentence

to the section about multi module repositories saying that go.work

files shouldn't be checked in for other cases.

 

If I have two worspace modules A and B, and A depends on B, would
running "go list all" in B list all of A's packages? This is how I
understand the proposal. It does make the proposal more consistent, as
there is just one build list for the entire workspace, no matter what
module one is in. But, from the user's point of view, I'd find it a bit
surprising if the "go list all" command in B listed packages which are
not transitive dependencies of the current module.

Hm, I'm hoping that most users end up with the mental model

 "I'm running 'go list all' in my workspace" rather than

 "I'm running 'go list all' in module B". Some users might already

be used to thinking like that if they were using go in the GOPATH

days. Though of course I'm sure there are things we can

do to make that more clear.

 

This is enough of a wall of text, so I'll stop here :)

Thanks for the comments!

[Sean Liao]

This looks great! A few comments:

I'm still a bit confused about the scope of a directory directive,

does it point to a single module , where the 

// module/path comment would make sense,

or to all the modules under the directory,

 as your reply here seems to hint at

Also I agree the replace semantics are somewhat confusing,

and goes counter to what we have been telling users about

replace only mattering in the main module.

I was expecting each package in the workspace to be built

with the main module being the one it is in,

and only replaces there and in workspace taking effect.

Is the replace redundant with directory?

replace (

    golang.org/x/tools => ./tools

)

Single dot or ./ ?

directory ( 

    .

)

Is the * something new?

replace example.com/foo * => example.com/foo v0.3.4

[t hepudds]

Hi Michael / all,

This is very exciting to see!

Just a couple quick comments.

One thing is the shared build list semantics might be problematic in some cases, including in a future with lazy module load. If the modules in the workspace have shared dependencies, it seems it might be possible in some cases for module A to end up with a different version selected for a shared dependency if module B is in the same workspace, compared to if module A was just consuming the exact same version of module B normally. For example, if module A was just importing a single package from module B, in the normal case without a workspace under lazy load, module A wouldn’t see the full set of direct and indirect dependencies for all of B’s packages, but it sounds like the shared build list in the workspace would see the full set of requirements for all of B’s packages, which could then mean different versions being selected for the same code & go.mod files depending on whether or not a workspace is being used. If so, that seems problematic?

That said, it might be that I'm not understanding the proposal, or I might not understand lazy load. ;-)

Separately, rather than an init command, it might be better to have a sync command that adds new modules to the workspace file or removes them as needed, ideally reporting what it did. That might be more helpful over time than a one shot init command?

For multi-modules workspace, a flag to disable the workspace as outlined in the proposal likely covers the common cases of wanting to test against published versions prior to release, but it also seems commenting out the directories in the workspace file could cover additional cases (e.g., if you have a 4 module workspace, and want to test how the changes in 2 of the modules work against the published versions of the other 2 modules). Is that true, and if so, is it worth mentioning?

It might also be useful to have something like ‘go work status’ (however it might be spelled) that gives an easily digestible summary. 

In any event, very nice, and sorry if I’ve misunderstood anything…

Regards,

thepudds 

On Apr 17, 2021, at 7:50 AM, Sean Liao <seank...@gmail.com> wrote:



--
You received this message because you are subscribed to a topic in the Google Groups "golang-tools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-tools/0HfA1HWaTJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/7f99fe8f-21a8-4731-a52f-d69333fe2c36n%40googlegroups.com.

[Jan Mercl]

I don't want to sound harsh, sorry if that impression kicks in anyway.

For a rather long time my typical operational mode is hacking more
packages at the same time. I don't want to use the replace directive
for this as I consider it useful only for permanent replacement. As a
part of a normal/daily workflow I refuse to risk I forget to remove it
before commiting. Low probability, but it accumulates. (I don't mind
the actual editing part of the go.mod file, that's a minute detail and
takes just a few seconds once a day.)

So far I'm just prefixing the go command(s) with GO111MODULE=off all
the time when and where needed to debug the work-in-progress of the
collection of packages I'm just working on. Again, I don't mind it. A
bit annoying, but takes just a few seconds and when I repeat the
edit-test cycle commands by endlessly pressing the up-arrow, I don't
have to type the env var setting one more time.

I read the proposal, hoping it will be the solution to the situation
when GO111MODULE will be ignored, IIRC starting with Go 1.17.

tl;dr: For me it is not. Because too much ceremony/configuration.

The best part of the GO111MODULE=off mode is the exact opposite.
Modulo the env var itself, there's no configuration necessary. I love
that. The packages I work at once on are my own. That makes them work
easily together in the gopath mode. I realize this is not the case
when versioning is essential to the build with "external", from my
POV, packages. But I don't work in the "package group mode" on
projects that would need that. IOW, my requirements are rather limited
compared to what the proposal offers - which is, IIUC a probably
better, general solution.

For my limited use case it's probably better for me to write a simple
set of tools, say `go-build` and `go-test`, that will just wrap the
'go tool compile` command, provided it is still able to simply compile
any set of files one throws at it as a package and without enforcing
the module mode somehow. I hope that is performed outside the compiler
itself. Haven't yet checked if that's actually the case. It would be
unfortunate if not - as then I'd have to stay on Go 1.16 for as long
as possible and hope some acceptable-for-me solution/idea will
materialize before too late.

[t hepudds]

Hi Jan,

You might have already seen this, but if not, you might be interested in this proposal:

#44649 “cmd/go: introduce 'module GOPATH' to allow transition from GOPATH to modules”
https://github.com/golang/go/issues/44649

(IIRC, you prefer not to have a github account, but do read issues there?)

Regards,
thepudds

> On Apr 17, 2021, at 10:09 AM, Jan Mercl <0xj...@gmail.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "golang-tools" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/CAA40n-UKJ%3D8ozK4K-uZ-rD09p7GEVursiUsds7YT%3DTdRyXfJJg%40mail.gmail.com.

[Jan Mercl]

On Sat, Apr 17, 2021 at 4:47 PM t hepudds <thepud...@gmail.com> wrote:

> You might have already seen this, but if not, you might be interested in this proposal:
>
> #44649 “cmd/go: introduce 'module GOPATH' to allow transition from GOPATH to modules”
> https://github.com/golang/go/issues/44649
>
> (IIRC, you prefer not to have a github account, but do read issues there?)

Thanks.

I think bcmills pointed me to this issue before at Slack, very early,
because since then there are now many interesting replies.

I do skim the issue tracker a few times a week. Unfortunately, there's
no way to subscribe to the follow ups w/o a github account.

Wrt the issue itself, I have trouble understanding it. It uses terms
like "main module", "unprefixed module", "workspace", "global
namespace", "local namespace" - but none of those are ever
mentioned/defined in [0] or [1], for example, and I did not have time
to immediately investigate, neither I can ask at the issue tracker. So
it fell off my radar as in "I will revisit later", but that never
happened. As in many other similar cases before. My excuse is not
enough time, but the truth is probably that I'm just a lazy student,
allocating the precious time to learning only what I cannot avoid
learning.

[0]: https://golang.org/ref/spec
[1]: https://golang.org/doc/modules/managing-dependencies

[Michael Matloob]

On Sat, Apr 17, 2021 at 7:50 AM Sean Liao <seank...@gmail.com> wrote:

This looks great! A few comments:

I'm still a bit confused about the scope of a directory directive,

does it point to a single module , where the 

// module/path comment would make sense,

or to all the modules under the directory,

 as your reply here seems to hint at

The scope of a directory directive is a single module. I should have

been more clear in my response here: the directory directive applies

to a single module, and each submodule in a multi-module repo

must be added individually. Is there anywhere in the doc itself

that isn't clear? If so let me know and I'll fix it.

 

Also I agree the replace semantics are somewhat confusing,

and goes counter to what we have been telling users about

replace only mattering in the main module.

I was expecting each package in the workspace to be built

with the main module being the one it is in,

and only replaces there and in workspace taking effect.

That would be an alternate way of creating workspaces, but

while it works when running the go command to test one

module in development against another,

it wouldn't solve the problem of having a consistent view of the

workspace when working from an editor: files in one module

would see different dependency from files in another module.

Which means that various editor tooling would show inconsistent

information to users as they move between modules.

I think if we can make it easy for users to internalize that there are multiple

main modules in workspace mode rather than just one, the replace semantics

shouldn't be surprising: the replaces from all the main modules matter

in all the main modules.

Is the replace redundant with directory?

replace (

    golang.org/x/tools => ./tools

)

It's redundant in one direction but not another: if a module is specified

in the directory directive, then adding a replace is redundant, but if

a module is replaced, adding a directory directive promotes it to

a main module, which significantly changes the semantics.

 

Single dot or ./ ?

directory ( 

    .

)

I think . should be fine (we don't require '/' at the end other directory names)

 

Is the * something new?

replace example.com/foo * => example.com/foo v0.3.4

This was a mistake I just meant replace example.com/foo => example.com/foo v0.3.4.

 

Thanks for the feedback!

--
You received this message because you are subscribed to a topic in the Google Groups "golang-tools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-tools/0HfA1HWaTJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/7f99fe8f-21a8-4731-a52f-d69333fe2c36n%40googlegroups.com.

[Michael Matloob]

On Sat, Apr 17, 2021 at 9:21 AM t hepudds <thepud...@gmail.com> wrote:

Hi Michael / all,

This is very exciting to see!

Just a couple quick comments.

One thing is the shared build list semantics might be problematic in some cases, including in a future with lazy module load. If the modules in the workspace have shared dependencies, it seems it might be possible in some cases for module A to end up with a different version selected for a shared dependency if module B is in the same workspace, compared to if module A was just consuming the exact same version of module B normally. For example, if module A was just importing a single package from module B, in the normal case without a workspace under lazy load, module A wouldn’t see the full set of direct and indirect dependencies for all of B’s packages, but it sounds like the shared build list in the workspace would see the full set of requirements for all of B’s packages, which could then mean different versions being selected for the same code & go.mod files depending on whether or not a workspace is being used. If so, that seems problematic?

That's correct, but I don't think it's problematic. It can happen even outside of lazy loading, 'promoting' a dependency module to a workspace module changes the semantics: for example, its replace directives apply. And of course, adding a non-dependency module to the workspace can also change the set of dependencies. The reason I don't think it's problematic is that the set of dependencies that a developer sees building a library module will usually not be the same as the set of dependencies that a user of that library module sees. So trying to be faithful to the single module buildlist doesn't buy that much. Of course that's not true if the module isn't a library but rather provides a binary so that the build list is fully determined. But even then, I'd expect that the developer would need to test with module mode off before releasing, so that they make sure that the version dependencies that are actually in the go.mod file have been updated to depend on the new versions of the workspace modules that have been changed.

 

That said, it might be that I'm not understanding the proposal, or I might not understand lazy load. ;-)

Separately, rather than an init command, it might be better to have a sync command that adds new modules to the workspace file or removes them as needed, ideally reporting what it did. That might be more helpful over time than a one shot init command?

Are you suggesting that there's no init command, or that in addition to the init command there's a sync command that adds modules to the workspace file? Would the sync command do anything in addition to what `go mod editwork -directory [foo]` would do?

 

For multi-modules workspace, a flag to disable the workspace as outlined in the proposal likely covers the common cases of wanting to test against published versions prior to release, but it also seems commenting out the directories in the workspace file could cover additional cases (e.g., if you have a 4 module workspace, and want to test how the changes in 2 of the modules work against the published versions of the other 2 modules). Is that true, and if so, is it worth mentioning?

Added a sentence to the Rationale > Workflows > Switching between multiple configurations section.
 

It might also be useful to have something like ‘go work status’ (however it might be spelled) that gives an easily digestible summary. 

Added a sentence to the Future Work > Listing the module versions in the workspace section about this. I think the most important thing in the summary is exactly what's in the workspace, but were you thinking of other things?

You received this message because you are subscribed to the Google Groups "golang-tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/6AD3EEE7-E509-4FBC-8664-5E24833AD6DD%40gmail.com.

[Michael Matloob]

Yes, the module stuff is done out of the compiler. (that's how Bazel can

work even without modules).

Above in the thread we were discussing an option

of specifying all the modules in a directory with a directive like

`directory ./...` which isn't in the proposal as is, but is something

we've been thinking about for a v2. Would that help with your use case?

Then the ceremony/configuration reduces to (1) adding a single workspace

file with that directive to where your gopath area is (2) running your

commands in a directory contained in that directory and (3)

adding go.mod files. If that doesn't help, I'd be interested in why.

[Compl Yue]

Thanks Michael! For working on this.

Your proposal looks exactly like what I would wish for, I like https://github.com/golang/go/issues/44347 the most before seeing your proposal, but now yours is my most favorite.

I'm repeating something I've commented on a few GH issues (especially https://github.com/golang/go/issues/27542#issuecomment-820425053) here, but intend to show my support and in hope to be any help if possible.

I used to have GOPATH=/globally-cached/go-deps:/personal-tinkering/go-devs:/team-tinkering/go-pkgs, works perfectly well with organization-wide shared filesystems. And I hesitate to pickup further development of my rusted Go projects at this time, just because go.mod is the right way to go, but it still lacks sufficient support rival to my historical workflows with GOPATH like that.

I had been and still am missing Go ever since switched to Haskell  a couple of years ago, Go tooling is always the most pragmatic one to my experience, (but I need more powerful abstraction tools there plus the M:N scheduler of GHC RTS as well as Go offers). But w.r.t. the "project" thing, I realize that both Haskell build tools, namely Cabal and Stack, got it right beyond modules and packages. 

Situation wrt building tools are even messier and more confusing to outsiders in the Haskell ecosystem, but this stackoverflow answer may explain why "project" is important (as well as to Gophers): https://stackoverflow.com/a/49776281/6394508

Note: I would suggest you replace "package" as heard from a Haskeller, with "import-root" in your Gopher's mind for clearer understanding. I realize that Java might have started use "package" this way even before Go, that's apparently not wrong. But when you talk about "package" and "module" to Haskellers, or Python guys, the terminology can be a source of confusion.

> The *.cabal file is the package-level configuration. ... This configuration provides essential information about the package: dependencies, exported components (libraries, executables, test suites), and settings for the build process (preprocessors, custom Setup.hs). ...

> The stack.yaml file is the project-level configuration, which specifies a particular environment to make a build reproducible, pinning versions of compiler and dependencies. This is usually specified by a resolver (such as lts-11.4).

> stack.yaml is not redundant with package.yaml. package.yaml specifies what dependencies are needed. stack.yaml indicates one way to consistently resolve dependencies (specific package version, and/or where to get it from, for example: on Hackage, a remote repository, or a local directory).

> stack.yaml is most useful to developers: we don't want our build to suddenly break because a dependency upgraded while working on another project, hence we pin everything with stack.yaml. This file is less useful to users, who may have external constraints (typically, versions are already fixed by some distribution).

Release engineering/management concerns had driven a gang of Haskellers to have created [Stackage](https://www.stackage.org), it recruit curators to work on nightly snapshots of the central registry/repository of Haskell packages (https://hackage.haskell.org), making sure all packages are vetted (compiles, passing test suites, etc.) then regularly release such snapshots as [LTS Haskell](https://github.com/commercialhaskell/lts-haskell#readme). [Stackage Curators](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md) are great people and smart CI tools making all that happen.

I do think it's okay if all modules in a local `go.work` are engineered in a lock-step fashion, but for wider collaboration among upstream/downstream projects (inevitable for open source development), I don't think that solvable by a simple tool. More intensive processes/workflows with dedication of human forces might be inevitable, and https://www.snoyman.com/blog/2018/11/why-i-believe-stackage-succeeded may provide an illustration for what it will look like after done right.

Best regards,

Compl

[Paul Jolly]

Hi Michael,

Thanks very much for putting this together and presenting on the tools
call. Apologies for the slow response - house move!

My overall sense is that the move towards a project/workspace concept
file feels "right", so I very much welcome the direction of this
proposal.

FWIW my thinking about this proposal is likely skewed towards the
editor perspective and previous gohack experience. My mental model is
roughly:

* Developer wants to work on module M
* Developer creates a project/workspace for M (not intending any
LSP/proposal semantics for either project/workspace here)
* Developer decides they want to also work on dependency D1, D2 and D3
at the same time (related changes)
* Developer therefore "flips" D1, D2 and D3 from being dependencies
served from the module cache to being served from a local directory, a
la gohack
* ...

(the location of the dependencies D1, D2 and D3 could be opaque to the
user, unless they want that same location to be used in another
project/workspace - I'll skip over this detail).

In that context I view the go.work file potentially being the "thing"
that a developer adds to their editor as a project/workspace "root" in
those situations where it is necessary.

My understanding is that the primary desire to support multiple main
modules comes from wanting to emulate a GOPATH-like experience - is
that correct? Are there other motivating examples? My only question
(concern?) here would be, as more main modules get added to the
workspace, the versions being used in development and testing might
well drift further from the actual versions of dependencies that would
be resolved via an individual main module's go.mod file. Could this
perhaps become a regular source of frustration?

On a related point: I'm assuming gopls' super module will effectively
be deprecated in favour of go.work. How will go.mod related
errors/suggested fixes be presented in the case of a go.work
workspace?

As proposed the go.work file appears to be trying to solve two problems:

Scenario 1: "Multiple modules in the same repository that depend on
each other", in which case the go.work file is checked in. In this
case I would consider the go.work file a project/workspace file.
Scenario 2: all other scenarios, in which case the go.work file is not
checked in. In this case I would consider the go.work file serving
more of a go.mod.local purpose.

If, as proposed, one could place a go.work file at the root of one's
"GOPATH", how would nested go.work files work? Logically within the
editor one would expect the outermost go.work file to be "in control"
(on the basis that go.work file or its containing directory would have
been "added" to the editor) but I suspect cmd/go would obey the
innermost?

Somewhat of a follow-on to the GOPATH point. If a developer was to
place a go.work file at the root of their "GOPATH", my understanding
is that _all_ modules under that file must be part of that file
(indeed this appears to be supported by the fact that you considered a
./... directory directive value) and that to "opt out" of this, either
the user/their editor needs to set -workfile=off or create dummy
go.work files (that each must contain the module in question or
presumably ./...). My sense is that this will create too much churn
and headache and that instead the developer should have to explicitly
add (possibly via ./...) directories. This could be made to work with
editors using gopls: "editing" a file within a module that is not part
of the go.work file would generate a warning/error: cmd/go would work,
ignoring the go.work file (because it is not part of the workspace).

Another question about scenario 1; what if I wanted to locally develop
dependency D1 that is _not_ part of the repository that contains the
modules that depend on each other. I presumably end up modifying the
go.work file that has been committed at the repo root, and hence in
the same situation as today with needing to be careful that I don't
commit my changes? On that basis it feels like a go.work.local file
becomes necessary. On a related point, do you foresee people creating
a go.work file for every module they work on? I'm guessing not - i.e.
it would still work to add a module as a project/workspace in an
editor - in which case we still have the requirement for a
go.mod.local file (or equivalent).

> The go command will report errors for replacements of workspace modules that don't refer to the same directory as the workspace module. If any of those exist in a workspace module replacing another workspace module, the user will have to explicitly replace that workspace module with its path on disk.

It might just be me, but I struggled to follow this point :)

> Users might want to easily be able to test their modules with different configurations of dependencies

I'm not clear this would be something that happens via a go.work file.
Because presumably this sort of testing would also happen as part of
CI, where there would not be a go.work file in some/most cases? Seems
like this could be done within an existing go.mod?

> #39005 proposal: cmd/go: introduce a build configurations file

mvdan and I discussed this offline, and we weren't clear this could go
in the go.work file, principally because it won't be committed in some
scenarios. Does this sort of configuration more clearly belong in a
go.mod perhaps, because it's a statement of the supported build
configurations for packages within that module?

That ended up being rather long and rambling. Hopefully some of it makes sense.


Paul

> --
> You received this message because you are subscribed to the Google Groups "golang-tools" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-tools...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/CADVvUc1q3DsfoDFS8UW2rV5_ifZszjUMb7bOhAfN2cOvMvn9Rw%40mail.gmail.com.

[Michael Matloob]

On Tue, Apr 20, 2021 at 10:30 AM Compl Yue <comp...@gmail.com> wrote:

Thanks Michael! For working on this.

Your proposal looks exactly like what I would wish for, I like https://github.com/golang/go/issues/44347 the most before seeing your proposal, but now yours is my most favorite.

Thanks!!!

 

I'm repeating something I've commented on a few GH issues (especially https://github.com/golang/go/issues/27542#issuecomment-820425053) here, but intend to show my support and in hope to be any help if possible.

I used to have GOPATH=/globally-cached/go-deps:/personal-tinkering/go-devs:/team-tinkering/go-pkgs, works perfectly well with organization-wide shared filesystems. And I hesitate to pickup further development of my rusted Go projects at this time, just because go.mod is the right way to go, but it still lacks sufficient support rival to my historical workflows with GOPATH like that.

I had been and still am missing Go ever since switched to Haskell  a couple of years ago, Go tooling is always the most pragmatic one to my experience, (but I need more powerful abstraction tools there plus the M:N scheduler of GHC RTS as well as Go offers). But w.r.t. the "project" thing, I realize that both Haskell build tools, namely Cabal and Stack, got it right beyond modules and packages. 

Situation wrt building tools are even messier and more confusing to outsiders in the Haskell ecosystem, but this stackoverflow answer may explain why "project" is important (as well as to Gophers): https://stackoverflow.com/a/49776281/6394508

Note: I would suggest you replace "package" as heard from a Haskeller, with "import-root" in your Gopher's mind for clearer understanding. I realize that Java might have started use "package" this way even before Go, that's apparently not wrong. But when you talk about "package" and "module" to Haskellers, or Python guys, the terminology can be a source of confusion.

> The *.cabal file is the package-level configuration. ... This configuration provides essential information about the package: dependencies, exported components (libraries, executables, test suites), and settings for the build process (preprocessors, custom Setup.hs). ...

> The stack.yaml file is the project-level configuration, which specifies a particular environment to make a build reproducible, pinning versions of compiler and dependencies. This is usually specified by a resolver (such as lts-11.4).

> stack.yaml is not redundant with package.yaml. package.yaml specifies what dependencies are needed. stack.yaml indicates one way to consistently resolve dependencies (specific package version, and/or where to get it from, for example: on Hackage, a remote repository, or a local directory).

> stack.yaml is most useful to developers: we don't want our build to suddenly break because a dependency upgraded while working on another project, hence we pin everything with stack.yaml. This file is less useful to users, who may have external constraints (typically, versions are already fixed by some distribution).

Release engineering/management concerns had driven a gang of Haskellers to have created [Stackage](https://www.stackage.org), it recruit curators to work on nightly snapshots of the central registry/repository of Haskell packages (https://hackage.haskell.org), making sure all packages are vetted (compiles, passing test suites, etc.) then regularly release such snapshots as [LTS Haskell](https://github.com/commercialhaskell/lts-haskell#readme). [Stackage Curators](https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md) are great people and smart CI tools making all that happen.

I do think it's okay if all modules in a local `go.work` are engineered in a lock-step fashion, but for wider collaboration among upstream/downstream projects (inevitable for open source development), I don't think that solvable by a simple tool. More intensive processes/workflows with dedication of human forces might be inevitable, and https://www.snoyman.com/blog/2018/11/why-i-believe-stackage-succeeded may provide an illustration for what it will look like after done right.

I haven't had the chance to read the referenced in depth yet, but overall, I hope that the idea of workspaces can provide a good foundation for other tools outside the scope of the go command to make the developing and release process better: I reference that a bit in the doc in the future work section. And I hope it's flexible enough to be a fit for the development workflows of a wide variety of developers/organizations.

Thanks for the feedback!

 

Best regards,

Compl

On Thursday, April 15, 2021 at 10:30:18 AM UTC+8 mat...@golang.org wrote:

Hello!

I've put up a draft design for a new workspace mode in the go command to be used when working on multiple modules at the same time. It's here: https://go.googlesource.com/proposal/+/refs/heads/master/design/draft-workspace.md

I introduced it at the tools call earlier today, but didn't say anything that's not in the doc. (and the doc's gotten better editing than my remarks on the call!)

It's not quite ready for a proposal yet, but I wanted to see what y'all think about it, so please feel free to discuss and leave any comments here!

Thanks!

Michael

--
You received this message because you are subscribed to a topic in the Google Groups "golang-tools" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-tools/0HfA1HWaTJY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-tools/24b8869f-35e1-4df3-8cc6-4f2e2eb3d535n%40googlegroups.com.

[Michael Matloob]

Hello, I just wanted to update this thread that I've published the proposal on github here: https://github.com/golang/go/issues/45713

Feel free to comment there if you want to continue the conversation!

Thanks,

Michael

[Jan Mercl]

On Mon, Apr 26, 2021 at 7:34 PM Michael Matloob <mat...@golang.org> wrote:

> Hello, I just wanted to update this thread that I've published the proposal on github here: https://github.com/golang/go/issues/45713
>
> Feel free to comment there if you want to continue the conversation!

I don't have a github account so let me comment here. After thinking
about it for a while I came up with a minimalistic solution that so
far seems to serve my needs well enough with no need for a manually
written configuration file. Sharing it in case someone else may find
it useful - or the approach it's based on:
https://pkg.go.dev/modernc.org/gomod