Host two distinct modules in one git repo?

[Everton Marques]

Multiple modules in single repo, currently a supported layout?

Module 1:

github.com/udhos/modhello/modlib/modlib.go

github.com/udhos/modhello/modlib/go.mod

Module 2:

github.com/udhos/modhello/modapp/main.go

github.com/udhos/modhello/modapp/go.mod

I am hitting this:

lab@ubu1:~/eraseme/modhello/modapp$ go list -m

go: finding github.com/udhos/modhello/modlib v1.0.0

go: github.com/udhos/modhello/mod...@v1.0.0: unknown revision modlib/v1.0.0

go: error loading module requirements

How do I publish version v1.0.0 only for modlib (but not for modapp) ? 

Testing with go1.11beta2.

Thanks,

Everton

[Paul Jolly]

> Multiple modules in single repo, currently a supported layout?

Absolutely. Here's a complete example:

$ mkdir go-modules-by-example-submodules
$ cd go-modules-by-example-submodules
$ git init -q
$ git remote add origin
https://github.com/myitcv/go-modules-by-example-submodules
$ go mod -init -module github.com/myitcv/go-modules-by-example-submodules
go: creating new go.mod: module
github.com/myitcv/go-modules-by-example-submodules
$ git add go.mod
$ git commit -q -am 'Initial commit'
$ git push -q
$ mkdir b
$ cd b
$ cat <<EOD >b.go
package b

const Name = "Gopher"
EOD
$ go mod -init -module github.com/myitcv/go-modules-by-example-submodules/b
go: creating new go.mod: module
github.com/myitcv/go-modules-by-example-submodules/b
$ go test
? github.com/myitcv/go-modules-by-example-submodules/b [no test files]
$ cd ..
$ git add b
$ git commit -q -am 'Add package b'
$ git push -q
$ git tag b/v0.1.1
$ git push -q origin b/v0.1.1
$ mkdir a
$ cd a
$ cat <<EOD >.gitignore
/a
EOD
$ cat <<EOD >a.go
package main

import (
"github.com/myitcv/go-modules-by-example-submodules/b"
"fmt"
)

const Name = b.Name

func main() {
fmt.Println(Name)
}
EOD
$ go mod -init -module github.com/myitcv/go-modules-by-example-submodules/a
go: creating new go.mod: module
github.com/myitcv/go-modules-by-example-submodules/a
$ go build
go: finding github.com/myitcv/go-modules-by-example-submodules/b v0.1.1
go: downloading github.com/myitcv/go-modules-by-example-submodules/b v0.1.1
$ ./a
Gopher
$ cat go.mod
module github.com/myitcv/go-modules-by-example-submodules/a

require github.com/myitcv/go-modules-by-example-submodules/b v0.1.1
$ cd ..
$ git add a
$ git commit -q -am 'Add package a'
$ git push -q
$ git tag a/v1.0.0
$ git push -q origin a/v1.0.0

[Everton Marques]

Em quarta-feira, 25 de julho de 2018 13:12:01 UTC-3, Paul Jolly escreveu:

> Multiple modules in single repo, currently a supported layout?

Absolutely. Here's a complete example:

This did the trick: git tag b/v0.1.1 

Awesome!

Thanks a lot.

Everton

[Everton Marques]

Em quarta-feira, 25 de julho de 2018 13:12:01 UTC-3, Paul Jolly escreveu:

> Multiple modules in single repo, currently a supported layout?

Absolutely. Here's a complete example:

Paul,

I see from your example that one can add a tag as "package/vN.N.N" to publish a new version for the package/sub-module.

Is it possible to host multiple packages within a module?

I mean:

repo1/module1/package1

repo1/module1/package2

repo1/module2/package1

Is there a recipe to publish newer versions for repo1/module1 while importing its individual packages?

Thanks,

Everton

[Paul Jolly]

Everton - a module is a collection of related Go packages. So, in short, yes :)

In the example I gave above, it just so happens that each module
contains just one module-top-level package. But that is by no means a
restriction, just a coincidence of this example.

So the tag is actually for the sub- _module_, not the package.

Hope that helps.

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Everton Marques]

Em quinta-feira, 26 de julho de 2018 11:32:12 UTC-3, Paul Jolly escreveu:

Everton - a module is a collection of related Go packages. So, in short, yes :)

In the example I gave above, it just so happens that each module
contains just one module-top-level package. But that is by no means a
restriction, just a coincidence of this example.

So the tag is actually for the sub- _module_, not the package.

Paul,

Happy to hear, however I'm facing a difficulty.

Below, I show what I hit when importing package "github.com/udhos/modhello/modlib/lib".

Repo has tag modlib/v1.0.0.

Sorry to abuse, but can you spot what I am doing wrong?

modapp$ go list -m all

github.com/udhos/modhello/modapp

github.com/udhos/modhello/modlib v1.0.0

modapp$ go mod -sync

go: downloading github.com/udhos/modhello/modlib/lib v1.0.0

go: finding github.com/udhos/modhello latest

go: import "github.com/udhos/modhello/modapp" ->

        import "github.com/udhos/modhello/modlib/lib": cannot find module providing package github.com/udhos/modhello/modlib/lib

$ cat modlib/go.mod

module github.com/udhos/modhello/modlib

$ cat modlib/lib/modlib.go

package lib

func Sum(a, b int) int {

        return a + b

}

$ cat modapp/go.mod

module github.com/udhos/modhello/modapp

require github.com/udhos/modhello/modlib v1.0.0

$ cat modapp/main.go

package main

import (

        "log"

        "github.com/udhos/modhello/modlib/lib"

)

func main() {

        run(1, 2)

}

func run(a, b int) {

        log.Printf("Sum(%d,%d) = %d", a, b, lib.Sum(a, b))

} 

[Paul Jolly]

Just a quick check: what version of git are you using?

[Everton Marques]

Em quinta-feira, 26 de julho de 2018 12:05:18 UTC-3, Paul Jolly escreveu:

Just a quick check: what version of git are you using? 

$ git --version

git version 2.17.1

$ go version

go version go1.11beta2 linux/amd64

 

[wcole...@gmail.com]

Hi Paul,

Does this imply that, to test local changes in the inter-dependent sub modules you have to commit each change and push (much less tag) to be able to test the changes or have them reflected in local builds?

This is the behavior I'm seeing using beta2 and converting a repository that has a couple of separate but related/dependent packages. The repo is still in my standard GOPATH. When I try to build one of the sub modules that is dependent on a peer module it downloads an older version of the repository and then complains about "ambiguous import" listing the local GOPATH location and the sub module from the downloaded and cached sub module in $GOPATH/src/mod/... 

Also, should each sub module have its own require statements or should they all be in the root go.mod file? 

Thank you.

[Cole Davis]

Since I posted this question earlier today I've actually come to a better understanding of this. I was a bit confused on how "packages" fit into the "module" relationship. A couple things I've found that appear to be true:

- not every package in a module needs to be its own module unless you intend to version it separately from the root module

- the root module need not have go code in the root as long as it has packages in its tree

- the idea to point your module at a local directory while developing is via the -replace flag/go.mod entry

So, this all solved part of my issue but I've run into something else I don't know if it's a bug or yet more I don't understand. I have a repository that is now a module that contains multiple packages of shared utility. They all build/test fine everything looks just dandy. But I haven't committed the changes or pushed them anywhere. I wanted to test the changes against a service in another repository that imports it. I converted the service to a module in its repository root. The layout of the project has the main service in a structure such as

repo-root/

    go.mod

    cmd/

        service/

go.mod in this case has all of the dependencies in require and vgo/go1.11beta2 seems happy to build against old cached code. When I use replace it parses it fine, it points to the correct directory, but appears to be ignored.

If I instead add a go.mod to service/ and add the replace there it immediately reflects the desired code but I would have to move my requires into that go.mod it seems because the one in root no longer applies (even though the same imports are reflected by the roots requires and those modules can be found in the GOPATH anyway).

Is it intentional that replace is not respected as a root of a module...there are other packages in this repository for the service. Would I have to treat service/ as a module on its own to get that to work? That doesn't seem right.

[Cole Davis]

Nesting a module in cmd/service allows the correct code to be used but confuses the tool. The layout of the project was one that was heralded prior to vgo I believe so I don't know if they are incompatible.

Here's the structure in more detail

$GOPATH/src/path/to/shared/

    a/

    b/

    c/

    go.mod

$GOPATH/src/path/to/service/

    cmd/

        service/

            {...imports "path/to/service/pkg/d", "path/to/shared"...}

    pkg/

        d/

            {...imports "grpc"...}

    go.mod

Prior to switching to vgo/go1.11beta2 we used dep. From the root of the service I could do `go build -x -a -v -o ./service ./cmd/service` and it would build fine. Using modules I'm seeing a couple issues. It seems like the modules are competing with each other. If I use `go mod -sync` in the cmd/service module directory it has an ambiguous import error with itself. It doesn't import itself so I'm not sure where that is coming from. If I manually add the requires that should let it compile I still have to have some requires in the root so that the pkg/d package can build. It just so happens there is some overlap in their imports so it seems redundant to have it in both modules (root and cmd/service) not to mention running `go mod -sync` from the root has all sorts of unnecessary indirect requires added some of them only useful in cmd/service but it's ignore that it has a go.mod file apparently. All that to say if I run the build command that used to work from the root too it has the same ambiguous import error.

Looking at the output it appears that it's attempting to download the cmd/service package from remote instead of building it from the local source. 

Do the structure of projects need to change to accommodate go modules and working with local development versions of dependencies?

[Everton Marques]

I have filled an issue: https://github.com/golang/go/issues/26695

Then I found out that the issue does not affect a brand new Linux user on the same system.

So I guess the failure is related to some Go caching stored on the homedir of my original test user.

Everton

[Paul Jolly]

Everton - apologies, things got rather busy and this thread dropped
off my radar.

Is it possible for you to test with the latest build of go from tip?

There have been a number of fixes since beta3.

Thanks

[Paul Jolly]

Cole - apologies, things got busy in the last few weeks so this thread
dropped off my radar.

You've raised a number of questions in this thread; how many of these
remain since the release of beta3? Or indeed the fixes that have hit
go tip since beta3?

Happy to work through anything that's outstanding.

Thanks,


Paul

[Paul Jolly]

And incidentally, #modules on Gophers Slack will likely be a better
place to work things through; it's got a great number of people
helping/responding.

[Sam Whited]

On Wed, Aug 8, 2018, at 11:29, Paul Jolly wrote:
> And incidentally, #modules on Gophers Slack will likely be a better
> place to work things through; it's got a great number of people
> helping/responding.

This thread is of great importance to a lot of people in the community, and the information you can get from it is very important. Please do not move it to a closed system like Slack. Not everyone can get a Slack account to participate, or even read any discussion that happens there. I for one would like to continue being able to see this discussion.

—Sam

[Paul Jolly]

Hi Sam,

> This thread is of great importance to a lot of people in the community, and the information you can get from it is very important. Please do not move it to a closed system like Slack. Not everyone can get a Slack account to participate, or even read any discussion that happens there. I for one would like to continue being able to see this discussion.

This is a very fair point.

The thinking behind the Slack suggestion was:

* it's easier to go to and fro on Q&A when diagnosing issues,
uncovering facts etc.
* there are many people actively answering a large number of
module-related questions on a day-to-day basis

But it is a poor medium to digest/synthesise, I agree.

My suggestion would be that, in cases such as this thread, any
answers/insights derived from Slack be reported back to the go-nuts
thread, for the benefit of anyone else following along. This also has
the benefit of cutting out any noise along the road to those
answers/insights.

Best,


Paul

[Sam Whited]

On Wed, Aug 8, 2018, at 11:46, Paul Jolly wrote:
> My suggestion would be that, in cases such as this thread, any
> answers/insights derived from Slack be reported back to the go-nuts
> thread, for the benefit of anyone else following along. This also has
> the benefit of cutting out any noise along the road to those
> answers/insights.

This still doesn't allow people who may have something to contribute to participate. You're doing yourself a disservice by limiting the discussion to a smaller audience.

Chat in general isn't a good medium for this sort of things because you limit responses primarily to those in the same couple of timezones and who happen to not be busy, etc. and Slack is even worse by virtue of being impossible to use for a large number of people (for accessibility or legal reasons).

—Sam

[Everton Marques]

Paul,

Since the issue affected only a specific Linux user running beta2, I suspected it was a caching issuen then used "go clean -modcache" (as suggested in https://github.com/golang/go/issues/26695) to fix it. And it worked.

Unfortunately I don't know how to reproduce the caching (?) issue. I guess I should have saved the user's homedir for reference before running "go clean -modcache".

I can no longer see the issue in beta2 or beta3... AFAICT modules are working like a charm.

Everton

[Paul Jolly]

I can no longer see the issue in beta2 or beta3... AFAICT modules are working like a charm.

Great!