GO Mod (modules) for dummies. Please help.

[sdd.d...@gmail.com]

Hi.

I have been using GO for about a year and I love the language and ideas behind the language. I am also a Java developer for many years, I switched from Delphi to Java 1, the new and exciting language from Sun (a bit like GO is now from Google).

 

In Java we have Maven and Gradle (sorry Ant) to make dependency hell more manageable so I understand the need for modules in Go. I have just installed GO 1.13 and thought I would convert an existing 'pet' project to use modules. It did NOT go well!

 

What I need is a dummies guide to the GO module so I can build good, reliable, standards compliant GO applications.

 

I needs to explain the new terminology in the context of a module, like 'vendor'.  Not just a one liner, I NEED to understand!

 

I know how to use Google but the quality of the articles I have read on this subject is minimal and just brushes the surface.

 

If I have a reasonably large and complex (pet) project with local packages in it. I like to split my project in to small units with 'namespaces' to keep it manageable. These are NOT reusable components until I decide they qualify and publish on Github.

• Why MUST I import them as if they are from github and then 'replace' them, and if I don’t 'MUST' then you have failed to explain this feature to me!
• My local packages are part of my application. They are, I agree still 'dependencies' but they are not 'DEPENDENCIES' that I need (or even want) to import from a repository. They are part of my project.
• What if I do not want to host my project on a GIT repo (Shock horror!).
• Why do all imports start with github.com. Is that a requirement, what is the rational for this.
• How does a 'import' resolve its 'reference'.
• Should I add the go.mod and go.sum files to my repository or should the developer who cloned my project have to do a go mod init (bummer!).

Can someone please explain, properly!

We must have Modules and Repositories (like Maven Central) for the 'Enterprise' to manage dependencies but what about 'keep it simple' for the rest of us (and for that matter more mature enterprise developers like myself).

 

Please help me get this understood. This is the sort of thing that can raise a language above the rest and I would really like that to happen. Go is brilliant…

 

Regards

 

Stuart

[Ian Davis]

Hi,

On Mon, 23 Sep 2019, at 11:04 AM, sdd.d...@gmail.com wrote:

If I have a reasonably large and complex (pet) project with local packages in it. I like to split my project in to small units with 'namespaces' to keep it manageable. These are NOT reusable components until I decide they qualify and publish on Github.

• Why MUST I import them as if they are from github and then 'replace' them, and if I don’t 'MUST' then you have failed to explain this feature to me!
  

Can you provide an example of what you are doing or a link to the documentation that is telling you to do this?

Every module in Go has a name and when you have modules in different locations, even just peer directories on your local machine, you need to write the mapping in go.mod between the name of the module and its location. For example, I have a non-public library that is used by another module on my machine. I have this in my go.mod to link the two together without needing to publish the library: replace ian.local/tools => ../tools

• My local packages are part of my application. They are, I agree still 'dependencies' but they are not 'DEPENDENCIES' that I need (or even want) to import from a repository. They are part of my project.
  

That's fine. Packages that are in subdirectories are usually simply part of the parent module. However you still need to name them so you can import them in your code. The pattern is that each package name is appended to the module name like a path, so in my example above a subpackage of tools would be referred to as ian.local/tools/subpackage

• What if I do not want to host my project on a GIT repo (Shock horror!).
  

The go tool also supports Bazaar, Fossil, Mercurial and Subversion. See https://golang.org/cmd/go/#hdr-Remote_import_paths

• Why do all imports start with github.com. Is that a requirement, what is the rational for this.
  

It is not a requirement. You are simply seeing the consequences of 90% of packages being hosted there. You may also commonly see packages hosted at golang.org, gopkg.in, bitbucket.org and many others. There are no restrictions and you can host on your own domain.

• How does a 'import' resolve its 'reference'.
  

The go tool uses go.mod to find out how to resolve a package name. Usually it simply maps directly to a remote repository (in which case go.mod will also contain some version information) but it could also map to another compatible package or to a location on your local machine.

• Should I add the go.mod and go.sum files to my repository or should the developer who cloned my project have to do a go mod init (bummer!).
  

  
  

You should do it. Give your module a name so people know what to use when importing it. As far as I know the only restriction is that the name must have a dot in it. By convention people name their modules with the repository name but that is not required. If you don't plan to share your code then you can make up a name like I did above: example.local works for instance.

Can someone please explain, properly!

I feel your pain! Modules are conceptually simple (a collection of related packages) but the interactions with code repositories, tools, build systems and user expectations are phenomenally complicated. The documentation is comprehensive and extensive but still misses some important "getting started" information. If you have specific requirements or problems with it then raising an issue for the Go team to track would be useful.

Ian

[Stuart Davies]

I am having real issues understanding what is going wrong.

When my project was self contained I thought I understood.
Now I am trying to package it up I am in trouble.
I had a project layout as follows (simplified)

webserver/config/config.go
webserver/template/template.go
webserver/servermain.go
webserver/go.mod

servermain.go is a sample project while I get the packages (config and template) right. It also serves as an example so I need to keep it in the project in git.

go.mod (contains)

  require github.com/mygit/webserver v0.0.0-20191018134507-cb1d694390ab // indirect
  replace github.com/mygit/webserver => ./

This all worked fine, I could update the code in config.go and it would be used in servermain.go without doing any downloads.

I commited ALL of this and started a new project with the plan to use config and template package from above.

New project layout as follows (simplified)

ws2/server2.go
go.mod (contains)

  require github.com/mygit/webserver v0.0.0-20191018134507-cb1d694390ab // indirect

I could not build this as it kept saying that the module was for servermain.go. I am a little vague hers as this was some time ago.

I gave up after a while and restructured the webserver as follows (adding example):

webserver/config/config.go
webserver/template/template.go
webserver/example/servermain.go
webserver/example/go.mod

From the 'example' dir I ran 'go mod init' and 'go mod tidy'
No problem, it all ran and my go.mod file has:

require github.com/mygit/webserver v0.0.0-20191018134507-cb1d694390ab // indirect

I realised that config and template had been downloaded in to GOPATH/pkg/mod...

If I make changes to config locally they are NOT seen in example/servermain.go.

So I set up the go.mod with a 'replace' but I cannot find a path that works.

replace github.com/mygit/webserver => ./

Fails with :

build command-line-arguments: cannot load github.com/mygit/webserver/config: module github.com/mygit/webserver@latest (v0.0.0-20191018134507-cb1d694390ab) found, but does not contain package github.com/mygit/webserver/config

I have been trying various paths. They either reload GOPATH/pkg/mod (I delete this before each run) and ignote any changes I make to config.go or they fail to find the dependency.

What is the correct way to proceed while developing my servermain.go.

What is the correct way to proceed when developing servers that import webserver (ws2/server2.go) and use config and template (without a replace in go.mod)

I am sure I am missing somthing minor. Could you please help.

Regards Stuart

[Dimas Prawira]

Hi, I have reply thread about Go mod. You can search in the mailist history.

Cheers

[Stuart Davies]

Thanks but I had alread seen your post. But I cannot get the replace to work.

I have given more detail below:

Project layout:

 ── webserver    
    ├── config
    │   ├── config.go
    ├── example
    │   ├── go.mod
    │   ├── go.sum
    │   ├── webserver.go
    ├── exec
    │   ├── exec.go
    │   └── exec_test.go

in webserver.go I import:

    "github.com/mygit/webserver/config"

    "github.com/mygit/webserver/largefile"

examples/go.mod contains:

    module webserver.go
    go 1.13

    require github.com/mygit/webserver v0.0.0-20191018134507-cb1d694390ab // indirect
    replace github.com/mygit/webserver => ./

GOPATH:

    /home/dev/go

    /home/dev/go/pkg is empty

From in the examples directory!

If go.mod contains:

replace github.com/mygit/webserver/config => ../webserver/config

replace github.com/mygit/webserver/exec => ../webserver/exec

This re-populates /home/dev/go/pkg/mod. Local changes do not take effect

If go.mod contains:

replace github.com/mygit/webserver => ../webserver

parsing ../webServerBase/go.mod: open /home/dev/golang/webserver/webserver/go.mod: no such file or directory

if go.mod contains:

replace github.com/mygit/webserver => ../../webserver

parsing ../go.mod: open /home/dev/golang/webserver/go.mod: no such file or directory

if go.mod contains:

replace github.com/mygit/webserver => /home/dev/golang/webserver

parsing ../go.mod: open /home/dev/golang/webServerBase/go.mod: no such file or directory

What do I need to add to example/go.mod to make it use the local pacakges?

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[kddavi...@gmail.com]

Hi Stuart,

(I'm not 100% confident, so take this with a grain of salt ;) )

In my experience, I have always found it best to have go.mod in the project's root directory (as well as only using one go.mod per repo). You could also use go mod vendor command, which will download all of the dependencies to a local "vendor" folder in the root of your main module, and rewrite the replace directives to point to those local paths (note you would need to use "go build -mod=vendor" to build).

Also note that the go tool prefers the latest "tagged" release when considering dependencies, so it may be helpful to tag a working version of your module (which shouldn't contain replace directives to local paths) with something like "v0.0.1-alpha" or similar. You can find more info by running "go help modules" command.

-K

[Stuart Davies]

-K Thanks for the comments. I am thinking I should separate the example from the modules and they should both have go.mod files.

I have to say that up until now I was 100% focused on go but I have spend the last couple of days focused on modules and it is very frustrating. 

I will see what happens.

Thanks 

Stuart 

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[Stuart Davies]

OK I think I have cracked it!

I needed a go.mod file in the root path. 

module github.com/mygit/webserver

go 1.13

This basically means that the go.mod file in examples can find a go.mod file at ../ 

replace github.com/mygit/webserver => ../

The clue was in the error message from earlier.

The only thing I missed was that when you do:

go mod init <name>

The <name> must be the module name 'github.com/mygit/webserver' not the go file name.

While in example the <name> is the application (main) name --> webserver.go.

I can see the logic for this but I do not think the distinction is clear in the documentation.

I think it's: 

If your going to code something that will be imported use the github.com/x/y for the name. If its an application that is importing then use the application name.

Please correct me if I am wrong.

Thanks every one for you contributions. I am back to coding :-) 

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[Stuart Davies]

Cannot make local changes. Dispite previous comments This is still a frustrating issue!

I have the following project structure:

 ── webserverbase    
    ├── config
    │   ├── config.go
    ├── example
    │   ├── go.mod  // Mod file 1

    │   ├── go.sum
    │   ├── webserver.go
    ├── exec
    │   ├── exec.go
    │   └── exec_test.

go
    ├── go.mod         // Mod file 2
    ├── go.sum

Mod File 1:
    module webserver.go
    go 1.13
    require github.com/mygit/webserverbase v0.1.0-alpha // indirect
    replace github.com/mygit/webserverbase/config v0.1.0-alpha => ../config

Mod File 2:
    module github.com/mygit/webserverbase
    go 1.13

At $GOPATH/pkg \mod\github.com\mygit\web!server!ba...@v0.1.0-alpha\ The whole project source branch

In the examples dir I can:
    go install webserver.go
I get an exe in:
    C:\Users\user\go\bin
   
If I change webserverbase\config\config.go NOTHING CHANGES! Dispite the replace in the go.mod file.

I need to make local changes to config before pushing to git. What am I dooing wrong!

I even introduced a syntax error in to config.go. It is IGNORED!

Please Please help. I cannot see where I am going wrong! It should not be this hard!

Stuart

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[t hepudds]

Hello Stuart,

I haven't digested everything here, but one comment is that all of the directives in go.mod like 'require' and 'replace' all deal with module paths, and not package import paths.

This 'replace' I think does nothing:

    replace github.com/mygit/webserverbase/config v0.1.0-alpha => ../config

because it is using a package import path and not a module path. The module path was defined in the other go.mod on the 'module' line as:

    

    module github.com/mygit/webserverbase

Backing up, I suspect part of what is tripping you up is the exact relationship between repositories vs. modules vs. packages vs. import paths.

Here is a summary of the relationships between those concepts which might help:

 * A _repository_ contains one or more Go _modules_ (usually exactly one module in the repository root).

 * Each module contains one or more Go _packages_.

 * Each package consists of one or more Go _source files_ that all reside in a _single directory_.

 * Go source code:

   * declares its own package with a `package foo` statement.

   * automatically has access to other Go source code in the same package.

   * imports code from another package via an _import path_ supplied in an import statement such as `import "github.com/some/repo/pkg1"`. The import path always starts with the module path of that package, regardless of whether that package is in the same module or a different module.

    

Also, I would like to repeat the advice that someone gave you earlier that it is best to avoid >1 module in a repo if you can, especially when starting out with modules, which sounds like the case here if I followed.

The most common and easiest approach is a single go.mod per repository, with the single go.mod file placed in the repository root, and using the repository name as the module path declared in the module line in the go.mod.

That avoids multiple subtleties...

One alternative you could consider is using a leading underscore for your examples directory, as shown for example here:

  https://github.com/loov/hrtime/tree/master/_example

From the go documentation:

  > Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

  

Regards,

thepudds

[Stuart Davies]

Hi thepudds

Thanks for the reply. Somtimes I think I have it and somtimes I most definitly do not.

I have been developing code for some time now, mostly in Java. I have used Mavan and Gradle and both give me issues with 'dependency hell'. I prefer Maven over Gradle any day because you can see what is going on!

Go Mod however has it's own flavor of dependency hell and I am not sure I have got it yet. I love the language but mod is driving me mad! Sorry, end of rant!

Ok this is my current understanding. Please correct me if I am wrong.

1) The main purpose of a go.mod file is to indicate where the dependencies are and their version number.
 
2) In my small project the 'module' is 'where the go.mod file is' (root of webserver).

3) A module contains packages in sub dirs (these are what we import).
    a) If it has no packages in sub dirs is it still a module?
    b) Import dependencies between packages in a module must use the full module path and the package name
    c) There should be no main() in a module

4) The directory 'example' is a module (it has a go.mod) file.
    a) It has NO packages of it's own.
    b) It has a dependency on the 'webserver' module which must have a go.mod file (as it does).
    c) The purpose of the 'example' mod file is to locate the module dependencies (not the packages)
    d) Strictly speaking this is NOT a module (nothing will depend on it) so I should either move it outside the module or get go mod to ignore it by prefixing with '_'
    e) It is where the main() is.
    f) Is there any reason it could not be a module and be a dependent of another module?

Modules are normally stored in git (or equivilant) but even if they are not they need a module name that includes the git path and a go.mod file.

My 'webserver' go.mod file was created with the name:
    github.com/mygit/webserver

My 'example' was created with the name:
    github.com/mygit/webserver/example
   
I have a sneeking feeling that if outside of 'webserver' it would be:
    github.com/mygit/example
Could it be:
    github.com/mygit/webserver/example
Or if renamed:
    github.com/mygit/webserver/_example

I am really not sure what the correct name is here. If not a dependency, does it matter?

Now for 'replace' this is where I get a bit vague.

Replace, changes a module's dependency path. Either changing the version number or the actual modules full path. E.g. => github.com/someothergit/webserver.

Replace can change a modules path to a local directory path.

a) A local path of ./ indicates that the dependency (module) is in the same 'module' (as with example currently)

b) A Local path of ../ indicates that the dependency (module) is one level up.

    Is this in the same module or one levelup from the go.mod file?

c) A Local path of ../foo indicates that the dependency (module) is one level up in the directory called 'foo'.

    In my case if 'foo' contained 'webserver' would the path be:
    ../foo/webserver
    Or
    ../foo

Are there any diagnostics for dependency resolution.

Is there anyway of logging what the actual resultant 'replace' finds (or not). It seems to fallback, ignoring the replace, if it cannot be found. This would be really informative.

Sorry for the log response but I am almost there. I just need some fine detail or clarity. If I am to convince my boss to go for Go I must understand!

Regards

Stuart

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[Volker Dobler]

Hi

I think one of the main struggles stems from "dependency" having two

technical meanings:

 a) If a package a imports a package b then a depends on b.

     This is a priori agnostic to versions.

     This type of dependency is expressed by a simple import "import/path/of/b"

     in a's source code.

 b) Some set of packages might depend on an other set of packages

     in a specific version.

     This is expressed by a require directive in a go.mod file.

Do not conflate these two.

One more point to takeaway: A module is a set of packages which are

developed together and released together with a single version number.

On Wednesday, 23 October 2019 21:09:51 UTC+2, Stuart Davies wrote:

Ok this is my current understanding. Please correct me if I am wrong.

1) The main purpose of a go.mod file is to indicate where the dependencies are and their version number.
 

Not exactly. There are two main purposes:

 - Specify the name of the module. As the name of the module is kinda

   prefix of all package import path living in this module this a _very_ important

   thing and must be done properly (especially for version >= 2.0.0 of a module).

 - List the direct and indirect dependencies and their (minimal) version.

Dependencies in a go.mod are modules. A module is a set of packages

with a common version (versioned together). Packages  are identified by their

import path.

The "where the dependencies are" is secondary. The "wehere" can be changed

by a replace directive. 

 

2) In my small project the 'module' is 'where the go.mod file is' (root of webserver).

You project is badly organized or I missread your last post.

The best advice is: Stop trying to get fancy until you feel comfortable

working with modules. So please: Put the go.mod in the toplevel

dir of your git repo an have just one go.mod file and thus just one

module. (It is perfectly fine to have tens of module in a git repo but

such a setup is not helpful while trying to understand the basics.)

 

3) A module contains packages in sub dirs (these are what we import).
    a) If it has no packages in sub dirs is it still a module?

No really. This is just plain nonsense. The right way to think of modules

is: A module groups packages. A module without packages is nothing

sensible. A set of packages versioned together is a module.

    b) Import dependencies between packages in a module must use the full module path and the package name

No. What you state here is not really wrong, it just is not right either.

It is kinda the other way around. The main concept is: Packages are

identifed by their import path. Package path lookup works by finding

the module and the package inside the module. If you have a package

in subdirectory foo/bar of your module xyz/wuz then its import path

is xyz/wuz/foo/bar.

(The package name is what you call the package in its first line in the

source code with the "package name" declaration

and this is of no concern for dependencies or importing packages).

    c) There should be no main() in a module

No. 100% false. You can have a lots of package main in a singel

module. Remember: "A module is a group of packages versioned

together." Whether these packages are "libraries" or "program"

does not matter at all. 

4) The directory 'example' is a module (it has a go.mod) file.

Yes

 

    a) It has NO packages of it's own.

Wrong. It contains a file webserver.go and this is the

package making up this module. 

    b) It has a dependency on the 'webserver' module which must have a go.mod file (as it does).

Well, true, but nonsensical.

 

    c) The purpose of the 'example' mod file is to locate the module dependencies (not the packages)

There is no purpose for a go.mod file in the example directory except.

Really. This is complete nonsense.

 

    d) Strictly speaking this is NOT a module (nothing will depend on it) so I should either move it outside the module or get go mod to ignore it by prefixing with '_'

Whether something depend on something has no influence on being a

module. A module is a group of packages versioned together. Nothing

less but especially nothing more.

Go code depends on packages (that's what you import). Packages which

work together and are versioned are grouped in a module. A module

has a version 

    e) It is where the main() is.

Location of package main and func main() has literally nothing to do

with modules.

 

    f) Is there any reason it could not be a module and be a dependent of another module?

If have no idea what you mean here.

Best advice: Stop going micro-module.

Technically each and every of your packages you ever develop

could live in its own module. But this is nonsense in real life

where you develop a program, a set of programs working togther

or a set of library packages working together and this "together"

is tested, versioned and released together under one version number.

 

Modules are normally stored in git (or equivilant) but even if they are not they need a module name that includes the git path and a go.mod file.

Technically no. Now you enter a different real here: How to

find the source code of a module. Formally you could name

your module x whatever you like, store it on disk wherever you like

and add a replace directive to each module which import a package

from x pointing to that disk location.

So basically a "Yes!" on this. Your module are named

    github.com/<youraccount>/<repo>

 

My 'webserver' go.mod file was created with the name:
    github.com/mygit/webserver

My 'example' was created with the name:
    github.com/mygit/webserver/example
   
I have a sneeking feeling that if outside of 'webserver' it would be:
    github.com/mygit/example
Could it be:
    github.com/mygit/webserver/example
Or if renamed:
    github.com/mygit/webserver/_example

I do not understand your structure well enough to comment on this.

 

I am really not sure what the correct name is here. If not a dependency, does it matter?

Probably you are conflating import dependency and version dependency.

On any case: Yes the module name does matter!

If you have a 

    module foo/bar/xyz

then the import path of each and every package in this module

start with foo/bar/xyz, e.g. foo/bar/xyz/wuz/kik for a package

in wuz/kik.

 

Now for 'replace' this is where I get a bit vague.

Replace, changes a module's dependency path. Either changing the version number or the actual modules full path. E.g. => github.com/someothergit/webserver.

Not really. It can change the version and/or where to find it.

There simply is no "module path". A module has a name

specified with the module directive in the go.mod. This name

cannot be changed with a replace directive.

 

Replace can change a modules path to a local directory path.

Correct if you thing of "module path" as "where to find the module".

 

a) A local path of ./ indicates that the dependency (module) is in the same 'module' (as with example currently)

Maybe. This is useless and never needed.

 

b) A Local path of ../ indicates that the dependency (module) is one level up.

    Is this in the same module or one levelup from the go.mod file?

Even more strange here. No specifiying that the source of a module lifes

one level up in the file system directory structure is plain nonsense. 

c) A Local path of ../foo indicates that the dependency (module) is one level up in the directory called 'foo'.

    In my case if 'foo' contained 'webserver' would the path be:
    ../foo/webserver
    Or
    ../foo

Maybe. At least this is no longer nonsensical.

Stop going clever with ../ and use an absolute path, with this you

will understand what is going on. Such a replace directive tell Go

where on disk to find the source code of that module. Thats all

to know.

 

Again: The best advice is: Use one go.mod at the source of your git repo

and treat everything in there as one module. This is perfectly fine for

your repo: You want to version your example and your server together!

Nobody ever will want your webserver code in version 1.2.3 but your

example in version 3.4.5!

Are there any diagnostics for dependency resolution.

Is there anyway of logging what the actual resultant 'replace' finds (or not). It seems to fallback, ignoring the replace, if it cannot be found. This would be really informative.

I doubt it fallback.

The problems in your code stems from not treating a set of

packages versioned together as a single module.

Your could would not need any replace directive.

V. 

[Stuart Davies]

Well I can see that I am not getting this. I will carry on coding and see if the understanding I have resolves itself.

Thanks for all you contributions.

On Monday, 23 September 2019 16:25:30 UTC+1, Stuart Davies wrote:

[Robert Engels]

Don’t be frustrated. The design could be better imo. I am assuming much of the complexity comes from trying to optimize the build across projects that share common modules. I think there are easier ways to accomplish this. Simple mappings to version labels would be easier imo. 

On Oct 25, 2019, at 9:14 AM, Stuart Davies <sdd.d...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a9c56752-3660-4c6b-bd9e-36887c7ff417%40googlegroups.com.