Better dependency management in go

[Shirshendu Bhowmick]

Hi All,

I create a github issue to suggest a proposal for making go dependency managment better.

As people there suggested that this group is probably the best place to discuss this, I am posting the exact same post here:

Even after the introduction of Go modules, the dependency management is complex and not so developer friendly. For beginners it takes a good amount of time to understand what's going on behind the scene.

Some of this may be because of lack of well written documentation about dependency management.

• There are some confusing syntax, for example
  github.com/myorganzation/mypackage/pkg

  This url results in 404 in browsers but somehow go resolves it, so it seems like depending on hosting providers such as GitHub, BitBucket etc go have different mechanisms of resolving the URL.

  Which is somewhat described in here

• Upgrading to major version with a suffix like vX

• Though i am not sure about this but i didn't find any way by which i can tell that this indirect dependency x is from the dependency y, just by looking into go.sum or go.mod files.

• Error messages not being so helpful
  If i try to go get a package in a directory which is not a module i get an error message which is not so helpful for beginners

go get github.com/gofiber/fiber/v2 cannot find package "github.com/gofiber/fiber/v2" in any of: /usr/local/go/src/github.com/gofiber/fiber/v2 (from $GOROOT) /Users/shirshendubhowmick/go/src/github.com/gofiber/fiber/v2 (from $GOPATH)

• There is no easy way of adding dev only dependencies.

Gophers please do let me know your thoughts about this, also request you to think about these issues from a beginners perspective especially if someone is coming from JavaScript or Python background.

I feel like there is a steep learning curve for go dependency management, which can be made easy with little changes in go and it's documentation.

Edit: Adding some suggestion to deal with the problems I mentioned above
Some high level suggestion to deal with the current problems

• Problem 1: The current way of downloading, using & maintaining a package

  I like the idea of not having a central registry like npm or pip. However using repo URLs (that is also some modified URL) everywhere in the codebase to import the package doesn't seem to be the best way.

  Instead what we can do is use git URLs git+ssh://g...@github.com/myorganization/mypackage (HTTPS url works too), only at one place, i.e. our dependency file (currently go.mod)
  With this, we can also refer to a particular branch, tag or commit.

  Now our dependency file (currently go.mod) will have a mapping of module name and its URL to create an alias for the module, for example
  mypackage git+ssh://g...@github.com/myorganization/mypackage

  Everywhere in the code base a consumer will use the alias name instead of a URL to import the package, for example

  import "mypackage"

  Now how do we know where the module is located inside the repo ? Right now we add a go.mod file in every module root directory. With this change maybe we can add a single file in the repo root which will tell where the modules are located relative to the repo root. I guess this will give more flexibility to both the module developer and consumer.

  For updating version, we can either do it manually by changing the URL or maybe via some tool like go update mypackage

• Problem 2: Improving the error messages
  This might be a simpler problem to solve compared to the above one. There is no specific solution to this. I think the best way is to run an audit to figure out point of failures while working with modules in go. And try to have as much meaningful error message as possible with some detailed log.

This is very high level solution proposal, happy to discuss more on it and also pros & cons, gotchas, bottlenecks etc.

Original github issue link:
https://github.com/golang/go/issues/41510

[Axel Wagner]

Hi,

On Tue, Sep 22, 2020 at 12:01 AM Shirshendu Bhowmick <shirshendu...@gmail.com> wrote:

There are some confusing syntax, for example
github.com/myorganzation/mypackage/pkg
This url results in 404 in browsers but somehow go resolves it, so it seems like depending on hosting providers such as GitHub, BitBucket etc go have different mechanisms of resolving the URL.
Which is somewhat described in here

I understand that there is lots of third-party material out there calling import-paths "URLs", but they aren't. For one, they don't have a scheme. I know that this kind of proves your point, that the situation might be confusing for beginners, but it's an important distinction, IMO, specifically to avoid this kind of problem. 

• Though i am not sure about this but i didn't find any way by which i can tell that this indirect dependency x is from the dependency y, just by looking into go.sum or go.mod files.

`go mod why` can tell you. I don't feel this is a reasonable expectation to make it visible in the dependency files itself. It would require every module to ship a graph-representation of its dependency graph, for the sole benefit of not having to invoke the go tool - and even then, you'll probably still want to do that, so you don't have to mentally walk the graph yourself.

• Error messages not being so helpful
  If i try to go get a package in a directory which is not a module i get an error message which is not so helpful for beginners

go get github.com/gofiber/fiber/v2 cannot find package "github.com/gofiber/fiber/v2" in any of: /usr/local/go/src/github.com/gofiber/fiber/v2 (from $GOROOT) /Users/shirshendubhowmick/go/src/github.com/gofiber/fiber/v2 (from $GOPATH)

I think this particular example might get simpler in the future, when GO111MODULE=on becomes the default, because the go tool won't look for dependencies in GOPATH. In general, I don't think error messages can be improved holistically - changing the approach will just lead to new, confusing error messages. I think error messages specifically are an area where you *have* to look at specific examples and fix them one by one.

Now our dependency file (currently go.mod) will have a mapping of module name and its URL to create an alias for the module, for example 

mypackage git+ssh://g...@github.com/myorganization/mypackage

 At the end of the day, the Go compiler requires unambiguous unique package names. How would you address the case where you and a dependency of yours would choose the same alias for different packages or different aliases for the same package? If an alias has to be changed to resolve such a conflict, how do you edit the code of your dependency to reflect that? Wouldn't you, in the end, need to manually maintain a list of all your transitive dependencies to give names to things?

That's the primary reason for putting a domain-name into the import-path. It allows you to piggyback on an existing system to assign unambiguous names to things. It is pretty central to having no centralized package repository - usually, this central repository provides the uniqueness required by the language. We have to replace it with something. Putting a domain-name in there is a simple solution that also has the side-effect of allowing us to layer automatic package discovery on top of it.

• Everywhere in the code base a consumer will use the alias name instead of a URL to import the package, for example

  import "mypackage"

  Now how do we know where the module is located inside the repo ? Right now we add a go.mod file in every module root directory. With this change maybe we can add a single file in the repo root which will tell where the modules are located relative to the repo root. I guess this will give more flexibility to both the module developer and consumer.

  For updating version, we can either do it manually by changing the URL or maybe via some tool like go update mypackage

• Problem 2: Improving the error messages
  This might be a simpler problem to solve compared to the above one. There is no specific solution to this. I think the best way is to run an audit to figure out point of failures while working with modules in go. And try to have as much meaningful error message as possible with some detailed log.

This is very high level solution proposal, happy to discuss more on it and also pros & cons, gotchas, bottlenecks etc.

Original github issue link:
https://github.com/golang/go/issues/41510

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cdc0ff54-65dc-4ee3-a18e-b3e88c450c9dn%40googlegroups.com.

[Vladimir Varankin]

Hey!

Several months ago I opened the proposal for extending the systax of go.mod in order to allow to specify/overwrite the source URL for a dependency (https://github.com/golang/go/issues/39536). It feels it could also solve what you described in "Problem 1", although the motivation behind the proposal is a bit different.

Let me know what you think.