go get awkwardness with packages outside of GOPATH

152 views
Skip to first unread message

Florian Uekermann

unread,
Jul 19, 2017, 2:45:00 PM7/19/17
to golang-nuts
Hi everyone,

I never bothered too much with the details of "go get" until the issue recently came up in a conversation with another developer who was also annoyed by the clumsiness of the go tool in practice. I often work with programs that are not in a GOPATH, but import some non-local packages. I find the behavior a bit strange. Before opening a bug I wanted to ask here if someone can shed some light on why things are the way they are.

In the following I'll assume that the directory $X contains the source of a package ("main" in my case) which resides outside the GOPATH, but imports some go get-able packages (like "golang.org/x/net/websocket"). If the $X is the current directory it may be omitted when using go get. I may be mistaken, but the options at the moment are pretty much this:

go get $X:
-Downloads missing dependencies
-Does not update sources of dependencies
-Executes go install $X
This is what I used to use ("go get; go build -i -o out && ./out"). This actually works fine. But it will return 1 (error), due to "go install $X" failing.

go get -d &X:
-Downloads missing dependencies
-Does not update source of dependency
I guess this is what I should have used instead of "go get". Instal

go get -u $X:
-Downloads missing dependencies
-Updates sources of existing dependencies
-Executes go install $X
This works great inside a GOPATH, but outside it is utterly useless when used on a package outside the GOPATH, since it fails immediately with "unrecognized import path ...". "go get -u -d" has the same problem.

So that leaves me wondering. How do I update the dependencies of a program whose code resides outside of a GOPATH? "go get -u ..." runs way to long if you have a lot of stuff in GOPATH.

Did I misunderstand something? Does anyone have a solution for this? If not, here are three proposals in order of preference (not mutually exclusive):
1. Change go get $X: Skip the "go install $X" if $X is outside the GOPATH and just install the dependencies.
2. Change go get -u $X: Just skip the "unrecognized import path" issue and continue updating the dependencies (I feel like this should make "go get -u -d $X" work for me).
3. Don't change "go get" at all, but add the -u flag to "go build". Then "go get -d && go build -u -i" would work great.

I know that none of these exactly fit the original idea of the "get" or "build" command perfectly, but this is a real world problem in need of a solution.

tldr: Please let me update dependencies of packages that are in arbitrary directories.

Best regards,
Florian

Wojciech S. Czarnecki

unread,
Jul 19, 2017, 3:59:56 PM7/19/17
to golan...@googlegroups.com
On Wed, 19 Jul 2017 11:45:00 -0700 (PDT)
"'Florian Uekermann' via golang-nuts" <golan...@googlegroups.com> wrote:

> tldr: Please let me update dependencies of packages that are in arbitrary
> directories.

1. You may add your current "arbitrary" to the local environment GOPATH
as in
export GOPATH="$GOPATH:/arbitrary/src"

(Go searches each directory listed in GOPATH to find source code, but new
packages are always downloaded into the first directory in the list.)

2. You may use some third party package management:
https://getgb.io/
https://github.com/bumptech/glide

Hope this helps.

> Best regards,
> Florian
>

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE

Jan Mercl

unread,
Jul 19, 2017, 4:09:23 PM7/19/17
to Florian Uekermann, golang-nuts
On Wed, Jul 19, 2017 at 8:45 PM 'Florian Uekermann' via golang-nuts <golan...@googlegroups.com> wrote:

> tldr: Please let me update dependencies of packages that are in arbitrary directories.

Dependencies are derived from import paths. Import paths are not complete paths. They are transformed to complete paths by searching every part in $GOPATH as in filepath.Join(part, "src", importPath).

How do you propose to update dependencies outside a $GOPATH? How should be the import paths transformed to complete paths?



--

-j

flo...@infobaleen.com

unread,
Jul 19, 2017, 4:18:14 PM7/19/17
to golang-nuts
I always forget that there is no edit button... sorry for not proofreading thoroughly before posting.

Florian Uekermann

unread,
Jul 20, 2017, 6:08:59 AM7/20/17
to golang-nuts
Thanks for the suggestions. I may not have explained the setup well enough. I don't think your suggestions work if you just have your code in a folder without any additional structure (no src directory).
But your first suggestion prompted my to play around with having the package in a GOPATH and I realized that my problem is much less specific than I thought, since go get -u also does not work with local packages (no vc or no git remote origin). So I guess I jumped to a much too specific conclusion and problem statement.

I should restate the issue as: The go tool does not provide a way to update dependencies of packages that are not go get-able. Inside or outside GOPATH has nothing to do with it.

Thanks,
Florian

Florian Uekermann

unread,
Jul 20, 2017, 6:19:41 AM7/20/17
to golang-nuts, florian....@googlemail.com
See my previous answer to ohir for a less convoluted and less specific problem statement. My problem does not seem to be related to being inside or outside a GOPATH.
 
Dependencies are derived from import paths. Import paths are not complete paths. They are transformed to complete paths by searching every part in $GOPATH as in filepath.Join(part, "src", importPath).

How do you propose to update dependencies outside a $GOPATH? How should be the import paths transformed to complete paths?

Thanks for your answer.
The import path transformation should work same way it works right now. "go get" works perfectly fine outside a GOPATH until the install step (which you can prevent using -d) or if you use -u. I believe it just uses the first "part" of GOPATH (or $HOME/go as of 1.8 if GOPATH is not defined). I fail to see the problem. Could you elaborate if I misunderstood you?

Best regards,
Florian

Wojciech S. Czarnecki

unread,
Jul 20, 2017, 9:25:03 AM7/20/17
to golan...@googlegroups.com
On Thu, 20 Jul 2017 03:08:58 -0700 (PDT)
"'Florian Uekermann' via golang-nuts" <golan...@googlegroups.com> wrote:

> The go tool does not provide a way to update
> dependencies of packages that are not go get-able.

Yes it is. Go tooling is based on and has a convention to follow. Hence there
is no support for "place pieces at will" scenarios.

Once you broke away from go convention you're pretty much on your own.
Eg. you may consider a makefile with something like "updatedeps" target
or you may vendor your dependencies at some mirror place. Note, that go
tooling does *not* allow symlinks!

I'd suggest keeping to go tool expectations though. Third party pm-s and
home baked scripts will not survive enough long to be on pair with evolving
genuine tool.

>
> Thanks,
> Florian

Hope this helps,

Simon Ritchie

unread,
Jul 21, 2017, 4:35:33 AM7/21/17
to golang-nuts
Just a simple clarification.  Your GOPATH variable can contain many directories, as already pointed out.  If so, go get will download packages into the FIRST directory in the path.

One use of this is to have your own go projects in separate directories with a single shared set of packages loaded using go get.  To work on project A, you set GOPATH to contain the shared directory, followed by the project directory for A.

However, if your GOPATH contains many directories, some IDEs don't handle the situation very well and are defeated by this approach.   If you like to use an IDE, you should experiment to see how well it manages.

Regards

Simon
Reply all
Reply to author
Forward
0 new messages