GOPATH for a project written in several languages (for example Go and Python)?

[Nicolas Grilly]

What is the recommended way to organize the source tree of a project written in several programming languages (for example Go and Python)?

Ideally, we would want to keep everything in the same git repository because 1) it makes easier to keep the Python and Go logic in sync and 2) some assets are shared by the Python and Go code.

One obvious way is to checkout the project source in some arbitrary directory like $HOME/projects/myproject, create a subdirectory go containing all the Go code, and make it accessible in $GOPATH with ln -s $HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks like a dead-end because symbolic links were declared "not supported" many times on golang-nuts and golang-dev.

Another way is to checkout the full project source (Go and Python) in $GOPATH/src/myproject and import "myproject/go" instead of "myproject". It's perfect and idiomatic from Go's point of view. But from Python's point of view, it's weird to be forced to live in $GOPATH. Go requires the project source to live in a directory src under $GOPATH. Now let's imagine that Python would require the project source to be rooted in a directory lib (for example) under its own $PYPATH. That would make Go's and Python's requirements contradictory and impossible to satisfy at the same time.

Another way is to create a Go workspace directly in the project source tree and add it to the $GOPATH. This implies myproject would have its own src/bin/pkg directories and this approach was explicitly not recommended on this mailing list in the past (https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).

I'm not sure how to solve this in a simple and clean way. What is your experience solving this issue in your own projects?

Nicolas

[Kyle Lemons]

Since Go is almost always the primary language for such projects of mine, I just make all of the source live in $GOPATH/src/github.com/kylelemons/project.  The Go code imports itself as github.com/kylelemons/project/whatever and the rest of the code doesn't really care where it lives.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Nicolas Grilly]

On Thu, Aug 15, 2013 at 6:51 AM, Kyle Lemons <kev...@google.com> wrote:

Since Go is almost always the primary language for such projects of mine, I just make all of the source live in $GOPATH/src/github.com/kylelemons/project.  The Go code imports itself as github.com/kylelemons/project/whatever and the rest of the code doesn't really care where it lives.

Yes, that's a perfect layout when the project main language is Go. But what to do when Go is not (yet) the main language?

[frou]

The symlink approach does work, doesn't it? If it stops working then you'll know about it and can find an alternative.

[Nicolas Grilly]

On Thursday, August 15, 2013 2:21:21 PM UTC+2, frou wrote:

The symlink approach does work, doesn't it? If it stops working then you'll know about it and can find an alternative.

No, it does not.

In my experience, many things don't work with symlinks:

- go build/fmt/test myproject/... does not find packages rooted under the symlinked directory (this is because the related code in the Go standard library and tools uses filepath.Walk which does not follow symbolic links)

- Packages rooted under the symlinked directory do not appear in godoc

- go install myproject does not know in which directory to install the resulting binary

- etc.

And according to some of Go authors on this mailing list, symlinks are avoided by design because their usage solves some issues (like mine) but also brings a lot of problems (like the possibility of creating circular references and having several different paths leading to the same file, none of them being canonical).

But maybe I'm missing something. Do you use symlinks in your own projects, without the problems I listed above?

[Jeremy Wall]

You can have multiple GOPATHs. Why not just add an entry for the path where your go code in the larger project lives?

--

[Nicolas Grilly]

On Thursday, August 15, 2013 3:55:47 PM UTC+2, Jeremy Wall wrote:

You can have multiple GOPATHs. Why not just add an entry for the path where your go code in the larger project lives?

Just to make sure I understand your suggestion, let's say the Go files of my project are in github.com/ngrilly/myproject/go/*.go. What would you add to GOPATH?

[frou]

go {build,install,fmt,test} proj/a/b

work for me where proj is a symlink in GOPATH to outside of GOPATH.

Indeed those packages don't show up in godoc.

I may be missing something myself. I have only experimented with one such multi-language project and haven't read the ML discussions you mention.

[Jeremy Wall]

Well first you'd have to do some rearranging. Add a gopath like directory with a src/ and pkg/ directory in your repo. github.com/ngrilly/myproject/go/{src,pkg} for instance

Then move the *.go files into a reasonable directory in the src/ directory. .../go/src/myproject/*.go for instance.

Then in your build script which you will need anyway since this is a multi language project add e.g. github.com/ngrilly/myproject/go/src/ to your GOPATH so all go builds of myproject/... will work.

This will work for all your developers and packagers. It will not work for making the the go code go gettable but this is a multilanguage project where Go is not the primarly language so that is likely not really a goal in the first place.

if go gettability *is* a goal then you should probably have the go code in a separate repo instead.

[Nicolas Grilly]

On Thursday, August 15, 2013 4:26:32 PM UTC+2, frou wrote:

go {build,install,fmt,test} proj/a/b

work for me where proj is a symlink in GOPATH to outside of GOPATH.

Indeed those packages don't show up in godoc.

I may be missing something myself. I have only experimented with one such multi-language project and haven't read the ML discussions you mention.

Yes, some things work, some don't.

Works:

- go {build,install,fmt,test,doc} proj/a/b

- go install proj/a/b correctly installs the produced binaries in $GOPATH/bin and $GOPATH/pkg (contrary to what I wrote in my previous message in this thread)

- cd $GOPATH/src && go {build,install,fmt,test} ./proj/...

Doesn't work:

- godoc -http=:6060

- go {build,install,fmt,test} proj/... ("proj/..." matches no packages)

- cd $HOME/myprojects/proj && go {build,install,fmt,test} ./... (no install location for directory $HOME/myprojects/proj/... outisde GOPATH)

Conclusion:

It's difficult to rely on Go tools while using symlinks. I especially miss godoc, which is one of Go's main selling points (but maybe there is some command lines arguments to make it work?). And some things don't work as expected. Because of this, I doubt symlinks are the way to go. But I'd be happy to be proven wrong :)

[TR NS]

I deplore the design of a centralized workspace! I order my projects by many other criteria than what language they are written in. I think the designers of Go should rethink this design and come up with a more flexible solution.

In the mean time, to work around I've had to do what you call the "obvious" way, but instead of using a symlink I have a small shell script in my project that reassigns the $GOPATH, which I can run before working on the project. Annoying but effective. Perhaps a more ideal solution would be to do the exporting when running a local build tool (e.g. Rake). By exporting the path in such a script it would only be temporary while the build is taking place, but then of course you have to write a bunch of build tasks to route to the go tool.

[Nicolas Grilly]

On Thursday, August 15, 2013 9:32:33 PM UTC+2, TR NS wrote:

In the mean time, to work around I've had to do what you call the "obvious" way, but instead of using a symlink I have a small shell script in my project that reassigns the $GOPATH, which I can run before working on the project. Annoying but effective. Perhaps a more ideal solution would be to do the exporting when running a local build tool (e.g. Rake). By exporting the path in such a script it would only be temporary while the build is taking place, but then of course you have to write a bunch of build tasks to route to the go tool.

You're right: what I called the "obvious way" is more hacky than obvious :)

Could you give some details about your setup? What is the layout of your source tree? Is the root of the Go workspace inside or outside your source tree? To which directory $GOPATH points?

-- Nicolas

[Rodrigo Kochenburger]

By the way, guys. If you need more flexibility you can always call the tools directly (compiler, linker, etc) instead of using the higher level tool and build your own build script.

go tool 6c foo.go

go tool 6l -o foo foo.8

./foo

I definitely not recommend doing that for libraries because it's much more important for them to be go-getable but if it's a private application that you won't be sharing, you should be fine.

- RK

--

[Nicolas Grilly]

On Thursday, August 15, 2013 9:54:50 PM UTC+2, Rodrigo Kochenburger wrote:

By the way, guys. If you need more flexibility you can always call the tools directly (compiler, linker, etc) instead of using the higher level tool and build your own build script.

Yes, I'm aware of the possibility to call tools directly. It's okay for compiling and linking. But what about go test and godoc? Is it possible to use these two tools without conforming to the recommended directory layout?

[andrey mirtchovski]

I don't understand the issue here. GOPATH is only really needed when
you're doing "go get" or "go install". as long as you have a default
GOPATH for the stuff you "go get", then the rest of your projects can
be wherever you want. use relative imports to build library
hierarchies completely outside the canon, but still use the go tool to
compile them. I have plenty of projects outside GOPATH (including
stuff with relative imports), the go tool manages them just fine*.

$ echo $GOPATH
/Users/aam
$ cd work/logs/serve/
$ go clean
$ go build
$ ls -l serve
-rwxr-xr-x 1 aam aam 9232204 15 Aug 14:15 serve
$ go install
go install: no install location for directory
/Users/aam/work/logs/serve outside GOPATH
$

[Nicolas Grilly]

On Thursday, August 15, 2013 10:22:05 PM UTC+2, andrey mirtchovski wrote:

I don't understand the issue here. GOPATH is only really needed when
you're doing "go get" or "go install". as long as you have a default
GOPATH for the stuff you "go get", then the rest of your projects can
be wherever you want. use relative imports to build library
hierarchies completely outside the canon, but still use the go tool to
compile them. I have plenty of projects outside GOPATH (including
stuff with relative imports), the go tool manages them just fine*. 

Putting packages outside GOPATH forces the use of relative import paths, as you wrote. The problem is that the use of relative import paths is heavily discouraged by most members of the Go team, according to what I've read in this list.

As an example, here is a link to a related issue "cmd/go: Remove support for relative import paths": https://code.google.com/p/go/issues/detail?id=6147

If I'm not mistaken, I see two other problems with this approach:

- Every package is recompiled each time you build/test (binary packages are not stored in GOPATH/pkg as usual).

- godoc -http is not usable to browse your package documentation.

-- Nicolas

[andrey mirtchovski]

> The problem is that the use of relative import paths is heavily
> discouraged by most members of the Go team, according to what I've read in
> this list.

and putting stuff inside GOPATH is the recommended way of doing
things. we shouldn't have this discussion then :)

> - Every package is recompiled each time you build/test (binary packages are
> not stored in GOPATH/pkg as usual).

not a problem: go compiles fast.

> - godoc -http is not usable to browse your package documentation.

godoc doesn't list a web browser in its requirements. use from the command line.

that said, the relative import trick is not for everybody, i agree. if
i understand your constraints correctly, what you really are asking
for is to have your Go code live in two places at the same time:
inside GOPATH when convenient, and outside GOPATH the rest of the
time. you can do with some degree of success with union mounts on
unix-y operating systems, or you can even use sshfs to do it. just
mount at the right path and, even though you'll be editing outside
GOPATH the go tool will find your code. that gives you a fine control
over naming libraries as only the last part of the path must agree
with the stated package name. here's an example:

http://play.golang.org/p/twUCNd2UhJ

[andrey mirtchovski]

Completely unrelated, but only just so: i discovered a Mark V Shaney
quote (from his last posting on usenet) that is not only apt, but very
appropriate. paraphrasing only slightly, replacing Unix with Go:

"There is no problem and thus we disagree only if you don't feel like Go."

[TR NS]

First, let me qualify things by saying I am new to Go, so I am still learning how to do things too. Having said that, do you know about Ruby Gemsets? We'll its basically like that. By changing the $GOPATH to point to each project as I work on it, I have an isolated set of Go installs all geared for use by the project I am working on. Of course the downside it that It creates redundancy, but I can live with it.

[Benjamin Measures]

On Thursday, 15 August 2013 00:01:01 UTC+1, Nicolas Grilly wrote:

Another way is to checkout the full project source (Go and Python) in $GOPATH/src/myproject and import "myproject/go" instead of "myproject". It's perfect and idiomatic from Go's point of view. But from Python's point of view, it's weird to be forced to live in $GOPATH. Go requires the project source to live in a directory src under $GOPATH. Now let's imagine that Python would require the project source to be rooted in a directory lib (for example) under its own $PYPATH. That would make Go's and Python's requirements contradictory and impossible to satisfy at the same time.

It's only impossible if you've really completely ruled out symlinks. If you can't symlink go, why not python? Eg.

ln -s $GOPATH/src/myproject/python $PYPATH/lib

[Nicolas Grilly]

On Sun, Aug 18, 2013 at 2:55 AM, Benjamin Measures <saint....@gmail.com> wrote:

It's only impossible if you've really completely ruled out symlinks. If you can't symlink go, why not python? Eg.

ln -s $GOPATH/src/myproject/python $PYPATH/lib

You're right, it will work this way. Luckily, Python is more forgiving than Go with regards to symlinks :)

[Nicolas Grilly]

On Thursday, August 15, 2013 5:32:30 PM UTC+2, Jeremy Wall wrote:

Well first you'd have to do some rearranging. Add a gopath like directory with a src/ and pkg/ directory in your repo. github.com/ngrilly/myproject/go/{src,pkg} for instance

Then move the *.go files into a reasonable directory in the src/ directory. .../go/src/myproject/*.go for instance.

Then in your build script which you will need anyway since this is a multi language project add e.g. github.com/ngrilly/myproject/go/src/ to your GOPATH so all go builds of myproject/... will work.

This will work for all your developers and packagers. It will not work for making the the go code go gettable but this is a multilanguage project where Go is not the primarly language so that is likely not really a goal in the first place.

if go gettability *is* a goal then you should probably have the go code in a separate repo instead.

Jeremy, thank you for the detailed explanation!

I've eventually decided to not follow this way. After some tests, I prefer the "usual" approach of checking out the project source tree in $GOPATH/src/myproject, instead of hosting the Go workspace under the project source tree in $HOME/projects/myproject/go/src.

I perceive the $GOPATH as a kind of temporary workspace, and I prefer it to live outside my source repository, instead on inside.

[Nicolas Grilly]

On Sunday, August 18, 2013 2:12:01 AM UTC+2, TR NS wrote:

First, let me qualify things by saying I am new to Go, so I am still learning how to do things too. Having said that, do you know about Ruby Gemsets? We'll its basically like that. By changing the $GOPATH to point to each project as I work on it, I have an isolated set of Go installs all geared for use by the project I am working on. Of course the downside it that It creates redundancy, but I can live with it.

Yes, I know about Ruby Gemsets (and Python virtualenv too) :)

You're right, it's easy to get something similar in Go by having one workspace per project and changing the $GOPATH each time you switch to another project. But I would argue it's less necessary in Go than in Ruby and Python because Go produces executable binaries. You need the workspace only when you build and test your project, not when you simply execute it. It's very different in Ruby and Python where having the correct environment is essential to running your project.

Because of this, and after a few weeks using Go, I now think that maintaining multiple workspaces is overkill in Go.

[Nicolas Grilly]

On Thursday, August 15, 2013 6:51:13 AM UTC+2, Kyle Lemons wrote:

Since Go is almost always the primary language for such projects of mine, I just make all of the source live in $GOPATH/src/github.com/kylelemons/project.  The Go code imports itself as github.com/kylelemons/project/whatever and the rest of the code doesn't really care where it lives.

Kyle, I like your answer because it's simple and it just works. As often, less is more :)

And you're right about "the rest of the code doesn't really care where it lives".

I noticed that other experienced Go developers gave the same advice in other discussions:

I recommend every beginner to set his $GOPATH=$HOME and forget about it forever. Just code. -- Jan Mercl

I always had my source code in $HOME/src anyway. Yes, it's not only Go code, but so what? It's great that I can have my Go and non-Go code together in the same place. Another benefit is that it will put binaries in $HOME/bin, which many people already add to their $PATH. -- Aram Hăvărneanu

On my machine, I have my GOPATH=$HOME, so my src in in $HOME/src, binaries get installed to $HOME/bin (which is in my $PATH), and $HOME/pkg is a junk cache directory of temporary object files (which I kinda ignore). -- Brad Fitzpatrick

I just followed these advices and it works well. For the record, I was afraid of godoc becoming slow because of having to scan all my projects, including the non-Go ones, but it appears it's really quick and not a problem in practice.

[Nicolas Grilly]

On Thursday, August 15, 2013 11:43:27 PM UTC+2, andrey mirtchovski wrote:

that said, the relative import trick is not for everybody, i agree. if
i understand your constraints correctly, what you really are asking
for is to have your Go code live in two places at the same time:
inside GOPATH when convenient, and outside GOPATH the rest of the
time. you can do with some degree of success with union mounts on
unix-y operating systems, or you can even use sshfs to do it. just
mount at the right path and, even though you'll be editing outside
GOPATH the go tool will find your code. that gives you a fine control
over naming libraries as only the last part of the path must agree
with the stated package name. here's an example:

http://play.golang.org/p/twUCNd2UhJ

Andrey, that's a very clever solution :)

I would not recommend this as the default way to install the project I'm working on, because I don't want to impose the usage of union mounts and sshfs to my co-workers, but I could definitely use it for myself.

Your suggestion reminds me of Plan 9 name spaces and the paper "Lexical File Names in Plan 9 or Getting Dot-Dot Right" by Rob Pike (http://plan9.bell-labs.com/sys/doc/lexnames.html).

-- Nicolas

[andrey mirtchovski]

> Your suggestion reminds me of Plan 9 name spaces and the paper "Lexical File
> Names in Plan 9 or Getting Dot-Dot Right" by Rob Pike
> (http://plan9.bell-labs.com/sys/doc/lexnames.html).

that's where it comes from :)

[fmcc...@gmail.com]

I'm having a similar problem with GOPATH and compiling. The issue I'm having is with multiple branches of code. A typical project looks like:

project_foo/

    trunk/

    branch1/

    branch2/

I don't have too many branches going at once, but even a single branch is causing problems because if I have this in the src/ directory of the GOPATH, then the branches are assumed to be part of the package names. It seems that the root location of a project or a branch of the project is tied to the package name. Is there any way around this?

[Caleb Doxsey]

I often use symlinks. They mostly work.

Suppose you have:

/Users/whoever/projects/project_foo/trunk

You might want it here:

/Users/whoever/go/src/code.google.com/p/project_foo

And you can do:

ln -s /Users/whoever/projects/project_foo/trunk /Users/whoever/go/src/code.google.com/p/project_foo

In windows you can use junctions. (http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html)

--

[Fatih Arslan]

We (Koding.com) use a single repository with several different
programing languages in our source directory. Go code is a lot in our
codebase, what we do is:

1. We have a directory called "go" in which we put all our source code
written in Go. This folder is inside our Git repo.
2. This "go" directory is lied out as GOPATH, that means the inside
there are directories like src/, pkg/, bin/ and so on.
3. We have a custom build.sh script that lives in this "go" directory,
when invoked it sets the GOPATH like:

export GOPATH=$(cd "$(dirname "$0")"; pwd)

and many other minor settings.

4. The script also builds each our "main" packages with "go install".
All the binaries will be copied by "go install" to "go/bin"

It works perfectly and without any problem. If you want you can set
the GOPATH in your bashrc/zshr and that way each "go get" package will
also installed in that particular go directory (I'm working personally
that way).

However our repository is growing and we look to split out the Go
directory into a separate git submodule. We are currently evaluating
if this is going to work, because there are some conflicts we have to
solve prior (we have config files that are shared and used by many
other applications of us).

On Wed, Aug 14, 2013 at 4:01 PM, Nicolas Grilly
<nic...@vocationcity.com> wrote:
> What is the recommended way to organize the source tree of a project written
> in several programming languages (for example Go and Python)?
>
> Ideally, we would want to keep everything in the same git repository because
> 1) it makes easier to keep the Python and Go logic in sync and 2) some
> assets are shared by the Python and Go code.
>
> One obvious way is to checkout the project source in some arbitrary
> directory like $HOME/projects/myproject, create a subdirectory go containing
> all the Go code, and make it accessible in $GOPATH with ln -s
> $HOME/projects/myproject/go $GOPATH/src/myproject. But this solution looks
> like a dead-end because symbolic links were declared "not supported" many
> times on golang-nuts and golang-dev.
>

> Another way is to checkout the full project source (Go and Python) in
> $GOPATH/src/myproject and import "myproject/go" instead of "myproject". It's
> perfect and idiomatic from Go's point of view. But from Python's point of
> view, it's weird to be forced to live in $GOPATH. Go requires the project
> source to live in a directory src under $GOPATH. Now let's imagine that
> Python would require the project source to be rooted in a directory lib (for
> example) under its own $PYPATH. That would make Go's and Python's
> requirements contradictory and impossible to satisfy at the same time.
>

> Another way is to create a Go workspace directly in the project source tree
> and add it to the $GOPATH. This implies myproject would have its own
> src/bin/pkg directories and this approach was explicitly not recommended on
> this mailing list in the past
> (https://groups.google.com/forum/#!topic/golang-nuts/dxOFYPpJEXo).
>
> I'm not sure how to solve this in a simple and clean way. What is your
> experience solving this issue in your own projects?
>
> Nicolas
>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Fatih Arslan

[fmcc...@gmail.com]

I realize that this is a very late response to this thread, but project layout has been something I’ve continued to struggle with since day one with Go.

I’ve tried capturing the entire gopath in version control, but this is awkward and makes creating go-gettable libraries difficult. I’ve gone back and forth on this and I think the simple answer is if you can reasonably work inside a single gopath, that’s probably the best way of working.

I can’t quite pull that off because of how I do branching, so I’ve been relying on home grown tools to mess around with the gopath. I’ve opensourced them today at:

http://gojank.com

I still consider this something of an experiment. If you can work inside a single gopath, that’s the way to go, but this is what i’m doing for now.