[RFC] Debian packaging of Go libraries and programs

[Michael Stapelberg]

Dear Go community,

I spent some time trying to figure out how to best package Go libraries and
programs in Debian.

You can read the discussion on the debian-devel mailinglist at:
http://thread.gmane.org/gmane.linux.debian.devel.general/179252

To summarize, the following questions remain:

1. What is the best/intended way for a Linux distribution to ship a Go library?
E.g. ship just its source or also ship the static library (pkg/)?

2. Is it required to have the full source of a library available or is there
something like C’s header files?

3. Which compiler should a distribution chose to support? gc, gccgo, both?
Are there specific reasons or gotchas for any of the options?

4. The issue that Go’s host identifier linux_arm is too unspecific came up.
Debian differentiates between arm-linux-gnueabi and arm-linux-gnueabihf.
Read more about that at http://wiki.debian.org/Multiarch/Tuples
Would you consider a switch to these identifiers for easier cross-compiling
in the future?

5. There are arguments against static linking in the context of Debian:
http://thread.gmane.org/gmane.linux.debian.devel.general/179252/focus=179269
What is the current state of dynamic linking with Go, and would you
recommend using it? If not, why not?

Thanks in advance,
Best regards,
Michael

[Mike Rosset]

On Wed, Jan 2, 2013 at 1:14 PM, Michael Stapelberg
<stape...@debian.org> wrote:
> Dear Go community,
>
> I spent some time trying to figure out how to best package Go libraries and
> programs in Debian.
>
> You can read the discussion on the debian-devel mailinglist at:
> http://thread.gmane.org/gmane.linux.debian.devel.general/179252
>
> To summarize, the following questions remain:
>
> 1. What is the best/intended way for a Linux distribution to ship a Go library?
> E.g. ship just its source or also ship the static library (pkg/)?

This might be a good start, personally I think packaging the binaries
is redundant if everyone is useing GOPATH.

> 2. Is it required to have the full source of a library available or is there
> something like C’s header files?

you need the source, go source files do the book keeping. also there
part of docs.

> 3. Which compiler should a distribution chose to support? gc, gccgo, both?

for now I would say gc. however I think ultimately when it comes to
packaging gccgo fits better. to early to tell here.

> Are there specific reasons or gotchas for any of the options?

go tool has no concept of DESTDIR other then GOPATH. so you need to do
some real trickery with GOPATH to emulate DESTDIR.

> 4. The issue that Go’s host identifier linux_arm is too unspecific came up.
> Debian differentiates between arm-linux-gnueabi and arm-linux-gnueabihf.
> Read more about that at http://wiki.debian.org/Multiarch/Tuples
> Would you consider a switch to these identifiers for easier cross-compiling
> in the future?

> 5. There are arguments against static linking in the context of Debian:
> http://thread.gmane.org/gmane.linux.debian.devel.general/179252/focus=179269
> What is the current state of dynamic linking with Go, and would you
> recommend using it? If not, why not?
>
> Thanks in advance,
> Best regards,
> Michael

ultimately I think you could make one master autoconf project to
package any go installable project. and upstream developers can
continue to use go tool but you get all the advantages of host tuples,
prefixing etc from autoconf just for packaging. this is just a theory
not sure if its doable.

[Marcelo]

Hi,

On Wed, Jan 2, 2013 at 5:39 PM, Mike Rosset <mike....@gmail.com> wrote:
>> 3. Which compiler should a distribution chose to support? gc, gccgo, both?
>
> for now I would say gc. however I think ultimately when it comes to
> packaging gccgo fits better. to early to tell here.

You also have to keep in mind that Debian supports more architectures
than gc does, and in those architectures the only alternative is
gccgo. For example, at the moment I have an interest in developing
with Go for i386, ARM, MIPS and PPC.

I would rephrase the question as what to do in the architectures
supported by both gc and gccgo.

Marcelo

[Dave Cheney]

Hello Michael,

Thank you for reaching out to the Go community. I will try to answer
your questions below.

> 1. What is the best/intended way for a Linux distribution to ship a Go library?
> E.g. ship just its source or also ship the static library (pkg/)?

At the moment both source and .a should be included. Please see the next point.

> 2. Is it required to have the full source of a library available or is there
> something like C’s header files?

Although a compiled package, a .a includes all the data necessary for
compilation, there is an issue which was not resolved before the Go
1.0 release, which requires the source to also be present.
golang.org/issue/2775. Hopefully this will be resolved for Go 1.1, at
which point, a library/package can be described and distributed only
as a .a.

Others may disagree with me on this next recommendation, however, I
think that any libraries (packages) packaged by Debian should be
installed in $GOROOT, this has the same properties as the endorsed
directory that the JDK uses.

> 3. Which compiler should a distribution chose to support? gc, gccgo, both?
> Are there specific reasons or gotchas for any of the options?

Ideally both, but I'm not experience enough with gccgo to describe
their interoperability. The current golang-go package describes
version 1.0.2 of the gc compiler suite, I'd expect that name to
continue. I recommend talking to Ian Lance Taylor, ia...@golang.org.

btw. what is involved in getting that package updated to 1.0.3 ?

> 4. The issue that Go’s host identifier linux_arm is too unspecific came up.
> Debian differentiates between arm-linux-gnueabi and arm-linux-gnueabihf.
> Read more about that at http://wiki.debian.org/Multiarch/Tuples
> Would you consider a switch to these identifiers for easier cross-compiling
> in the future?

Yes, this is a quite complex issue. It's actually two issues.

The first is the instruction set used to compile the Go compilers
(5{c,g,a,l} in this case), this is the standard armel vs armhf debate.
You can use a armel 5{g,c,a,l} binary on armhf, but not vice versa.

The second part is what arm arch does the 5l linker target, which is
decided completely at the link time of the final go executable (not
the go compiler). A .a library is both armel and armhf as it is the
linker that makes the final decision to use hardware or software
floating point, and which floating point support to include. This
defaults to ARMv6 but can be overridden by an environment variable
(GOARM) or hard coded into the linker when it is compiled, or both.

> 5. There are arguments against static linking in the context of Debian:
> http://thread.gmane.org/gmane.linux.debian.devel.general/179252/focus=179269
> What is the current state of dynamic linking with Go, and would you
> recommend using it? If not, why not?

There are two parts to this question

1. Can go programs share common code via shared (dynamic) libraries ?

2. Can go programs be extended at run time by loading new code via a
shared (dynamic) library mechanism ?

At the moment neither is possible, and I expect the gc compiler suite
will continue to produce static executables for the Go 1.1 timeframe.
I cannot speak authoritatively about gccgo.

Cheers

Dave

[Michael Stapelberg]

Hi Dave,

Dave Cheney <da...@cheney.net> writes:
> Others may disagree with me on this next recommendation, however, I
> think that any libraries (packages) packaged by Debian should be
> installed in $GOROOT, this has the same properties as the endorsed
> directory that the JDK uses.

What properties do you mean specifically? That is, why is $GOROOT a
better fit than, say, /usr/lib/gocode?

> Ideally both, but I'm not experience enough with gccgo to describe
> their interoperability. The current golang-go package describes
> version 1.0.2 of the gc compiler suite, I'd expect that name to
> continue. I recommend talking to Ian Lance Taylor, ia...@golang.org.

Yes, but we also have gccgo in Debian.

> btw. what is involved in getting that package updated to 1.0.3 ?

Given that we are currently in a freeze, we could only upload 1.0.3 to
Debian experimental. The reason it wasn’t done yet is that other things
were more important :-).

>> 5. There are arguments against static linking in the context of Debian:
>> http://thread.gmane.org/gmane.linux.debian.devel.general/179252/focus=179269
>> What is the current state of dynamic linking with Go, and would you
>> recommend using it? If not, why not?
>
> There are two parts to this question
>
> 1. Can go programs share common code via shared (dynamic) libraries ?
>
> 2. Can go programs be extended at run time by loading new code via a
> shared (dynamic) library mechanism ?
>
> At the moment neither is possible, and I expect the gc compiler suite
> will continue to produce static executables for the Go 1.1 timeframe.
> I cannot speak authoritatively about gccgo.

Ian: Could you comment on whether either of these is possible using
gccgo please?

--
Best regards,
Michael

[Mike Rosset]

> What properties do you mean specifically? That is, why is $GOROOT a
> better fit than, say, /usr/lib/gocode?

GOROOT tends to be a better fit because gc is designed to be self
contained. the upstream suggested install location recommends
/usr/local/go from a packaging FHS standpoint that translates "for me
anyways" to say /opt/go. see
http://www.pathname.com/fhs/pub/fhs-2.3.html#OPTADDONAPPLICATIONSOFTWAREPACKAGES

I did some work cleaning up the archlinux package and it seemed to me
it would be much easier to stay inline with upstream in this regards
then to work against it.

[minux]

On Thu, Jan 3, 2013 at 5:14 AM, Michael Stapelberg <stape...@debian.org> wrote:

I spent some time trying to figure out how to best package Go libraries and
programs in Debian.

Thank you! 

You can read the discussion on the debian-devel mailinglist at:
http://thread.gmane.org/gmane.linux.debian.devel.general/179252

To summarize, the following questions remain:

1. What is the best/intended way for a Linux distribution to ship a Go library?
   E.g. ship just its source or also ship the static library (pkg/)?

ship the static library is enough (note that, the library embeds the specific Go release

that compiled them, so if you update the compiler, all of them need to be updated).

also note that, if you want to provide a GOPATH like environment, you need to provide

at least one dummy Go source file (a single package declaration suffices) with a

timestamp older than its static library. it might be changed for Go 1.1, however.

2. Is it required to have the full source of a library available or is there
   something like C’s header files?

in theory, the source is not required. however, as for now, we don't support

binary only packages (see above for a workaround). 

3. Which compiler should a distribution chose to support? gc, gccgo, both?
   Are there specific reasons or gotchas for any of the options?

definitely both. those two compilers have different strengths and drawbacks.

4. The issue that Go’s host identifier linux_arm is too unspecific came up.
   Debian differentiates between arm-linux-gnueabi and arm-linux-gnueabihf.
   Read more about that at http://wiki.debian.org/Multiarch/Tuples
   Would you consider a switch to these identifiers for easier cross-compiling
   in the future?

because our package archive (*.a) is truly just specific to linux-arm (eabi).

whether it's hf or not is set by GOARM environment variable during linking (5l).

5. There are arguments against static linking in the context of Debian:
   http://thread.gmane.org/gmane.linux.debian.devel.general/179252/focus=179269
   What is the current state of dynamic linking with Go, and would you
   recommend using it? If not, why not?

I think static linking is a major feature of Go. If you want dynamic linking, please

consider gccgo.

However, I think major 3rd-party packages out there all assume it's built by gc

toolchain (esp. when there is assembly).

[Ian Lance Taylor]

On Thu, Jan 3, 2013 at 12:58 AM, Michael Stapelberg
<stape...@debian.org> wrote:

>
> Dave Cheney <da...@cheney.net> writes:
>>
>> There are two parts to this question
>>
>> 1. Can go programs share common code via shared (dynamic) libraries ?
>>
>> 2. Can go programs be extended at run time by loading new code via a
>> shared (dynamic) library mechanism ?
>>
>> At the moment neither is possible, and I expect the gc compiler suite
>> will continue to produce static executables for the Go 1.1 timeframe.
>> I cannot speak authoritatively about gccgo.
> Ian: Could you comment on whether either of these is possible using
> gccgo please?

When using gccgo you can create and use shared libraries. Loading new
code via dlopen doesn't really work; the Go initialization code for
the dlopen'ed shared library will not be run.

Ian

[Dave Cheney]

> What properties do you mean specifically? That is, why is $GOROOT a
> better fit than, say, /usr/lib/gocode?

The go tool will look in two locations to satisfy package
dependencies, $GOROOT and $GOPATH.

$GOROOT is a single entry that points to the location that the source,
compiled packages, and tools that make up the Go distribution. In the
current golang-go dpkg, this is /usr/lib/go. That value comes from the
$GOROOT_FINAL value that is present when building the deb.

$GOPATH is a shell path like variable that can contain multiple
additional paths to search for source and packages. These paths are
subservient to $GOROOT; you cannot override a package in $GOROOT by
placing an entry in $GOPATH. The first entry in $GOPATH also has
special properties that the go get command will use that location as a
destination when fetching remote Go packages.

My interpretation of this is

a. $GOPATH must only contain entries which the user has permission to
write too, otherwise go get will break, and people will use the sudo
hammer.

b. $GOPATH should not be used to provide distro packaged additional
packages, as we've just gone through a year of educating people to
_set_ the value, and it would be disastrous if they had to relearn to
append their own paths to the value of $GOPATH.

c. $GOROOT is not expected to be writable by the user, so it is a good
location to install other distro packaged Go packages (whew,
mouthful), as the .a and src can be placed there providing the .a has
a later timestamp than the source it represents (see Minux's comments
about fixing this restriction for go 1.1)

> Given that we are currently in a freeze, we could only upload 1.0.3 to
> Debian experimental. The reason it wasn’t done yet is that other things
> were more important :-).

Fair enough. It's probably too late now, Go 1.1 will be shipping soon.

Cheers

Dave

[Michael Stapelberg]

Hi Ian,

Ian Lance Taylor <ia...@google.com> writes:
> When using gccgo you can create and use shared libraries. Loading new
> code via dlopen doesn't really work; the Go initialization code for
> the dlopen'ed shared library will not be run.

Thanks for clarifying. Could you please give a short example on how I
would compile a Go package (e.g. github.com/mstap/godebiancontrol) into
a shared library using gccgo? I have googled this several times and was
unable to find the instructions.

--
Best regards,
Michael

[Andrew Gerrand]

Hi Michael,

Thanks for your interest in packaging Go for Debian. The current state of affairs is unfortunate; among other things, godoc doesn't work.

I have a couple of requests/suggestions.

Please keep the entire Go distribution inside one $GOROOT. The tools assume this is the case, and attempts to do otherwise have broken the tools. In our binary distributions we assume /usr/local/go (which may be overridden by setting $GOROOT), but /opt/go or really anything else is fine so long as all the pieces are inside the one root directory. The notable exception to this is the binaries inside $GOROOT/bin, which may exist anywhere in a user's path.

Please include the source. If it must be in a separate package (as is the case now), please make sure the golang metapackage installs the source. It is documentation and (as Dave Cheney mentioned) is required to build programs that use the standard packages (even if those packages' objects are present).

Regarding static/dynamic linking, the entire discussion is moot. The gc compiler should definitely be the compiler you get when you "apt-get install golang" (assuming you only get one), and it does not support dynamic linking.

Thanks,

Andrew

[Mike Rosset]

Andrew thanks for the concise email, I know this thread pertains to
debian and possibly ubuntu, but it might help me to convince the
archlinux packagers to follow the upstrem suggested GOROOT method.
Mike

> --
>
>

[Michael Stapelberg]

Hi Andrew, Dave,

Andrew Gerrand <a...@golang.org> writes:
> Thanks for your interest in packaging Go for Debian. The current state of
> affairs is unfortunate; among other things, godoc doesn't work.

In golang-go 2:1.0.2-1.1 (current version to be released with the next
Debian release) there was a fix for http://bugs.debian.org/679692, and I
can use e.g. /usr/bin/godoc net/http just fine.

Are you sure that the issue you are talking about still persists? If so,
please tell me how to reproduce and I’ll see to get it fixed.

> Please keep the entire Go distribution inside one $GOROOT. The tools assume
> this is the case, and attempts to do otherwise have broken the tools. In
> our binary distributions we assume /usr/local/go (which may be overridden
> by setting $GOROOT), but /opt/go or really anything else is fine so long as
> all the pieces are inside the one root directory. The notable exception to
> this is the binaries inside $GOROOT/bin, which may exist anywhere in a
> user's path.

Yours and Dave’s recommendations make sense. I will update the wiki page
later and summarize the conclusions I’ve come to so that we can move
forward.

> Please include the source. If it must be in a separate package (as is the
> case now), please make sure the golang metapackage installs the source. It
> is documentation and (as Dave Cheney mentioned) is required to build
> programs that use the standard packages (even if those packages' objects
> are present).

golang depends on golang-src, so it will be installed.

> Regarding static/dynamic linking, the entire discussion is moot. The gc
> compiler should definitely be the compiler you get when you "apt-get
> install golang" (assuming you only get one), and it does not support
> dynamic linking.

To clarify: You currently get gc when you apt-get install golang, you
get gccgo when you apt-get install gccgo :-).

My question on which compiler to support probably should have been
“Which compiler should we use to compile Go packages and programs?”
E.g. could we provide gc but use gccgo to build everything so that we
can use dynamic linking. Or is there an obvious downside to that?

--
Best regards,
Michael

[Aram Hăvărneanu]

Go packages are installed via go get. If you provide Debian packages
for some 3rd party Go package, the only sane place to put them is in
$GOROOT, as Dave mentioned. If you put them in $GOROOT, then they take
precedence over packages installed by users. This is bad because it
means there's no way for a user to override a system provided package
using the system provided Go.

I suggest dropping the idea of providing more Debian packages other
than the Go distributions itself. C libraries using autotools are a
PITA to install, and providing development Debian packages for them
makes sense, but Go packages are not like that. apt-get as an
alternative to go get is a solution to a problem that no one has.

--
Aram Hăvărneanu

[Mike Rosset]

> I suggest dropping the idea of providing more Debian packages other
> than the Go distributions itself. C libraries using autotools are a
> PITA to install, and providing development Debian packages for them
> makes sense, but Go packages are not like that. apt-get as an
> alternative to go get is a solution to a problem that no one has.

If you never want your go program to be distributed by any linux
distro then this attitude might work, however. what if the linux
distribution in question has its own tools some of which are written
in go. what do they do then? do they suggest everyone using that
distro should install these tools with go get?

You can not use go get to distro package go binaries, since go get can
and will grab from tip, which means replicating the package is not
fool proof. that and the package managers need some sort of DAG for at
least building that can replicated .. normally with the package
manager or some tool that is aware of system depends.

As for go language shops that are already using gc and deploying there
services by hand, they might not have a need for this at all and they
can maintain the status quo. However if they have client tools and
such they want to see go mainstream. Eventually some consensus on
distro packaging will be needed.

Mike

[Aram Hăvărneanu]

Read my post again. I was careful enough to qualify Debian packages
and Go packages. I didn't say anything about Go binaries. Package
whatever binaries written in Go you want, just don't package Go
*packages*.

--
Aram Hăvărneanu

[Mike Rosset]

understandable, however if you plan to distro package go binaries, you
also need to package the go packages it depends on, since you need to
replicate the build process, and freeze versions etc. again go get is
not enough in this regards. Its compounded by the fact the go get will
pull arbitrary urls during the process, none of which can be verified
in a sure fire way other then having them on disk before hand. And due
to the static nature of go binaries its possibly with this method to
have a binary built with sources that can never be verified, never
mind replicated.

[Marcelo]

Hi,

On Fri, Jan 4, 2013 at 4:56 AM, Aram Hăvărneanu <ara...@mgk.ro> wrote:

> I suggest dropping the idea of providing more Debian packages other
> than the Go distributions itself. C libraries using autotools are a
> PITA to install, and providing development Debian packages for them
> makes sense, but Go packages are not like that. apt-get as an
> alternative to go get is a solution to a problem that no one has.

That is the path taken by the Ruby (and more exactly, the Ruby Gems)
folks. It leads to a very unhappy place. I really hope Go doesn't
follow along.

There are scenarios where what you say _is_ the right solution. There
are others where it is the wrong one.

For example, consider the case of a very popular end-user-oriented
wonderful-program written in Go (and it is bound to show up sooner or
later, if one doesn't already exists). End-users want two things:

1. Easily install wonderful-program

2. Keep wonderful-program updated (security- and feature-wise) in a
somewhat automated way

For this Debian offers a great solution, which involves providing
Debian packages for Go packages. There _is_ a certain type of user
who feels comfortable (or even, prefers) following scripted
instructions ("type <this> at the prompt", "type <this> to upgrade",
...), who might find the solution you suggest acceptable, but that is
not true for every user.

Another use case where Debian provides a nice solution is for a
developer who wants to be able to code against an stable API. It
might be the case that more recent, more featureful releases of a Go
package, but I, as a developer, might be trying to provide a Debian
package for machines which are not under my control. In that
situation I like to know what the environment looks like and what's
available there (which I can express via dependencies communicated to
the packaging system).

Part of the reason for caring about the lack of shared objects is
precisely being able to provide security updates in a sane fashion.
You have to keep in mind that Debian cares about software licensing,
so it's not enough to provide a binary for a package and stick the
sources somewhere. Debian actually makes an effort to make sure that
you can rebuild that same package from the provided sources. This
means that if pkg-go-[1..n] all depend on pkg-go-0, if you update
pkg-go-0, you have to rebuild pkg-go-[1..n] and upload pkg-go-[0..n].
Maybe pkg-go-j doesn't use the code that was updated, but it has to be
rebuilt anyways because a) we need to verify that it still builds
(i.e. it still conforms to the terms of the license); b) we don't have
enough information to know, without further examination, that the
changed code doesn't affect the package (i.e. it's cheaper to just
rebuild). With shared objects this is relatively easy: update the
package providing the shared object; test the existing binaries;
rebuild everything in the dependency chain to make sure everything
still builds; upload only the updated package (and no, this is not
exactly the way Debian does it, but the concept still applies).

Another problem exhibited by the Ruby community is the lack of a
convention for specifying whether or not an API was updated and/or
broken.

With shared libraries it's relatively easy, even if it requires a lot
of discipline. The ELF headers include a SONAME, which is, within
some limits, pretty much an arbitrary string. Nothing forces you to
call something libfoo.so.1 and keep that name as long as the API is
not broken. You can add a function to the library and keep
libfoo.so.1 as the SONAME. This is a problem, because if binary foo
uses the new function, it will work as long as the libfoo.so.1 in the
system also has that function. On systems where that is not true,
stuff breaks, as you should have changed the SONAME (e.g.
libfoo.so.2). About a decade ago we had to fix a lot of stuff
precisely because of things like this. I'm guilty of developing
crutches to deal with this for specific cases, which are almost
certainly present in your computer if you happen to be using a modern
Linux distribution.

With systems that do not use shared libraries, you have to resort to
other ways of encoding this information. This is what Ruby developers
_do not_ do in the general case (and they too, have developed crutches
to deal with it). One way of dealing with this is simply encode an
API version in the package name. So your package is not foo, but
foo1. If you break your API, you switch to foo2. Many people will,
of course, have the luxury of not having to resort to this, but any
widely distributed code _should_ be doing this. Perl, with which Ruby
shares all the problems and shortcomings in this particular regard,
has, in general terms, done a wonderful job at getting things right.

Sorry to have veered off a tangent there, but I wanted to provide some
context as to why this is important.

Marcelo

PS: Where I say "Ruby", "Perl" and "Go", unless otherwise clear, I do
not mean the languages, but their respective (developer) communities.

[Ian Lance Taylor]

The gccgo compiler is just like gcc. To build a shared library:

gccgo -shared -o lib.so file.go

There is no way to use the go tool to build a shared library.

Ian

[Ian Lance Taylor]

On Fri, Jan 4, 2013 at 2:30 AM, Michael Stapelberg
<stape...@debian.org> wrote:
>
>> Regarding static/dynamic linking, the entire discussion is moot. The gc
>> compiler should definitely be the compiler you get when you "apt-get
>> install golang" (assuming you only get one), and it does not support
>> dynamic linking.
> To clarify: You currently get gc when you apt-get install golang, you
> get gccgo when you apt-get install gccgo :-).
>
> My question on which compiler to support probably should have been
> “Which compiler should we use to compile Go packages and programs?”
> E.g. could we provide gc but use gccgo to build everything so that we
> can use dynamic linking. Or is there an obvious downside to that?

In case it's not clear, you can't link gc-compiled code with
gccgo-compiled code. Providing gccgo compiled binary versions of
packages will only help people who use gccgo.

I don't really recommend providing packages as shared libraries even
when using gccgo, it's not what Go programmers expect.

Ian

[Michael Stapelberg]

Hi Dave,

Dave Cheney <da...@cheney.net> writes:

> a. $GOPATH must only contain entries which the user has permission to
> write too, otherwise go get will break, and people will use the sudo
> hammer.

True.

> b. $GOPATH should not be used to provide distro packaged additional
> packages, as we've just gone through a year of educating people to
> _set_ the value, and it would be disastrous if they had to relearn to
> append their own paths to the value of $GOPATH.

True.

> c. $GOROOT is not expected to be writable by the user, so it is a good
> location to install other distro packaged Go packages (whew,
> mouthful), as the .a and src can be placed there providing the .a has
> a later timestamp than the source it represents (see Minux's comments
> about fixing this restriction for go 1.1)

At the same time, placing packages in $GOROOT feels like making them
part of the Go standard library, which they are clearly not. I dislike
the consequence that users can no longer simply “go get” a package and
have it picked up. Compare that with perl and python, which both prefer
locally-installed packages (in /usr/local via cpan/pip).

What about patching golang-go in Debian so that it respects the user’s
$GOPATH, but always adds /usr/lib/gocode? Then, the user still can _set_
his $GOPATH and write permission is not required (since “go get” installs
to the user’s $GOPATH). Is there any downside I am missing?

Andrew: You talked about “keeping the entire Go distribution in a single
$GOROOT”. I’m unsure whether you mean Go packages, too? What is your
stance on that?

--
Best regards,
Michael

[Kyle Lemons]

If a prefix exists in the stdlib, you could not use that prefix in GOPATH.  Thus, if you installed github.com/anything in GOROOT, you would no longer be able to do github.com/anythingelse in GOPATH.

 

> Given that we are currently in a freeze, we could only upload 1.0.3 to
> Debian experimental. The reason it wasn’t done yet is that other things
> were more important :-).

Fair enough. It's probably too late now, Go 1.1 will be shipping soon.

Cheers

Dave

--

[Dave Cheney]

Orly? So if $GOROOT/src/pkg/girhub.com/ exists, then that will black ball any girhub based pig in GOPATH? Ill have to test that. 

[minux]

On Fri, Jan 4, 2013 at 11:12 PM, Marcelo <marcelo....@gmail.com> wrote:

For example, consider the case of a very popular end-user-oriented

wonderful-program written in Go (and it is bound to show up sooner or
later, if one doesn't already exists).  End-users want two things:

1. Easily install wonderful-program

2. Keep wonderful-program updated (security- and feature-wise) in a
somewhat automated way

As Go code compiled by gc is statically linked, what's wrong with just

package the program as a debian package? No, non of the build dependencies

need to be installed on end-user's machine, I don't understand the problem here.

just like C++, it's difficult to maintain a stable ABI, also because gc

compiler do inlining of functions and give error if an import is not used,

if pkgA and cmdA depends on pkgB, the latter package is truly depended,

and i suspect the only way to update pkgB is to update all pkgA and

cmdA that depends on pkgB.

Another thing to note is that, we don't guarantee stable object format

for the gc toolchain, so even updating the toolchain will require a complete

rebuild of all the packages installed.

Given all this and the fact that gc toolchain is lightning fast, maybe you can

just distribute the source and always recompile using the go command as

a post-install step.

PS: I think static linking is one of gc toolchain's most significant advantages,

I don't want to lose that benefit.

Regards security updates, in theory, you can wrap all such API with cgo and

use host libraries (like OpenSSL), but I suspect you will need to rewrite a

lot of packages to do that.

[Marcelo]

Hi,

On Sat, Jan 5, 2013 at 8:38 AM, minux <minu...@gmail.com> wrote:

> As Go code compiled by gc is statically linked, what's wrong with just
> package the program as a debian package? No, non of the build dependencies
> need to be installed on end-user's machine, I don't understand the problem
> here.

Oh, that's perfectly fine if you mean "package the build dependencies,
create a package with the static binary that depends on those only at
build time, install only that one package."

It's not fine if you mean "do not package the build dependencies at
all, do some magic at build time to pull them off the Internet,
generate a static binary, package that and install the result."

I assume you mean the former, but just for the record, the second is
not an option for multiple reasons:

* Reproduceability -- Debian tries really hard to be able to recreate
the entire binary distribution from a bootstrap environment and the
build dependencies each package declares.

* No requirement for Internet connectivity -- assuming you have
obtained the entire Debian distribution (which includes sources), you
must be able to rebuild the whole thing without Internet connectivity.

* Some licenses _require_ that you make the sources you used to build
that static binary available to the user receiving the static binary.
In turn this requires that you are able to recreate the entire build
environment, including build dependencies. Debian is very strict
about being able to legally redistribute binaries, i.e. obeying all
the terms of the licenses.

There are others, but I think these are the ones most (all?)
developers should be able to identify with rather simply.

My comment was addressed at the suggestion to "package just the base
system, and leave users to deal with installing all the other stuff
beyond that" (sorry for rephrasing, but that's the way I understood
the comment, I apologize if that wasn't the intention).

Marcelo

[Mike Rosset]

Marcelo thanks for explaining this much better then I could have.

> --
>
>

[Dave Cheney]

> Given all this and the fact that gc toolchain is lightning fast, maybe you
> can
> just distribute the source and always recompile using the go command as
> a post-install step.

This is an interesting point. Just as .py files are compiled to .pyc
as a post install step, go packages could be compiled as part of the
deb install process, the computational overhead is on par with .pyc
compilation and hardly arduous. That would ensure that both the source
requirements of any static binary are met, and sidestep any package
version (go package versioning that is) issues, but again enforcing
that the final binary is produced from known versioned (by virtue of
debian packaging) source.

Dave

[Dave Cheney]

> At the same time, placing packages in $GOROOT feels like making them
> part of the Go standard library, which they are clearly not. I dislike
> the consequence that users can no longer simply “go get” a package and
> have it picked up. Compare that with perl and python, which both prefer
> locally-installed packages (in /usr/local via cpan/pip).

Ok, I think there are two issues here

1. commingling 3rd party source/packages with the Go source/packages
inside $GOROOT.

2. deb packaging a Go package prevents users from being able to use go
get to download a newer version.

I'll tackle each separately, as I believe they are independent

On the first issue, 3rd party packages inside $GOROOT. This is how the
author (Russ Cox) of the go tool expected things to work, and in fact
the go tool itself will try to download source into $GOROOT as a
default if you have not set a $GOPATH variable. It turns out that this
got a bit broken when people switched to distro packaged Go because
they no longer had write access to /usr/lib/go (see sudo hammer).

Having said that, there is no reason why 3rd party packages cannot be
placed into $GOROOT, emulating the intent of the previous sentence.
dpkg establishes an owner for every file, so there is no reason to
place 3rd party packages into a different directory tree to indicate
ownership.

On the second issue, packages in $GOROOT always taking precedence over
$GOPATH. This is a tricky problem, and possibly why other responders
have suggested they don't see a use for packaging 3rdparty Go
packages, just the base Go distribution thanks. There are a number of
gotchas here, for example:

If I am writing some code that depends on pkg A and I have a version
of pkg A in my $GOPATH, then I apt-get install $SOMEUNRELATEDPACKAGE,
which happens to pull in a few Go packages into the $GOROOT (for the
sake of this argument, this is any location that is not $GOPATH). Now
there is a different package A which takes precedent over the pkg A in
my $GOPATH and my code is broken.

Sure, this situation is fixable with apt-get remove and some detective
work, but not ideal.

I think the answer to this second issue is package versioning, Go
package versioning, not deb package versioning. This has long been
known to be an issue, but hasn't got a seat at the table yet.

> What about patching golang-go in Debian so that it respects the user’s
> $GOPATH, but always adds /usr/lib/gocode? Then, the user still can _set_
> his $GOPATH and write permission is not required (since “go get” installs
> to the user’s $GOPATH). Is there any downside I am missing?

See above. I probably didn't answer your question in it's entirety on
this point, but I think the previous section is enough to chew on for
the moment.

Cheers

Dave

[Kyle Lemons]

Were you able to reproduce this behavior?  I tested it locally and it is still that way for me at tip.  Adding the package in GOROOT causes a previously working package to break.

[Michael Stapelberg]

cc +Russ Cox, since he wrote go(1).

Dave Cheney <da...@cheney.net> writes:
> 2. deb packaging a Go package prevents users from being able to use go
> get to download a newer version.

This is only the case if we place them in $GOROOT, which is why I
suggested to append to $GOPATH instead. In that sense, these problems
are not really independent (IMO).

> I think the answer to this second issue is package versioning, Go
> package versioning, not deb package versioning. This has long been
> known to be an issue, but hasn't got a seat at the table yet.

I have not read any of the previous discussions about Go package
versioning, but it seems likely to me that this requires touching a lot
of libraries and/or a new Go release. Given that Go 1.1 will be released
soon (I’ve been told in this thread), it seems unlikely that Go package
versioning will make its way into that release.

Therefore, while Go package versioning might solve this problem in the
mid- to long-term future, I’d like to have a solution which works right
now ;-).

>> What about patching golang-go in Debian so that it respects the user’s
>> $GOPATH, but always adds /usr/lib/gocode? Then, the user still can _set_
>> his $GOPATH and write permission is not required (since “go get” installs
>> to the user’s $GOPATH). Is there any downside I am missing?
>
> See above. I probably didn't answer your question in it's entirety on
> this point, but I think the previous section is enough to chew on for
> the moment.

Well, given that I suggested extending to $GOPATH within the go(1)
source for the second or third time by now and nobody had anything to
say against it, I’m really curious if there actually are reasons against
it.

Can you think of any?

--
Best regards,
Michael

[Dave Cheney]

> This is only the case if we place them in $GOROOT, which is why I
> suggested to append to $GOPATH instead. In that sense, these problems
> are not really independent (IMO).

Please see below.

> Therefore, while Go package versioning might solve this problem in the
> mid- to long-term future, I’d like to have a solution which works right
> now ;-).

Yes, that was my suggestion also. Sorry for not being clear.

> Well, given that I suggested extending to $GOPATH within the go(1)
> source for the second or third time by now and nobody had anything to
> say against it, I’m really curious if there actually are reasons against
> it.

I do not want to touch $GOPATH, that is owned wholly by the user and
should only refer to paths that they control. Here are some
alternative proposals:

1. Change $GOROOT from a value to a path, so $GOROOT may be
/usr/lib/go:/usr/lib/gocode on debian systems. This solves the problem
of commingling packages from the standard library and debian provided
packages. This would not solve the problem of $GOROOT packages taking
precedence over $GOPATH.

2. Create an additional variable, $GOOTHER (this is a terrible name,
consider it a working title) which is not owned by the user, and comes
after $GOPATH in the search path, $GOROOT -> $GOPATH -> $GOOTHER. This
solves the precedence issues, but introduces a new place packages can
hide.

To be honest, I think i'd have more luck proposing the former, but
that is a decision for Russ and Andrew.

Cheers

Dave

[Dave Cheney]

I will test tonight/tomorrow.

[minux]

On Sun, Jan 6, 2013 at 3:40 AM, Marcelo <marcelo....@gmail.com> wrote:

On Sat, Jan 5, 2013 at 8:38 AM, minux <minu...@gmail.com> wrote:

> As Go code compiled by gc is statically linked, what's wrong with just
> package the program as a debian package? No, non of the build dependencies
> need to be installed on end-user's machine, I don't understand the problem
> here.

* Reproduceability -- Debian tries really hard to be able to recreate
the entire binary distribution from a bootstrap environment and the
build dependencies each package declares.

* No requirement for Internet connectivity -- assuming you have
obtained the entire Debian distribution (which includes sources), you
must be able to rebuild the whole thing without Internet connectivity.

* Some licenses _require_ that you make the sources you used to build
that static binary available to the user receiving the static binary.
In turn this requires that you are able to recreate the entire build
environment, including build dependencies.  Debian is very strict
about being able to legally redistribute binaries, i.e. obeying all
the terms of the licenses.

For this issue, I've already recommended distributing package source code

as normal "binary" package (you need to figure out how to do proper versioning

yourself, but it's not a Go specific issue) and compile the package after

installation as a post-inst step.

In this case, you can grab packages for each of the depended packages

and build the final static program, i can't see any program with this scheme.

In this way, all your requirements are satisfied.

[Dave Cheney]

Hmm, I wasn't able to reproduce what I thought was the problem

lucky(~/go/src/pkg) % mkdir code.google.com
lucky(~/go/src/pkg) % go install -x code.google.com/p/go.crypto/ssh
WORK=/tmp/go-build110973919
mkdir -p $WORK/code.google.com/p/go.crypto/ssh/_obj/
cd /home/dfc/src/code.google.com/p/go.crypto/ssh
/home/dfc/go/pkg/tool/linux_amd64/6g -o
$WORK/code.google.com/p/go.crypto/ssh/_obj/_go_.6 -p
code.google.com/p/go.crypto/ssh -complete -D
_/home/dfc/src/code.google.com/p/go.crypto/ssh -I $WORK ./agent.go
./buffer.go ./certs.go ./channel.go ./cipher.go ./client.go
./client_auth.go ./common.go ./doc.go ./keys.go ./mac.go ./messages.go
./release.go ./server.go ./server_terminal.go ./session.go ./tcpip.go
./transport.go
/home/dfc/go/pkg/tool/linux_amd64/pack grcP $WORK
$WORK/code.google.com/p/go.crypto/ssh.a
$WORK/code.google.com/p/go.crypto/ssh/_obj/_go_.6
mkdir -p /home/dfc/pkg/linux_amd64/code.google.com/p/go.crypto/
cp $WORK/code.google.com/p/go.crypto/ssh.a
/home/dfc/pkg/linux_amd64/code.google.com/p/go.crypto/ssh.a

Is this the example you were thinking of ? Maybe we should take this
too another thread ?

[Kyle Lemons]

On Mon, Jan 7, 2013 at 2:06 AM, Dave Cheney <da...@cheney.net> wrote:

Hmm, I wasn't able to reproduce what I thought was the problem

Well that's annoying.  I can't reproduce now at tip, go1.0.3, or on the machine on which I (thought I) was able to reproduce it on (which is at some weird old-ish revision somewhere).  Looking at the code, it looks like only exact directory matches will override, not prefixes.  Unless I'm missing something somewhere.

I verified this by adding a "crypto/foobar" package in my GOPATH, and I can import and use it fine.  I was under the impression that we explicitly wanted to disallow this, because it means that adding a new stdlib package in a point release can break existing programs, which implies that the list of packages in the stdlib has to remain fixed for the go1 promise to apply... though I guess it is a tooling question, and tools aren't covered.