Best way to structure/package a Go server
Peter Bourgon
trouble finding the right keywords to search on.
I've got a server, comprised of a Go package (foo.go) that implements
the logic, plus a separate 'package main' server.go that imports foo
and runs as a daemon. I suspect this is a fairly common structure.
Right now it is pretty awkward to run and use. I can do a standard
'make' to build foo, but have to manually 8g/8l the server.go.
Is there a way to structure the Makefile to handle everything
automatically? That is, generate both the library and the executable?
I guess this is all predicated on the assumption that goinstall only
really handles libraries. Is that assumption correct, or can I finagle
goinstall to work with this somehow?
Or is there some totally different and superior way to package,
distribute and build this sort of thing?
David Nesting
Andrew Gerrand
you're adding to the global Go package namespace (under $GOROOT/pkg/).
If you have some project foo with component packages bar and baz, you
might call the packages
foo/bar
foo/baz
So you would include from the $GOROOT/src/Make.pkg Makefile for those
packages, with TARG=foo/bar, etc.
To build and install your main package, set up a Makefile that
includes from $GOROOT/src/Make.cmd (see $GOROOT/src/cmd/godoc/Makefile
for an example).
Then to build your whole project, you could write a simple bash script
to call make for each of the packages, followed by the main package.
The Go Makefiles were written for the core Go code-base, not really
with 3rd-party stuff in mind. You can use them (as I've just
described), but you may also choose write your own.
It would be nice to have a tool to manage this more simply, though.
Andrew
Peter Bourgon
> $GOROOT/src/Make.cmd
Aha! This was the example I was looking for. Thanks; if I come up with
something elegant that combines this + Make.pkg, I'll post it to the
list for review.
Russel Winder
[ . . . ]
> So you would include from the $GOROOT/src/Make.pkg Makefile for those
> packages, with TARG=foo/bar, etc.
>
> To build and install your main package, set up a Makefile that
> includes from $GOROOT/src/Make.cmd (see $GOROOT/src/cmd/godoc/Makefile
> for an example).
>
> Then to build your whole project, you could write a simple bash script
> to call make for each of the packages, followed by the main package.
>
> The Go Makefiles were written for the core Go code-base, not really
> with 3rd-party stuff in mind. You can use them (as I've just
> described), but you may also choose write your own.
>
> It would be nice to have a tool to manage this more simply, though.
There is an implicit implication here that Go mandates the use of Make
as the build tool. Is this the case or is use of SCons or Waf going to
be feasible?
--
Russel.
=============================================================================
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel...@ekiga.net
41 Buckmaster Road m: +44 7770 465 077 xmpp: rus...@russel.org.uk
London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Andrew Gerrand
> There is an implicit implication here that Go mandates the use of Make
> as the build tool. Is this the case or is use of SCons or Waf going to
> be feasible?
All implications are implicit. ;-)
When I said "or you could write your own," what I actually _meant_ was
that you can build them however you like. As long as GOARCH, GOOS, and
GOROOT are set, that should be enough for any build tool to know how
to invoke the go compiler and linker.
I personally use a little shell script to build most small programs as
I don't grok make as well as I should.
Andrew
David Leimbach
package.
I made an "all.bash" to build it all every time, because I didn't
worry about the dependencies of recompiling, because the compiler is
so fast to just rebuild EVERY TIME :-).
#!/bin/bash
buildreq () {
(cd reqhandler && make clean && make install)
}
buildpoller () {
(cd poller && make clean && make install)
}
buildexpect () {
(cd expect && make clean && make install)
}
buildexpect && buildreq && buildpoller && make clean && make
===================
Here is my upper level makefile for a program called testrh.go
===================
include $(GOROOT)/src/Make.$(GOARCH)
GC+=-B
TARG = testrh
GOFILES =\
testrh.go
include $(GOROOT)/src/Make.cmd
===================
Here is a makefile for one of the packages:
===================
include $(GOROOT)/src/Make.$(GOARCH)
GC+=-B
TARG = expect
GOFILES =\
expect.go
include $(GOROOT)/src/Make.pkg
===================
Those "included" make configurations make things super easy to deal
with.
David Leimbach
On Jun 2, 8:55 am, Andrew Gerrand <a...@golang.org> wrote:
> On 2 June 2010 17:52, Russel Winder <rus...@russel.org.uk> wrote:
>
> > There is an implicit implication here that Go mandates the use of Make
> > as the build tool. Is this the case or is use of SCons or Waf going to
> > be feasible?
>
> All implications are implicit. ;-)
>
> When I said "or you could write your own," what I actually _meant_ was
> that you can build them however you like. As long as GOARCH, GOOS, and
> GOROOT are set, that should be enough for any build tool to know how
> to invoke the go compiler and linker.
>
> I personally use a little shell script to build most small programs as
> I don't grok make as well as I should.
>
> Andrew
files make the work you have to do so incredibly minimal to play nice
with the Go system.
Building external code against Go might be trickier, but I've not seen
any evidence of that.
Dave
>
>
>
> > --
> > Russel.
> > =========================================================================== ==
> > Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.win...@ekiga.net
David Nesting
(cd reqhandler && make clean && make install)
Rob 'Commander' Pike
David Leimbach
On Jun 2, 9:44 am, David Nesting <da...@fastolfe.net> wrote:
> > (cd reqhandler && make clean && make install)
>
> This will install your packages beneath $GOROOT, right?
feel free to smack me if they can find an obvious (hopefully a non-
obvious) hole in my thinking.
Is it intended that
> $GOROOT be the location that all of these in-progress packages get installed
> to?
packages don't conflict with the names of Go's packages. Keep in mind
that your Go sandbox is just that... yours. Do whatever you want to
it.
This seems ugly and prone to polluting $GOROOT with a bunch of
packages
> that are half-completed, renamed or unusable in isolation, but it seems to
> be the path of least resistance.
Go tree, but if you don't want to include them in your build, don't do
it. A single broken package that nothing else depends on can not
break packages or applications that do not depend on it, so it's
safe! Also keep in mind that this arrangement avoids the need for any
additional INCLUDE_PATH or LIBRARY_PATH variables that can be
difficult to manage.
Unusable in isolation I don't understand, because if you track your
build time dependencies with a Makefile, then that package will not be
in isolation. In fact, you've installed it in your own Go tree, and
it is in excellent company!
Here's my Go-philosophy...
Disk is pretty cheap these days. My iMac I just bought shipped with a
1TB drive that I actually think is ludicrously big for any single
computer user, such as myself. If I had 100 users on my iMac, each
could have their own GOROOT, each weighing in at less than 400 MB, and
it would have almost no impact on the system in terms of my worrying
about disk utilization. With this in mind, ask yourself "why
shouldn't each user have their own sandbox for GO?"
>
> Is there an easy way to build my project's packages and import them locally
> without requiring that they be installed under $GOROOT?
would still install them in GOROOT, because it does not hurt anything
for my development tasks. All you must do is watch the filesystem for
namespace clashes, and you're golden!
Have I mentioned yet that I love this lack of DLL hell?
Dave
>
> David
Gustavo Hexsel
Gustavo Hexsel
David Leimbach
On Jun 2, 10:25 am, Gustavo Hexsel <ghex...@gmail.com> wrote:
> The reason why it may be ugly is because there's no support for versioning
> and the recommended namespaces are fairly small (packages named "math" and
> "server" and things like that).
>
I've not tried it yet, but I believe there are packages that follow
this pattern already in Go (for example the packages under
"compress"). I suppose I should look into that and see if it's hard
to structure my stuff that way as well.
Versioning is punted to Mercurial for now, and I really don't know the
difference between an "installed" Go package and one that's just there
for building against. Remember that Go statically links all the
needed packages into your binary, so the only thing you really have to
version is your own builds of applications, and your own builds of
packages. The standard Go packages are versioned by the release
number that Google has assigned which is accessible from your Go
sandbox via "hg identify". Maybe that's not the future of Go, but I
find this workable for now, and have no problems using it going
forward.
Hg makes it really easy for me to have a "master clone" that I make
all other local clones from so I can easily make sure I don't drift
from one particular Go release to the next. If I move the master
clone to the new release, I can update all the sandboxes from it. I
can even push my own packages to my master clone if I want them easily
accessible inside my organization. Namespace clashing is the tricky
bit :-)
Rob 'Commander' Pike
On Jun 2, 2010, at 10:38 AM, David Leimbach wrote:
>
>
> On Jun 2, 10:25 am, Gustavo Hexsel <ghex...@gmail.com> wrote:
>> The reason why it may be ugly is because there's no support for versioning
>> and the recommended namespaces are fairly small (packages named "math" and
>> "server" and things like that).
>>
>
> Can't I use something like "myorganization/mypackage"?
That's the intent. Google-internal libraries, for instance, are all imported from a path beginning "google".
-rob
Russel Winder
On Wed, 2010-06-02 at 10:12 -0700, David Leimbach wrote:
>
> On Jun 2, 9:44 am, David Nesting <da...@fastolfe.net> wrote:
> > On Wed, Jun 2, 2010 at 09:28, David Leimbach <leim...@gmail.com> wrote:
> > > (cd reqhandler && make clean && make install)
> >
> > This will install your packages beneath $GOROOT, right?
>
> Yes, read on to see why I think that's actually a good thing. Someone
> feel free to smack me if they can find an obvious (hopefully a non-
> obvious) hole in my thinking.
[ . . . ]
I guess the problem will come when things start to be packaged for
operating system distributions. There is then a conflict between
distribution and local variation. Python and Emacs handle this by
defining a special directory and requiring all installation to be done
by root. This is a sys admin nightmare.
The current structure works fine if the one and only method of
distributing Go is via Mercurial, but then many organization won't
accept its presence as they only allow things that can be handled by
operating system packages.
--
Russel.
=============================================================================
Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel...@ekiga.net
Kyle Lemons
packages" and have linked example below. To generate makefiles, the
plug-in analyzes all files and determines dependencies and generates
this with the build order the way it needs to be. You can simplify it
a lot if (for instance) you know you don't have spaces in your GOROOT
and HOME, and if you always have pkg.foo for package pkg you can omit
the build commands under both rules in Objects below in favor of an
implicit rule ($(OBJDIR)%.$O). If it would be helpful, I can write
(in Go, of course) a quick program that will generate makefiles that
does something similar. (This one's done by Java, of course, as it's
part of an Eclipse plugin.)
http://pastie.org/989399
Gustavo Hexsel
Kyle Lemons
and the makefile that it generates looks something like this:
http://pastie.org/989808
The only things that need to be changed to work with any project are
the $PACKAGE_OBJECT and $PACKAGE_SOURCE variables, and the (dependency
ordered) list of packages in $PACKAGES. It uses makefile magic to
generate everything else.
If you don't want to have to mess with it, godepgen will make it for
you:
http://code.google.com/p/kylelemons/source/browse/trunk/go/godepgen/godepgen.go
David Roundy
>> This seems ugly and prone to polluting $GOROOT with a bunch of
>> packages that are half-completed, renamed or unusable in isolation, but it seems
>> to be the path of least resistance.
>
> Yes this means that libraries that are in-development end up in your
> Go tree, but if you don't want to include them in your build, don't do
> it. A single broken package that nothing else depends on can not
> break packages or applications that do not depend on it, so it's
> safe! Also keep in mind that this arrangement avoids the need for any
> additional INCLUDE_PATH or LIBRARY_PATH variables that can be
> difficult to manage.
The trouble is that it makes things very complicated if you want to
have two different versions of your code, e.g. in two different
directories. I very often want to go back and test my code using
older versions, and that's pretty hard if the build process involves
installing packages into the global location. One answer is to not
use packages (just put everything in package main, or a single
package), and maybe we don't often need packages, since the go
compiler is so fast, but that really just sidesteps the question.
Personally, I'm not satisfied with a build process that won't let me
isolate the code that I'm working on in this way. But fortunately,
it's quite easy to achieve by avoiding creating my own packages.
David
Giles Lean
David Roundy <rou...@physics.oregonstate.edu> wrote:
> The trouble is that it makes things very complicated if you want to
> have two different versions of your code, e.g. in two different
> directories. I very often want to go back and test my code using
> older versions, and that's pretty hard if the build process involves
> installing packages into the global location.
I practically always have multiple go installations:
o my current "everything" version ($HOME/go, $HOME/gobin)
o the current top of tree, unchanged (in case I suspect a Go bug)
o the current release (ditto)
o one for each CL I'm working on, originally so that it could
be stable and independent of other work during the review
process; now I think I could manage all my CLs in one tree
with Mercurial Queues, but it's simpler and allows faster
context switching to keep them separate
o one for each independent project that won't become a CL
All I have to remember to do is run . ./set_env when I change
directory to work on a different project, which sets $GOROOT
and $GOBIN. This is no harder than setting e.g. a library
search path.
If I'm not going to work on something for a while, clean.bash
will reduce the footprint. If I give up on something I can
just save the diffs ... which still gives me the chance to
go back but saves the space of the whole tree.
One of the reasons distributed source code management has
taken off is because disk space is now cheap. My _notebook_
has 500GB. (Come to that, it's likely all my go trees would
fit on a USB stick.)
The days where only people who'd bought *lots* of disk space
could compile, say, X11 and whole software teams worked on a
single system with say 1.2GB (GB, not TB) are _over_.
Giles
Giles Lean
David Leimbach <lei...@gmail.com> wrote:
> I see no advantage in using anything but Make with Go when
> the include files make the work you have to do so incredibly
> minimal to play nice with the Go system.
Once you start using cgo, it becomes very difficult not to go
with the standard Makefiles, I found.
(If someone has figured an easy way, that might be
interesting: right now you need the Go source tree installed
to get those Makefiles, not just $GOBIN and packages under
$GOROOT.)
But perhaps it doesn't matter: the Makefiles are BSD licensed
and the source tree isn't that big v. current disk sizes, and
once a binary is built it's staticly linked. It's mostly that
it feels odd to be writing code under a different license yet
which has Makefiles with "include $(GOROOT)/src/Make.pkg" in
them.
Giles
Kyle Lemons
with that of Giles. Neither or both of us might be right, only you
can decide what's best for you.
On Jun 2, 10:51 pm, Giles Lean <giles.l...@pobox.com> wrote:
> I practically always have multiple go installations:
>
> o my current "everything" version ($HOME/go, $HOME/gobin)
> o the current top of tree, unchanged (in case I suspect a Go bug)
> o the current release (ditto)
that constitutes development all its own, but having multiple
installations for more projects just seems excessively complicated for
me to keep track of.
I want to have a global (in my case, /opt/go) installation against
which all of my "independent projects" build. When I goinstall
packages, I use them and they're my go-to package for performing the
same task in another project, and I wouldn't want to goinstall it
again there. Perhaps this comes from my time with C, but I have
always liked the idea that I have a directory (~/dev/go for me) under
which all of my projects reside, and each project contains only what's
necessary for that project. If I install a library, I want that to go
into the global installation, but if I am writing a utility package I
want it to stay local. When the project builds, its objects should go
into the local _obj/ folder and any packages in the same directory go
into _obj/ as well. No namespace cluttering, no make install,
nothing.
This all means that I have reawoken my Makefile skills again, but
that's why Make exists, and I've now developed a makefile that is easy
to maintain and doesn't require a LIBRARY_PATH or environment
changes. In fact, as in the makefile example I posted earlier
( http://pastie.org/989808 ), the only thing I ever use the $GOROOT/
src/ makefiles for is getting the architecture-dependent stuff (6g/
8g, .6/.8, etc), I don't use its targets for most anything. Even
those could reasonably be derived from the settings for $(GOARCH)
without needing to consult the files, it just makes life easier.
It's entirely possible that I'll run into a project (potentially even
very soon) that will change my development practices, but so far it's
worked for me.
> If I'm not going to work on something for a while, clean.bash
> will reduce the footprint. If I give up on something I can
> just save the diffs ... which still gives me the chance to
> go back but saves the space of the whole tree.
has locally applicable makefiles that apply recursively if necessary)
and rm -rf for the latter. I keep everything in subversion (I've been
using it so long and have never moved over to mercurial), so I can
check it out if I want to come back to it. Probably only because I
have a bad habit of deleting or overwriting files that I worked long
and hard on, but that's not the point ;-).
> One of the reasons distributed source code management has
> taken off is because disk space is now cheap. My _notebook_
> has 500GB. (Come to that, it's likely all my go trees would
> fit on a USB stick.)
than I ever expect to. As much as "diskspace is cheap" has become a
standby, I still think minimizing it should be key, because as digital
media gets bigger, it also gets easier to use up disk space. I am
constantly in the red on my poor macbook.
> The days where only people who'd bought *lots* of disk space
> could compile, say, X11 and whole software teams worked on a
> single system with say 1.2GB (GB, not TB) are _over_.
work.
~Kyle
Giles Lean
Kyle Lemons <ky...@kylelemons.net> wrote:
> First off, my comments below are just to contrast my
> development style with that of Giles. Neither or both of us
> might be right, only you can decide what's best for you.
Thanks Kyle. Always good to see alternatives. Indeed I
cheat a little and some projects share a build, where I'm not
hitting language or package instabilities (which I'm not as
much lately as I was a couple of months back).
> In fact, as in the makefile example I posted earlier (
> http://pastie.org/989808 ), the only thing I ever use the
> $GOROOT/ src/ makefiles for is getting the
> architecture-dependent stuff (6g/ 8g, .6/.8, etc), I don't
> use its targets for most anything. Even those could
> reasonably be derived from the settings for $(GOARCH)
> without needing to consult the files, it just makes life
> easier.
You don't use cgo? cgo is pretty hard wired to use the src/
Makefile bits, or else such a close copy of them that it'd be
cut&paste and a derivative work I suspect.
Also, testing? Is that picked up too?
> Sadly, I have never met the harddrive I can't fill.
I think it was Ken Thompson who said that the steady state
of disks is full.
These days I usually manage some headroom, but hit the limit
and upgraded my MacBook Pro last week from 200GB to 500GB. I
suspect I'd have had a less miserable time upgrading if the
machine hadn't taken a knock early on in its life which has
left one corner visibly bent. :-/
Giles
Kyle Lemons
>
> > In fact, as in the makefile example I posted earlier (
> > architecture-dependent stuff (6g/ 8g, .6/.8, etc), I don't
> > use its targets for most anything. Even those could
> > reasonably be derived from the settings for $(GOARCH)
> > without needing to consult the files, it just makes life
> > easier.
>
> You don't use cgo? cgo is pretty hard wired to use the src/
> Makefile bits, or else such a close copy of them that it'd be
> cut&paste and a derivative work I suspect.
>
> Also, testing? Is that picked up too?
>
http://pastebin.com/FQhErr5T
I haven't fooled around with testing yet, but I labor under the belief
that when I need it it will be WAY easier to simplify into my own
Makefile than the CGO stuff. If you'll notice, in the makefile above,
I treat cgo packages the same (from the "user" perspective) as other
packages, I just also put the package name in CGOPKGS and specify an
extra *_CFILES (because I'm not linking against a library, I am
linking against a local .o file, but changing it to work with e.g. -
l<lib> instead would be trivial). Oh, also note that I moved a few OS-
specific things into $(GOROOT)/src/Make.$(GOOS) from Make.pkg.
Actually, just one thing, _CGO_LDFLAGS_darwin, but I hope eventually
it's there for everyone, as that makes the most sense.
Maybe when I get this working with test, I should post the makefile
somewhere, as it should ideally work with *any* local project layout.
I also still need to update my godepgen and GoClipse to generate the
above makefile. All things in time, I suppose.
~Kyle
Giles Lean
Kyle Lemons <ky...@kylelemons.net> wrote:
> As far as cgo goes:
> http://pastebin.com/FQhErr5T
Hey, thanks! Dare I presume we're free to pick this up and
use it?
> I haven't fooled around with testing yet, but I labor under
> the belief that when I need it it will be WAY easier to
> simplify into my own Makefile than the CGO stuff.
It should be. And I'd encourage you to get into testing;
the Go test support is reasonably basic, but it works very
well.
I know it has helped me a lot when working on a large
library: I can start some parts top down and test them,
and other parts bottom up and test them, while I'm still
working on how they'll end up glued together.
(And when I say it's helped me a lot, I mean that it's not
just confirmed for me that something works, but (ahem) found
a bug or two.)
> Maybe when I get this working with test, I should post the
> makefile somewhere, as it should ideally work with *any*
> local project layout. I also still need to update my
> godepgen and GoClipse to generate the above makefile. All
> things in time, I suppose.
If you're willing, and are relaxed enough in your licensing,
I'd be appreciative. I've nothing against the makefiles
included in the Go tree, but given most Linux distributions'
tendancies to separate out "-devel" bits _and_ with the gcc Go
frontend coming along, it would be nice to have a self
contained source tree for a package that doesn't need Go
compiler source present and unpacked to build the package.
Perhaps I'm being overly fussy and sticking with "old think",
but Go presently breaks a number of conventions about what the
OS provides, what the compiler provides, and what third party
libraries provide. (This shouldn't be construed as support
for those divisions, but they exist, and Go is "different".)
In the Go team's place I'd have made the same or similar
shortcuts I am sure; philosophically I regard a system without
a working compiler as "broken"; and having the compiler source
is a benefit, but I doubt the whole world will agree. While
I'm reasonable and rational, there's always _someone_ in the
world who most unreasonably and illogically disagrees. :-) :-)
Giles
Kyle Lemons
Pay no attention to the man behind the curtain... and by that, I mean the 11-deep nesting about halfway in. Lol. Check the doc comments at the top for brief explanations about how it works. Take special note that if you're using it for cgo, you need to make some changes to the Make.* files for your GOOS and GOARCH, as I didn't want to pollute the makefile with error-prone and ugly grepping of Make.pkg.
As far as I know, and I haven't tested this with any major cgo stuff, this should be able to generate a makefile for pretty much any well-structured go source directory with at most one destination binary.
~Kyle