Meaning of imports in gccgo's .go_export output
Albert Strasheim
I'm working on some SCons tools to build Go code with both the
standard toolchain and gccgo.
To this end, I am trying to understand the import statement in
gccgo's .go_export output that one gets when compiling a package to an
ELF object file.
I am parsing this output to establish dependencies between packages so
that SCons only rebuilds what is needed and so that I can construct
the parameters needed for linking automatically.
With a project called foo and a package called bar, and using the -fgo-
prefix=foo option, I get the following in bar.o:
package bar;
prefix foo_bar;
priority 10;
import bar foo_bar.bar..import 10 math libgo_math.math..import 1
unicode libgo_unicode.unicode..import 1 syscall
libgo__.syscall..import 3 os libgo_os.os..import 4 io
libgo_io.io..import 5 strconv libgo_strconv.strconv..import 7 time
libgo_time.time..import 9;
Firstly,
import 1 syscall libgo__.syscall
seems a bit strange. Seems the go_prefix went missing there. Was that
intentional?
Then, I am trying to understand why package bar contains
import bar foo_bar.bar
again. This doesn't seem to happen for all my packages, though. If I
have a package baz the imports bar, baz doesn't have a
import baz foo_baz.baz
in its import line.
Any help appreciated.
Regards
Albert
P.S. I'm currently using gccgo r164388.
Graham Miller
https://launchpad.net/sconsgo
(It already has dependency management built).
graham
Albert Strasheim
On Sep 20, 5:29 pm, Graham Miller <graham.mil...@gmail.com> wrote:
> Could you build on top of this scons project:
> https://launchpad.net/sconsgo
> (It already has dependency management built).
application (and I'm trying to get a more general version of the
helper into the Go distribution), but the rest of my approach is a bit
different.
sconsgo has everything as a single tool. The approach I've taken is to
make separate tools for cgo, gopack, [68]g, [68]l and gccgo. I've
found this approach more flexible, especially when you want to
customise how things get built. I've also made some higher-level
functions, called GoPackage and GoCommand, to make it easy to build a
project that follows the src/{pkg,cmd} convention, and also to build
multiple projects that depend on each other, without installing
everything into GOROOT.
As soon as I have gccgo support properly fleshed out I'll release the
code.
Cheers
Albert
Ian Lance Taylor
> To this end, I am trying to understand the import statement in
> gccgo's .go_export output that one gets when compiling a package to an
> ELF object file.
The .go_export output has changed in the past and will change in the
future. I'm not making any commitment to keeping the same format.
Please bear that in mind if you want to parse it directly.
> I am parsing this output to establish dependencies between packages so
> that SCons only rebuilds what is needed and so that I can construct
> the parameters needed for linking automatically.
The gccgo compiler is going to want to do this too, so at some point
there will be a better way to determine which packages are imported.
That is not very high on my personal priority list, but of course it
would be great if somebody else wanted to tackle it.
> With a project called foo and a package called bar, and using the -fgo-
> prefix=foo option, I get the following in bar.o:
>
> package bar;
> prefix foo_bar;
> priority 10;
> import bar foo_bar.bar..import 10 math libgo_math.math..import 1
> unicode libgo_unicode.unicode..import 1 syscall
> libgo__.syscall..import 3 os libgo_os.os..import 4 io
> libgo_io.io..import 5 strconv libgo_strconv.strconv..import 7 time
> libgo_time.time..import 9;
>
> Firstly,
>
> import 1 syscall libgo__.syscall
>
> seems a bit strange. Seems the go_prefix went missing there. Was that
> intentional?
It was probably just a mistake on my part. I'm about to commit a patch
to fix it.
> Then, I am trying to understand why package bar contains
>
> import bar foo_bar.bar
>
> again. This doesn't seem to happen for all my packages, though. If I
> have a package baz the imports bar, baz doesn't have a
>
> import baz foo_baz.baz
>
> in its import line.
The import line lists all the initialization functions which must be run
at program startup time if you import that package. The numbers give
the order in which the import functions must be run, lowest to highest.
The list includes the initialization function for the current package,
but only if the current package actually has an initialization function.
A package with no init function and with no global variables that
require runtime initialization will not have an initialization function.
The list for a given package is the union of the lists for all imported
packages, plus the initialization function for the package itself if
any.
Note that in particular this means that not all imported packages are
listed on that line. There is currently no reliable way to determine
the set of imported packages.
Ian
Albert Strasheim
On Sep 20, 7:17 pm, Ian Lance Taylor <i...@google.com> wrote:
> The gccgo compiler is going to want to do this too, so at some point
> there will be a better way to determine which packages are imported.
> That is not very high on my personal priority list, but of course it
> would be great if somebody else wanted to tackle it.
>
> listed on that line. There is currently no reliable way to determine
> the set of imported packages.
What's the current thinking behind gccgo in the Go team? Should we be
looking to use it as our production compiler in the next 3 to 6
months, or are we better off sticking to the standard toolchain for a
while?
Another problem I've run into in my experiments is that we already
depend on libraries outside the Go distribution, like goprotobuf,
which doesn't have a gccgo build yet (AFAIK). A common build system
(included in the Go distribution?) is probably necessary to make it
easy to switch between the two toolchains across multiple projects.
Regards
Albert
Ian Lance Taylor
> What's the current thinking behind gccgo in the Go team? Should we be
> looking to use it as our production compiler in the next 3 to 6
> months, or are we better off sticking to the standard toolchain for a
> while?
I don't think there is one answer to that, as it depends on your plans
and expectations. What is preventing you from using it as your
production compiler now? It would be helpful if you reported them in
the issue tracker, to make it more likely that they will get fixed.
I'm currently the sole developer on gccgo so it's hard for me to predict
when any particular fixes will occur. Right now I'm working to get
gccgo merged into the mainline gcc distribution, and it should be part
of gcc 4.6. But of course the gcc release will inevitably lag language
and library changes in 6g.
> Another problem I've run into in my experiments is that we already
> depend on libraries outside the Go distribution, like goprotobuf,
> which doesn't have a gccgo build yet (AFAIK). A common build system
> (included in the Go distribution?) is probably necessary to make it
> easy to switch between the two toolchains across multiple projects.
Yes, that would be great. I haven't thought about that at all myself.
Ian
Albert Strasheim
On Sep 21, 4:14 pm, Ian Lance Taylor <i...@google.com> wrote:
> > What's the current thinking behind gccgo in the Go team? Should we be
> > looking to use it as our production compiler in the next 3 to 6
> > months, or are we better off sticking to the standard toolchain for a
> > while?
> I don't think there is one answer to that, as it depends on your plans
> and expectations. What is preventing you from using it as your
> production compiler now? It would be helpful if you reported them in
> the issue tracker, to make it more likely that they will get fixed.
Our setup looks something like this:
- 5 people and growing
- about 50 kLOC Go code, ~100 packages, and growing fast
- some integration with C libraries on Linux
- even though we are targeting Linux, most of our developers are on
Mac OS X
As I understand it, gccgo isn't going to be available on Mac OS X any
time soon, so developers that want to use it will have to log in to a
Linux server or run Linux inside VirtualBox.
However, for the 90% of our project that is pure Go, the developers on
Mac OS X would like to build and work with this code directly on their
machines, which means using the standard Go toolchain and some build
system (we started with Makefiles and we're moving to SCons).
Ideally, the Linux developers and the Hudson build server should be
using the same build system, and should be able to build with both
toolchains.
If I understand the limitations of gccgo correctly, we are going to
have to do quite a bit of work to get our code built (basically,
specifying a bunch of link lines by hand), since there is no reliable
way to determine the set of imported packages from gccgo's output.
I guess we could use the standard Go toolchain to build the code,
figure out the dependencies and then compile with gccgo. Maybe that's
a scheme I could look into.
We also have the problem that we will have to maintain a gccgo build
for our third-party dependencies (goprotobuf being the main one right
now).
> I'm currently the sole developer on gccgo so it's hard for me to predict
> when any particular fixes will occur. Right now I'm working to get
> gccgo merged into the mainline gcc distribution, and it should be part
> of gcc 4.6. But of course the gcc release will inevitably lag language
> and library changes in 6g.
Cheers,
Albert
Andrew Gerrand
> I guess we could use the standard Go toolchain to build the code,
> figure out the dependencies and then compile with gccgo. Maybe that's
> a scheme I could look into.
Another approach is to write a tool to read the dependancies from Go
source files. The packages under go/ in the standard library will help
you parse Go code. Also check out src/cmd/goinstall/parse.go.
Andrew
Albert Strasheim
On Sep 22, 12:43 am, Andrew Gerrand <a...@golang.org> wrote:
> > I guess we could use the standard Go toolchain to build the code,
> > figure out the dependencies and then compile with gccgo. Maybe that's
> > a scheme I could look into.
> Another approach is to write a tool to read the dependancies from Go
> source files. The packages under go/ in the standard library will help
> you parse Go code. Also check out src/cmd/goinstall/parse.go.
http://groups.google.com/group/golang-dev/browse_thread/thread/fd23e5a14e8f889c
http://groups.google.com/group/golang-dev/browse_thread/thread/e377a36cbbe183eb
but we have multiple projects, so we have to calculate dependencies
from a mix of code and compiled packages.
This is easy to do with 6g packages using "gopack p pkg.a __.PKGDEF",
but an equivalent doesn't exist for gccgo yet, as far as I understood
Ian.
Regards
Albert
Albert Strasheim
If I may revisit an old discussion...
On Sep 20 2010, 7:17 pm, Ian Lance Taylor <i...@google.com> wrote:
> The import line lists all the initialization functions which must be run
> at program startup time if you import that package. The numbers give
> the order in which the import functions must be run, lowest to highest.
> The list includes the initialization function for the current package,
> but only if the current package actually has an initialization function.
> A package with no init function and with no global variables that
> require runtime initialization will not have an initialization function.
> The list for a given package is the union of the lists for all imported
> packages, plus the initialization function for the package itself if
> any.
> Note that in particular this means that not all imported packages are
> listed on that line. There is currently no reliable way to determine
> the set of imported packages.
We've been building some of our code with gccgo for a while and the
only real pain point is this issue.
Do you have any new ideas on how to determine the set of imported
packages when dealing with a mix of source and binaries?
I think GCC 4.6 is in bugfix mode now, so an extra feature like this
will probably have to wait, right?
Regards
Albert
Ian Lance Taylor
gccgo is a new feature, so it can still be changed.
While it's obviously easy to get the list of imports from the source
code, there is no way to reliably get it from the binaries. It will
require a change to the export data. Want to try writing a patch? You
would need to change how imported_init_fns_ works, probably by renaming
it, and add a way for the imports line in export data to indicate an
imported package with no initialization function.
Ian