What is the fundamental unit of linking in Go?

[luka....@gmail.com]

I have come up against something a bit strange and interesting that I didn't really expect from the Go compiler.

I have a main which uses a structure created in a package, which contains a whole load of references to other packages.

What I discovered is that even though only one of these included packages is used, the whole set still gets included in the binary.

Am I right in saying that packages create a linker object file, as a whole, and every package imported is also linked?

In the case of my application it means I now have to manually break up a whole load of nice pretty neat declarations into pieces that return generator functions, and each part has to come from a different package, meaning a folder and one tiny little source file.

I couldn't find any reasonable terms that explained Go's linking mechanisms but it dawned on me as I thought about how each folder in the tree creates a binary object that is stitched together in the final binary, that there's really no way around this, and I suppose it makes nice neat little bite sized source files but for my case it was a surprise I hadn't expected.

After building a giant conglomeration I realised for some cases (namely, a backend for a mobile app) that one has to design the folder tree to split the parts. It's something to keep in mind for multi-function binaries if you want to also offer single function binaries for specific parts that are only needed alone.

I am probably, as usual, doing something unconventional and running up against things that nobody else ever thought of and discover something about a system that isn't obvious until you think about it, when one intends to do some particular thing.

In terms of memory utilisation, there is really not a great deal of difference in terms of performance, I presume, except providing the ability to cut up the whole thing into small pieces, a tiny amount extra on disk but I presume the final memory use, since all is a monolithic (more or less) static binary, that once it unwraps all the entry points and relative references that there isn't (much) extra in memory. I suppose also if one is targeting a constrained environment, it would make sense to use -s -w to strip out the debug symbols.

[Jan Mercl]

Please provide a minimal, self contained demonstration code that reproduces the problem. Then it should be hopefully easy to either explain why things work they do or it'll be a nice test case for the fix to the issue this may actually be. Thanks.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/36de6c0d-23fe-4dfa-ad9a-89d9734c9701%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[luka....@gmail.com]

I am pretty sure that every folder in a Go code repository creates one binary object, maybe two with a test package alongside it. It just didn't occur to me that it would not trace the execution path of the main, and where each of the closures/function references in this case are stored in a map[string]*functype.

If there could be an optimisation that omits them that would be cool but I wasn't really thinking about it that way when I posted about it but more just that 1 package = 1 module = 1 object. I unfortunately never had as much time as I wanted when I first found Oberon to have already learned this.

No, just more highlighting the correspondence between binaries and the folder tree as it replaces a makefile.

I could probably avoid splitting the packages if I used tags but they are not so easy to deal with and I'd rather learn the lesson of structuring. Plus when I started I had in mind all-in-one and not to be able to split things. I previously ran into another gotcha you can get when you start out to write a main package and then realise you want to be able to integrate its types and methods into another package. I would just say that maybe it would be a good thing to put at the beginning of a course or tutorial to highlight this pervasive but invisible factor when building software and how it creates a lot of work when you don't think in terms of modules and get stuck in the monolithic mindset.

On Friday, June 28, 2019 at 8:26:11 PM UTC+2, Jan Mercl wrote:

Please provide a minimal, self contained demonstration code that reproduces the problem. Then it should be hopefully easy to either explain why things work they do or it'll be a nice test case for the fix to the issue this may actually be. Thanks.

On Fri, Jun 28, 2019, 20:10 <luka...@gmail.com> wrote:

I have come up against something a bit strange and interesting that I didn't really expect from the Go compiler.

I have a main which uses a structure created in a package, which contains a whole load of references to other packages.

What I discovered is that even though only one of these included packages is used, the whole set still gets included in the binary.

Am I right in saying that packages create a linker object file, as a whole, and every package imported is also linked?

In the case of my application it means I now have to manually break up a whole load of nice pretty neat declarations into pieces that return generator functions, and each part has to come from a different package, meaning a folder and one tiny little source file.

I couldn't find any reasonable terms that explained Go's linking mechanisms but it dawned on me as I thought about how each folder in the tree creates a binary object that is stitched together in the final binary, that there's really no way around this, and I suppose it makes nice neat little bite sized source files but for my case it was a surprise I hadn't expected.

After building a giant conglomeration I realised for some cases (namely, a backend for a mobile app) that one has to design the folder tree to split the parts. It's something to keep in mind for multi-function binaries if you want to also offer single function binaries for specific parts that are only needed alone.

I am probably, as usual, doing something unconventional and running up against things that nobody else ever thought of and discover something about a system that isn't obvious until you think about it, when one intends to do some particular thing.

In terms of memory utilisation, there is really not a great deal of difference in terms of performance, I presume, except providing the ability to cut up the whole thing into small pieces, a tiny amount extra on disk but I presume the final memory use, since all is a monolithic (more or less) static binary, that once it unwraps all the entry points and relative references that there isn't (much) extra in memory. I suppose also if one is targeting a constrained environment, it would make sense to use -s -w to strip out the debug symbols.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.

[Ian Lance Taylor]

On Fri, Jun 28, 2019 at 11:39 AM <luka....@gmail.com> wrote:
>
> I am pretty sure that every folder in a Go code repository creates one binary object, maybe two with a test package alongside it. It just didn't occur to me that it would not trace the execution path of the main, and where each of the closures/function references in this case are stored in a map[string]*functype.
>
> If there could be an optimisation that omits them that would be cool but I wasn't really thinking about it that way when I posted about it but more just that 1 package = 1 module = 1 object. I unfortunately never had as much time as I wanted when I first found Oberon to have already learned this.
>
> No, just more highlighting the correspondence between binaries and the folder tree as it replaces a makefile.
>
> I could probably avoid splitting the packages if I used tags but they are not so easy to deal with and I'd rather learn the lesson of structuring. Plus when I started I had in mind all-in-one and not to be able to split things. I previously ran into another gotcha you can get when you start out to write a main package and then realise you want to be able to integrate its types and methods into another package. I would just say that maybe it would be a good thing to put at the beginning of a course or tutorial to highlight this pervasive but invisible factor when building software and how it creates a lot of work when you don't think in terms of modules and get stuck in the monolithic mindset.

Sorry, I'm not really sure what you mean.

That said, the linker does discard functions that are never
referenced. But it does not in general discard methods that are never
referenced, because that is very hard to detect. Code can use the
reflect package to get a reflect.Value of that type, and can then call
methods using reflect.Value.Method. So it's difficult in general to
know whether a method is used or not.

Ian

[luka....@gmail.com]

Yes, I thought about it and I am quite sure that it would be very complex code detecting referenced but unused code.

I am just starting with learning how to break things apart properly. Go makes it easy to modularise but old bad habits to make monolithic stuff don't dissolve overnight, and in my case I am working with a bitcoin client codebase that is very monolithic...

Once things are properly decomposed it makes composition a lot easier and bugs are easier to find and fix when things are more properly isolated.

On Friday, June 28, 2019 at 9:40:12 PM UTC+2, Ian Lance Taylor wrote: