Go execution modes

[Ian Lance Taylor]

I've written a document describing future directions for Go support
for shared libraries and related ways of building Go programs. This
is not the actual technical work. It's a proposal for how the
technical work should be done.

https://docs.google.com/a/golang.org/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit#

Please reply to this e-mail message with any comments.

Thanks.

Ian

[Taru Karttunen]

-buildmode=c-archive

Is there a reason to expose symbols (and thus probably generate
wrappers) for exported symbols in all packages as opposed to main?

A flag to limit those symbols could be very handy.

Also a -buildmode=guiexe for "-ldflags '-H windowsgui'" could be nice.

Why is it
func (p Plugin) Lookup(name string) (interface{}, error)
and not
func (p Plugin) Lookup(name string, interfaceptr interface{}) error
and
func (p Plugin) LookupC(name string, interfaceptr interface{}) error
like the various unmarshalling APIs are?

Thus the example code would become (notice casts going away):
var fc func()*C.char
e := p.LookupC("version", &fc)
// handle errors
fmt.Println(C.GoString(f))
var f func()string
e = p.Lookup("Version", &f)
// handle errors
fmt.Println(f())

Does the Go runtime grab a global lock for running init functions of a
loaded plugin? Currently init functions afaik run in a single
go-routine and thus may not be safe for concurrent use.

- Taru Karttunen

[Ian Lance Taylor]

On Thu, Sep 4, 2014 at 8:43 AM, Taru Karttunen <tar...@taruti.net> wrote:
> On 03.09 14:06, Ian Lance Taylor wrote:
>> I've written a document describing future directions for Go support
>> for shared libraries and related ways of building Go programs. This
>> is not the actual technical work. It's a proposal for how the
>> technical work should be done.
>>
>> https://docs.google.com/a/golang.org/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit#
>>
>> Please reply to this e-mail message with any comments.
>>
>
> -buildmode=c-archive
>
> Is there a reason to expose symbols (and thus probably generate
> wrappers) for exported symbols in all packages as opposed to main?

The idea here is that the main program serves a way to describe the
packages that should be included, via import statements. I don't see
a particular need to require that all exported symbols be in the main
package. That would make it painful for a single archive to group
together several different libraries.

> A flag to limit those symbols could be very handy.

Remember that the only exported symbols are those explicitly marked
with a //export comment, as described at http://golang.org/cmd/cgo/ .
In the future it may be necessary to add some way to limit the set of
exported symbols. However, it may never be needed, and I don't think
it should be present at the start.

> Also a -buildmode=guiexe for "-ldflags '-H windowsgui'" could be nice.

Thanks, I added a note to the doc.

> Why is it
> func (p Plugin) Lookup(name string) (interface{}, error)
> and not
> func (p Plugin) Lookup(name string, interfaceptr interface{}) error
> and
> func (p Plugin) LookupC(name string, interfaceptr interface{}) error
> like the various unmarshalling APIs are?
>
> Thus the example code would become (notice casts going away):
> var fc func()*C.char
> e := p.LookupC("version", &fc)
> // handle errors
> fmt.Println(C.GoString(f))
> var f func()string
> e = p.Lookup("Version", &f)
> // handle errors
> fmt.Println(f())

Thanks. That is clearly a good idea for LookupC, and I changed the
doc accordingly.

For Lookup I'm not sure. For Lookup the type of the function should
be known; there should be no need to pass the type in when calling
Lookup. Making it work that way is not clearly an advantage of the
current approach in the doc. Happy to hear other opinions.

> Does the Go runtime grab a global lock for running init functions of a
> loaded plugin? Currently init functions afaik run in a single
> go-routine and thus may not be safe for concurrent use.

It clearly can not grab a global lock, because an init function of a
plugin may itself load another plugin. But you're right that this
will have to be handled with care.


Thanks for the comments.

Ian

[Bryan Turley]

I think Plugin should be an interface for odd cases.

The oddest case I can currently think of is opengl on windows

opengl32, err := plugin.Open("opengl32.dll")

Won't work with a predefined system plugin struct because it would need to be written as

func (p *plgnWinGL) LookupC(name string, valptr interface{}) (err error) {
    var funcptr uintptr
    // ask special gl specific dlsym func
    funcptr, err  = wglGetProcAddress(name)
    // handle err

    if funcptr != 0 {
        // set valptr
        return // yay
    }

    // if we get here funcptr == 0 but the func may still exist in the original dll
    // so we call the actual system dlsym
    funcptr, err = win32.GetProcAddress(p.dll, name)
    // handle err
    // set valptr if funcptr != 0
    return // yay?
}

All <= opengl 1.3 (maybe 1.4) symbols come from opengl32.dll but newer stuff comes from wglGetProcAddress().
It is this way to override Microsoft doing their normal cripple the competition thing.
See http://www.opengl.org/wiki/Load_OpenGL_Functions#Windows for more (less?) info

[Ian Lance Taylor]

On Thu, Sep 4, 2014 at 11:03 AM, Bryan Turley <bryan...@gmail.com> wrote:
>
> I think Plugin should be an interface for odd cases.

I suppose I think it should be an interface for plugins. I don't see
how it could handle every case. I don't even know what that would
look like.

> The oddest case I can currently think of is opengl on windows

It sounds like you know how to handle that case without using the
plugin package at all.

Ian

[Steve McCoy]

I'm pretty sure wglGetProcAddress is all you need. It uses opengl32.dll. At least, GLEW uses only the wgl function and seems to get everything right.

Anyhow, this kind of thing seems outside the scope of a plugin package, because OpenGL isn't a plugin. Programs that use it depend on it being there.

[Bryan Turley]

On Thursday, September 4, 2014 2:33:29 PM UTC-5, Steve McCoy wrote:

I'm pretty sure wglGetProcAddress is all you need. It uses opengl32.dll. At least, GLEW uses only the wgl function and seems to get everything right.

glew uses the definitions from the original header file for the functions that wglGetProcAddress won't return.
https://github.com/nigels-com/glew/blob/master/auto/src/glew_head.h#L192 onward for a good bit

See http://www.opengl.org/wiki/Load_OpenGL_Functions#Windows for more (less?) info

Anyhow, this kind of thing seems outside the scope of a plugin package, because OpenGL isn't a plugin. Programs that use it depend on it being there.

opengl and it's extensions are definitely plugins, especially the extensions.

Ian is correct though, we are already doing this with cgo/syscall.
I thought this was about loading dynamic code, go compiled or not.
C programs loading Go libs was mentioned.
Perhaps I misread the doc...

[gerald...@gmail.com]

I read that you plan to support: Go code linked into, and called from, a non-Go program

It seems you can call golang code from c++? How does it work for non-go programs which can only load functions via c/c++ dynamic link libs?

[Ian Lance Taylor]

On Fri, Sep 5, 2014 at 6:04 AM, <gerald...@gmail.com> wrote:
>
> I read that you plan to support: Go code linked into, and called from, a non-Go program
>
> It seems you can call golang code from c++? How does it work for non-go programs which can only load functions via c/c++ dynamic link libs?

If I understand your question, that is covered in the doc. You are
describing the mode "Go code linked into a shared library loaded as a
plugin by a program (Go or non-Go) that supports a C style plugin
API", and that is supported using -buildmode=c-shared.

Ian

[gerald...@gmail.com]

could you load the plugin to matlab for example? 
http://www.mathworks.de/de/help/matlab/ref/loadlibrary.html

[Ian Lance Taylor]

On Fri, Sep 5, 2014 at 7:31 AM, <gerald...@gmail.com> wrote:
>
> could you load the plugin to matlab for example?
> http://www.mathworks.de/de/help/matlab/ref/loadlibrary.html

I know nothing about matlab, so let me turn the question around: what
makes you think that you couldn't?

Ian

> On Friday, September 5, 2014 3:31:32 PM UTC+2, Ian Lance Taylor wrote:
>>
>> On Fri, Sep 5, 2014 at 6:04 AM, <gerald...@gmail.com> wrote:
>> >
>> > I read that you plan to support: Go code linked into, and called from, a
>> > non-Go program
>> >
>> > It seems you can call golang code from c++? How does it work for non-go
>> > programs which can only load functions via c/c++ dynamic link libs?
>>
>> If I understand your question, that is covered in the doc. You are
>> describing the mode "Go code linked into a shared library loaded as a
>> plugin by a program (Go or non-Go) that supports a C style plugin
>> API", and that is supported using -buildmode=c-shared.
>>
>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[gerald...@gmail.com]

the go runtime...

matlab's loadlibrary (requires the path to the dll and the header file) does a dlopen in Unix or LoadLibrary in Windows.

it just loads a dynamically-linked library

[minux]

On Sep 5, 2014 11:09 AM, <gerald...@gmail.com> wrote:
>
> the go runtime...
>
> matlab's loadlibrary (requires the path to the dll and the header file) does a dlopen in Unix or LoadLibrary in Windows.
> it just loads a dynamically-linked library

it's possible to arrange the runtime to be initialized automatically when dlopened,
but we need to be careful when people load multiple go plugins to the same process.

btw, that's how you can have static initializers in C++ shared libraries.

[gerald...@gmail.com]

when do you plan the go execution modes to be integrated, 1.5?

[Ian Lance Taylor]

On Fri, Sep 5, 2014 at 8:21 AM, <gerald...@gmail.com> wrote:
>
> when do you plan the go execution modes to be integrated, 1.5?

I don't know. This is a proposal for how the work should be done, but
somebody has to actually do the work.

It certainly won't happen before 1.5.

Ian

[Kyle Lemons]

Apologies if these were already answered.

Why should "main" not be treated like an "init" for c-archive, plugin, et al?  It seems like requiring one, but not executing it, is a recipe for confusion.

LookupC sounds like it would work for variables; does it work for any exported symbols or just functions?

Not supporting unloading of plugins is a heartbreaker.  it's critical to pretty much every interesting thing I can come up with for using plugins.  Everything from IRC bots to pluggable webservers to IDEs seem like they don't benefit much if you have to restart the binary just to get a new version of a plugin.  Does the garbage collector not help us out here by letting us know when we still have active references to data in the plugin's space?

[Ian Lance Taylor]

On Mon, Sep 8, 2014 at 2:10 PM, Kyle Lemons <kev...@google.com> wrote:
>
> Why should "main" not be treated like an "init" for c-archive, plugin, et
> al? It seems like requiring one, but not executing it, is a recipe for
> confusion.

The proposal is to require a main package, not a main function. That
is, you have to have a package "main". That package may contain a
function "main," but it need not. And the function "main" is not
treated specially.

It's true that we could treat "main" as equivalent to "init", but 1)
there's no need for that, since we already have "init"; 2) it would be
potentially confusing in a different way, as exiting from the "main"
function would not cause the process to exit as it normally does.

> LookupC sounds like it would work for variables; does it work for any
> exported symbols or just functions?

You're right, LookupC should work for global variables. I amended the
comment.

> Not supporting unloading of plugins is a heartbreaker. it's critical to
> pretty much every interesting thing I can come up with for using plugins.
> Everything from IRC bots to pluggable webservers to IDEs seem like they
> don't benefit much if you have to restart the binary just to get a new
> version of a plugin. Does the garbage collector not help us out here by
> letting us know when we still have active references to data in the plugin's
> space?

I can't think of any reliable and safe way to unload a Go plugin. The
plugin might have started some goroutine that is sleeping and will
wake up weeks later. The program will run fine until the goroutine
wakes up and starts executing again, at which point it will, if you're
lucky, crash. The garbage collector won't know about this because the
sleeping goroutine might only refer to code in the plugin, and the
garbage collector doesn't track references to code. I think it would
be fine to support unloading of plugins if we can make it safe, but I
don't see how it can be in the initial implementation.

I think it might be more feasible to figure out how to safely load
multiple versions of the same plugin. If it's a c-shared plugin this
is probably not too hard. I think it mainly requires relaxing the
rule on having multiple copies of the same package, which we would do
by somehow forcing a different package path.

Ian

[Matthew Dempsky]

One minor comment: the "Versioning" section jumps from how Go code is currently required to use the same toolchain to then concluding that Go code must share the same runtime too.  But I think the implication actually goes in the opposite direction: if Go packages are to be able to interoperate normally (e.g., share pointers to GC'd objects, send/receive messages on channels, etc.), those packages need to share a single runtime; and to share a single runtime, they need to be compiled with the same toolchain.

As long as Go packages are okay with running independently (e.g., treating each other like they'd treat C code running in the same process), it seems like they could coexist in a single process using separate runtime packages, and even possibly compiled with separate toolchains.  Not sure such an execution model would actually find enough usage to warrant the cost of implementing/supporting it though.

[Ian Lance Taylor]

On Mon, Sep 8, 2014 at 4:30 PM, Matthew Dempsky <mdem...@google.com> wrote:
>
> One minor comment: the "Versioning" section jumps from how Go code is
> currently required to use the same toolchain to then concluding that Go code
> must share the same runtime too. But I think the implication actually goes
> in the opposite direction: if Go packages are to be able to interoperate
> normally (e.g., share pointers to GC'd objects, send/receive messages on
> channels, etc.), those packages need to share a single runtime; and to share
> a single runtime, they need to be compiled with the same toolchain.

Yes.

> As long as Go packages are okay with running independently (e.g., treating
> each other like they'd treat C code running in the same process), it seems
> like they could coexist in a single process using separate runtime packages,
> and even possibly compiled with separate toolchains. Not sure such an
> execution model would actually find enough usage to warrant the cost of
> implementing/supporting it though.

It's definitely possible, and in fact I originally wrote the document
that way. But nobody liked it, because it means that, for example,
you can't pass a channel from one Go runtime to another. I think the
current model is simpler to understand, so it's probably a better
place to start.

Ian

[gerald...@gmail.com]

Hello Ian,

what do you think about this approach?

https://groups.google.com/forum/#!topic/golang-nuts/0KnBmw2ARf0

-Gerald

[Ian Lance Taylor]

On Tue, Sep 16, 2014 at 1:07 AM, <gerald...@gmail.com> wrote:
>
> what do you think about this approach?
> https://groups.google.com/forum/#!topic/golang-nuts/0KnBmw2ARf0

I haven't looked at it in detail, but what I've seen looks great.

You asked on a thread about execution modes, though, and it doesn't
seem related to that. It seems more related to tools like cgo and
SWIG. Am I missing some connection to execution modes?

Ian

[mortdeus]

It would be cool if we could add an execution mode for building kernel modules.

[mortdeus]

Also, I really like the plugin pkg idea. If (when?) the runtime is rewritten in Go, could we use this plugin pkg to build a pluggable runtime?

[Ian Lance Taylor]

On Thu, Sep 18, 2014 at 11:05 PM, mortdeus <mort...@gmail.com> wrote:
> It would be cool if we could add an execution mode for building kernel modules.

That would be a lot of work. The Go runtime requires threading and
memory allocation support, and all of that would have to be written.
And I really can't see many people writing a kernel module in a
garbage collected language.

Ian

[Ian Lance Taylor]

On Thu, Sep 18, 2014 at 11:20 PM, mortdeus <mort...@gmail.com> wrote:
> Also, I really like the plugin pkg idea. If (when?) the runtime is rewritten in Go, could we use this plugin pkg to build a pluggable runtime?

In a sense, and with some further work, yes. The runtime could be
built as a shared library, and you could select which shared library
to use at runtime. It couldn't be a plugin--you couldn't switch
runtimes while the program is running--but you could choose a runtime
at program startup time.

Ian

[gerald...@gmail.com]

whats the expected startup delay to initialize the plugin/runtime?

[Ian Lance Taylor]

On Fri, Sep 19, 2014 at 8:05 AM, <gerald...@gmail.com> wrote:
> whats the expected startup delay to initialize the plugin/runtime?

As there is no implementation as yet, I have no idea. I would guess
that the additional delay of using a shared library for the runtime
would be comparable to the additional startup delay of a C program
that dynamically links against the C library.

Ian

[mortdeus]

On Friday, September 19, 2014 7:43:42 AM UTC-5, Ian Lance Taylor wrote:

On Thu, Sep 18, 2014 at 11:05 PM, mortdeus <mort...@gmail.com> wrote:
> It would be cool if we could add an execution mode for building kernel modules.

That would be a lot of work.  The Go runtime requires threading and
memory allocation support, and all of that would have to be written.

Can't we use kmalloc and kthreads? 

https://www.kernel.org/doc/htmldocs/kernel-api/API-kmalloc.html
https://prof.hti.bfh.ch/myf1/adv-linux/courseNotes/kthread-intro-1.2.pdf

 
 

And I really can't see many people writing a kernel module in a
garbage collected language.

http://www.jwz.org/doc/gc.html

 

Ian

[Ian Lance Taylor]

On Fri, Sep 19, 2014 at 8:21 AM, mortdeus <mort...@gmail.com> wrote:
>
>
> On Friday, September 19, 2014 7:43:42 AM UTC-5, Ian Lance Taylor wrote:
>>
>> On Thu, Sep 18, 2014 at 11:05 PM, mortdeus <mort...@gmail.com> wrote:
>> > It would be cool if we could add an execution mode for building kernel
>> > modules.
>>
>> That would be a lot of work. The Go runtime requires threading and
>> memory allocation support, and all of that would have to be written.
>
> Can't we use kmalloc and kthreads?

I didn't say it couldn't be done. I said it would be a lot of work.

>> And I really can't see many people writing a kernel module in a
>> garbage collected language.
>
>
> http://www.jwz.org/doc/gc.html

I work on a garbage collected language. I understand the arguments.
I still think that I am correct: I really can't see many people

writing a kernel module in a garbage collected language.

Ian

[gordon...@gmail.com]

On Friday, September 19, 2014 5:48:02 PM UTC+2, Ian Lance Taylor wrote:

I work on a garbage collected language.  I understand the arguments.
I still think that I am correct: I really can't see many people
writing a kernel module in a garbage collected language.

Just curious, do you say this simply because there are relatively few people that write kernel modules or because you think it would be a bad idea to use a garbage collected language to write kernel modules?

I don't have any experience writing kernel modules but somewhere down the road I might want to, at which point I can imagine it would be nice to be able to use Go (if that is even sensible).

Certainly fewer people will write kernel modules in a garbage collected language if a good such language doesn't exist.

[Brad Fitzpatrick]

You'll want to learn to write kernel modules in the same language and with the same APIs as everybody else. Once you've mastered that, then you can explore the technical and political challenges of using a different language in the kernel.

--

[Henrik Johansson]

When you do suggest a kernel module in Go on the kernel list please let me know before so I can get popcorn for the show! ;)

I am not saying it wouldn't be nice but like Brad so politely implied. You are in for quite a ride getting it in the kernel even if you get it to work. 

[mortdeus]

A kernel module doesn't have to be merged upstream. I understand the technological and socialogical challenges developers would face when using Go for kernel development.

However those challenges only make the proposal more attractive in my opinion. If upstream rejects a kernel module pull request solely based on the fact that it is written in Go; despite the module being well written and reasonably efficient, then I say their prejudice is in hostile conflict with the pace of innovation.

The good news is that outstanding technology can change the way people precieve what makes a technical idea good or bad. Go has already had such an impact on many developers who were convinced using a type-safe, garbage collected language for low level system development is a pipe dream held by loonies and heretics.

[paulo....@gmail.com]

There are a lot of heretics that have tasted the forbidden fruit...

Project Oberon (2013 revised documentation)
http://www.inf.ethz.ch/personal/wirth/ProjectOberon/
http://www.inf.ethz.ch/personal/wirth/projects.html
http://progtools.org/article.php?name=oberon&section=compilers&type=tutorial

Mesa/Cedar at Xerox PARC
https://archive.org/details/bitsavers_xeroxparcteCedarProgrammingEnvironmentAMidtermRepo_13518000

Modula-3 at DEC
http://en.wikipedia.org/wiki/SPIN_%28operating_system%29
http://www-spin.cs.washington.edu/external/overview.html

--
Paulo

[Michael Hudson-Doyle]

Ian Lance Taylor <ia...@golang.org> writes:

> I've written a document describing future directions for Go support
> for shared libraries and related ways of building Go programs. This
> is not the actual technical work. It's a proposal for how the
> technical work should be done.
>
> https://docs.google.com/a/golang.org/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ/edit#
>
> Please reply to this e-mail message with any comments.

Hi,

I've tried to read through this from the POV of Ubuntu, i.e. mainly the
"Building Go packages as a shared library" option. The proposal would
address our main concerns, I think, but the lack of ABI versioning
features would make it a bit cumbersome. I don't think having to
rebuild the world for a new version of the Go runtime is controversial,
but for other libraries it might be awkward. It would help if the go
tooling provided some kind of ABI hash to detect when a reverse
dependency needs to be rebuilt.

Do you view the lack of features such as versioned symbols as something
safe to avoid thinking about in a first implementation or as things that
will never be worth the complexity to implement?

Reading through the document, I'm not sure it's explicitly stated
anywhere it is the platform dynamic linker that is being used here.
Doing something different would be pretty crazy though...

Cheers,
mwh

[Ian Lance Taylor]

On Mon, Sep 22, 2014 at 9:18 PM, Michael Hudson-Doyle
<michael...@linaro.org> wrote:
>
> I've tried to read through this from the POV of Ubuntu, i.e. mainly the
> "Building Go packages as a shared library" option. The proposal would
> address our main concerns, I think, but the lack of ABI versioning
> features would make it a bit cumbersome. I don't think having to
> rebuild the world for a new version of the Go runtime is controversial,
> but for other libraries it might be awkward. It would help if the go
> tooling provided some kind of ABI hash to detect when a reverse
> dependency needs to be rebuilt.

I think you are saying: if package A imports package B, and package B
is built as a shared library, then it would be nice to know whether,
when package B is rebuilt, it is necessary to rebuild package A.

In general it will be necessary to recompile package A if any of the
export data in package B changes. So one approach would be some way
for package B to export a hash of the export data, and some way for
package A to record the hash of the export data of package B when
package A was built.

I suppose that such an approach could be built into the go tool. The
go tool would somehow record the hashes when building shared libraries
and when linking against shared libraries.

Would something like that address the issue?

> Do you view the lack of features such as versioned symbols as something
> safe to avoid thinking about in a first implementation or as things that
> will never be worth the complexity to implement?

Versioned symbols are very difficult to use correctly. Very few
packages pull it off today. I can't really ever see implementing them
for Go.

Versioned symbols are designed to solve the problem of permitting
multiple shared libraries, built with different header files and
linked against different versions of the system library, to live
together in the same program image. My doc doesn't permit that case
at all. Versioned symbols allow that case to be solved, with
considerable difficulty, for C. They don't work for C++ without
extensive compiler support that is only just now being written. For
Go, with its extensive runtime dependencies, I don't think it will
ever be possible to support linking together shared libraries built
against different versions of the Go runtime. So versioned symbols
are not an answer that is going to work for Go.

> Reading through the document, I'm not sure it's explicitly stated
> anywhere it is the platform dynamic linker that is being used here.
> Doing something different would be pretty crazy though...

Using the system dynamic linker is the intent, yes.

Ian

[Gustavo Niemeyer]

On Tue, Sep 23, 2014 at 11:20 AM, Ian Lance Taylor <ia...@golang.org> wrote:
> On Mon, Sep 22, 2014 at 9:18 PM, Michael Hudson-Doyle
> <michael...@linaro.org> wrote:
>>
>> I've tried to read through this from the POV of Ubuntu, i.e. mainly the
>> "Building Go packages as a shared library" option. The proposal would
>> address our main concerns, I think, but the lack of ABI versioning
>> features would make it a bit cumbersome. I don't think having to
>> rebuild the world for a new version of the Go runtime is controversial,
>> but for other libraries it might be awkward. It would help if the go
>> tooling provided some kind of ABI hash to detect when a reverse
>> dependency needs to be rebuilt.
>
> I think you are saying: if package A imports package B, and package B
> is built as a shared library, then it would be nice to know whether,
> when package B is rebuilt, it is necessary to rebuild package A.
>
> In general it will be necessary to recompile package A if any of the
> export data in package B changes. So one approach would be some way
> for package B to export a hash of the export data, and some way for
> package A to record the hash of the export data of package B when
> package A was built.

This makes me wonder what will happen to inlining and escape analysis
in that context. I suspect they might have to be disabled across
shared library boundaries, to prevent a trivial internal fix from
breaking such public interfaces.

> I suppose that such an approach could be built into the go tool. The
> go tool would somehow record the hashes when building shared libraries
> and when linking against shared libraries.
>
> Would something like that address the issue?

I believe it would. This information would also need to be made
available to external clients, so that a build farm could tell when
it's time to rebuild dependencies, but if the go tool itself can do
it, it should be easy to export the same mechanism externally so that
the build can be triggered.


gustavo @ http://niemeyer.net

[Ian Lance Taylor]

On Tue, Sep 23, 2014 at 8:30 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:
> On Tue, Sep 23, 2014 at 11:20 AM, Ian Lance Taylor <ia...@golang.org> wrote:
>> On Mon, Sep 22, 2014 at 9:18 PM, Michael Hudson-Doyle
>> <michael...@linaro.org> wrote:
>>>
>>> I've tried to read through this from the POV of Ubuntu, i.e. mainly the
>>> "Building Go packages as a shared library" option. The proposal would
>>> address our main concerns, I think, but the lack of ABI versioning
>>> features would make it a bit cumbersome. I don't think having to
>>> rebuild the world for a new version of the Go runtime is controversial,
>>> but for other libraries it might be awkward. It would help if the go
>>> tooling provided some kind of ABI hash to detect when a reverse
>>> dependency needs to be rebuilt.
>>
>> I think you are saying: if package A imports package B, and package B
>> is built as a shared library, then it would be nice to know whether,
>> when package B is rebuilt, it is necessary to rebuild package A.
>>
>> In general it will be necessary to recompile package A if any of the
>> export data in package B changes. So one approach would be some way
>> for package B to export a hash of the export data, and some way for
>> package A to record the hash of the export data of package B when
>> package A was built.
>
> This makes me wonder what will happen to inlining and escape analysis
> in that context. I suspect they might have to be disabled across
> shared library boundaries, to prevent a trivial internal fix from
> breaking such public interfaces.

The only thing that the compiler knows about the imported package B is
what is present in B's export data. When the compiler is able to do
inlining or escape analysis across packages, it is doing so based on
the export data. Therefore, if anything happens that would cause
earlier inlining or escape analysis to change, it will cause a change
in the export data, and that will be sufficient to trigger the
recompilation of package A.

Ian

[Gustavo Niemeyer]

On Tue, Sep 23, 2014 at 1:21 PM, Ian Lance Taylor <ia...@golang.org> wrote:
> The only thing that the compiler knows about the imported package B is
> what is present in B's export data. When the compiler is able to do
> inlining or escape analysis across packages, it is doing so based on
> the export data. Therefore, if anything happens that would cause
> earlier inlining or escape analysis to change, it will cause a change
> in the export data, and that will be sufficient to trigger the
> recompilation of package A.

Okay, so such trivial implementation changes will also break the ABI.


gustavo @ http://niemeyer.net

[Ian Lance Taylor]

Correct.

I think that the notion of a meaningful ABI for a shared library is
only supportable for C. It takes heroic effort to make it work for
C++. I don't see any plausible way to make it work for Go.

Ian

[Gustavo Niemeyer]

We could plausibly disable inlining and escape analysis for the shared
library boundaries, but it doesn't seem worth arguing about this right
now. Just having the basics working would be a great win, and with a
tool that at least tells what the current ABI is, one can tell what
needs to happen for things to work.


gustavo @ http://niemeyer.net

[Matthew Dempsky]

On Tue, Sep 23, 2014 at 9:54 AM, Ian Lance Taylor <ia...@golang.org> wrote:

I think that the notion of a meaningful ABI for a shared library is
only supportable for C.  It takes heroic effort to make it work for
C++.  I don't see any plausible way to make it work for Go.

I'd suspect it's doable if users were willing to maintain files analogous to Go's api/go*.txt files, but with additional ABI information (e.g., struct field offsets, inlining info).  Suppose you have

    abi/mypkg1.txt

        pkg mypkg, func Foo()

        pkg mypkg, type Point struct, X@0 uint64, Y@8 uint64

        ...

    abi/mypkg1.1.txt

        pkg mypkg, func Bar()

        ...

the linker could first check that the exposed ABI matches the claimed ABIs.  Next, it could assign strong names to each ABI revision; e.g.,

    abi1 = md5("pkg mypkg, func Foo()\n...")

    abi1.1 = md5(abi1 + "pkg mypkg, func Bar()\n...")

and incorporate those strong ABI names into each exported symbol's version; e.g.,

    mypkg.Foo@${abi1}

    mypkg.Bar@${abi2}

Similar to ELF versioning scripts, but automated by the toolchain.

I'd also think for users that value ABI stability over performance, the linker could simply (optionally) omit inlining info from its exported data.  Possibly escape analysis info too, assuming the compiler can degrade gracefully when it's missing?

[Ian Lance Taylor]

On Tue, Sep 23, 2014 at 10:19 AM, Matthew Dempsky <mdem...@google.com> wrote:
> On Tue, Sep 23, 2014 at 9:54 AM, Ian Lance Taylor <ia...@golang.org> wrote:
>>
>> I think that the notion of a meaningful ABI for a shared library is
>> only supportable for C. It takes heroic effort to make it work for
>> C++. I don't see any plausible way to make it work for Go.
>
>
> I'd suspect it's doable if users were willing to maintain files analogous to
> Go's api/go*.txt files, but with additional ABI information (e.g., struct
> field offsets, inlining info). Suppose you have
>
> abi/mypkg1.txt
> pkg mypkg, func Foo()
> pkg mypkg, type Point struct, X@0 uint64, Y@8 uint64
> ...
> abi/mypkg1.1.txt
> pkg mypkg, func Bar()
> ...
>
> the linker could first check that the exposed ABI matches the claimed ABIs.
> Next, it could assign strong names to each ABI revision; e.g.,
>
> abi1 = md5("pkg mypkg, func Foo()\n...")
> abi1.1 = md5(abi1 + "pkg mypkg, func Bar()\n...")
>
> and incorporate those strong ABI names into each exported symbol's version;
> e.g.,
>
> mypkg.Foo@${abi1}
> mypkg.Bar@${abi2}
>
> Similar to ELF versioning scripts, but automated by the toolchain.

I think you're thinking mainly thinking about functions. The
interesting cases are when a type changes and there are existing
functions that use that type. For example, mypkg1.1 decides that
Point needs to change to V@0 uint64, X@8 uint64, Y@16 uint64. mypkg1
has a function F that takes a Point argument, and so does mypkg1.1.
You have existing code that calls F and new code that calls F.

And even that is a simple case, because it's C api stuff and symbol
versioning can handle it. Now we need to talk about the methods
defined on a type: in mypkg1 Point has some methods, and in mypkg1.1
we drop some existing methods and add some new ones. We have a
function F2 that returns a value of type Point. Packages that import
mypkg1 need to get one set of methods and packages that import
mypkg1.1 need to get a different set.

And even that is a simple case. We need to talk about reflection.
There is Go code that thinks it knows what a PkgPath means, because
its documented to work a certain way. When your program imports two
different instances of the same package, what is the PkgPath value
when you use reflection on the different types?

> I'd also think for users that value ABI stability over performance, the
> linker could simply (optionally) omit inlining info from its exported data.
> Possibly escape analysis info too, assuming the compiler can degrade
> gracefully when it's missing?

Yes, both are possible.

Ian

[Matthew Dempsky]

On Tue, Sep 23, 2014 at 10:42 AM, Ian Lance Taylor <ia...@golang.org> wrote:

For example, mypkg1.1 decides that
Point needs to change to V@0 uint64, X@8 uint64, Y@16 uint64.

Agreed, but that's an ABI breaking change, which could be prevented by not exporting the struct's fields, just like you'd do in C.

mypkg1 has a function F that takes a Point argument, and so does mypkg1.1.

Perhaps the ELF versioning analogy was bad.  I know C libraries expose multiple versions of the same function/library within a DSO, but I was just trying to address the more basic problem of using versioning to detect/prevent library dependency mismatches, while also allowing libraries to both stay backwards compatible and add new exported names.

I meant to suggest that once a function is added in an earlier ABI, it's implicitly inherited by later ABIs too.  I.e., ABI compatibility would be a superset of API compatibility.  So just like strings.Index() is in the Go 1 API, it must be in the Go 1.1 API and so on; if F is in the mypkg1 ABI, it must then also be in the mypkg1.1 ABI and so on.

You have existing code that calls F and new code that calls F.

There'd only be one F, so they'd call the same F.

Now we need to talk about the methods
defined on a type: in mypkg1 Point has some methods, and in mypkg1.1
we drop some existing methods and add some new ones.

Dropping methods would break ABI compatibility, just like it would break API compatibility.

And even that is a simple case.  We need to talk about reflection.
There is Go code that thinks it knows what a PkgPath means, because
its documented to work a certain way.  When your program imports two
different instances of the same package, what is the PkgPath value
when you use reflection on the different types?

Again, only one instance of a package, so I don't think this applies.

[Gustavo Niemeyer]

I suspect this doesn't really matter so much right now. There's going
to be an ABI, whatever it is, and people will break their packages'
ABI, consciously or not. So what we need is a way to tell when
compatibility may have changed so that tooling may be developed around
it. Over time, the tooling may improve, but there's still a lot work
to do before it's meaningful to do so.

--

gustavo @ http://niemeyer.net

[Michael Hudson-Doyle]

Ian Lance Taylor <ia...@golang.org> writes:

> On Mon, Sep 22, 2014 at 9:18 PM, Michael Hudson-Doyle
> <michael...@linaro.org> wrote:
>>
>> I've tried to read through this from the POV of Ubuntu, i.e. mainly the
>> "Building Go packages as a shared library" option. The proposal would
>> address our main concerns, I think, but the lack of ABI versioning
>> features would make it a bit cumbersome. I don't think having to
>> rebuild the world for a new version of the Go runtime is controversial,
>> but for other libraries it might be awkward. It would help if the go
>> tooling provided some kind of ABI hash to detect when a reverse
>> dependency needs to be rebuilt.
>
> I think you are saying: if package A imports package B, and package B
> is built as a shared library, then it would be nice to know whether,
> when package B is rebuilt, it is necessary to rebuild package A.

Yes, exactly.

> In general it will be necessary to recompile package A if any of the
> export data in package B changes. So one approach would be some way
> for package B to export a hash of the export data, and some way for
> package A to record the hash of the export data of package B when
> package A was built.
>
> I suppose that such an approach could be built into the go tool. The
> go tool would somehow record the hashes when building shared libraries
> and when linking against shared libraries.
>
> Would something like that address the issue?

I believe so, yes.

>> Do you view the lack of features such as versioned symbols as something
>> safe to avoid thinking about in a first implementation or as things that
>> will never be worth the complexity to implement?
>
> Versioned symbols are very difficult to use correctly. Very few
> packages pull it off today. I can't really ever see implementing them
> for Go.

OK.

> Versioned symbols are designed to solve the problem of permitting
> multiple shared libraries, built with different header files and
> linked against different versions of the system library, to live
> together in the same program image.

Ah OK; I guess I wasn't super clear in my head about exactly what
versioned symbols were (I only meant to use them as an example of a
ABI-compatibility technique).

> My doc doesn't permit that case at all. Versioned symbols allow that
> case to be solved, with considerable difficulty, for C. They don't
> work for C++ without extensive compiler support that is only just now
> being written. For Go, with its extensive runtime dependencies, I
> don't think it will ever be possible to support linking together
> shared libraries built against different versions of the Go runtime.
> So versioned symbols are not an answer that is going to work for Go.

I certainly can see that attempting to use a shared library compiled for
one version of the Go runtime with different version (in the sense of
1.3 vs 1.4, not trivial bugfixes) would be so difficult as to be foolish
to attempt.

The situation I was more thinking of (and this is probably getting way
ahead of what it makes sense to think of right now) is more like this:

Suppose a mobile phone operating system allows the installation of
third-party binary applications written in Go, and allows said
applications to link against the system-provided shared library build of
go.crypto (or any library whose release cycle is not tied to the Go
releases -- not sure if this applies or not to go.crypto). If any
change to the ABI of go.crypto means that all binaries that link to it
need to be rebuilt, basically the system would have to ship all versions
of go.crypto that have ever had a third-party binary built against them.
And storage on phones is tight enough that this might be undesirable.
Obviously some changes will consitute an ABI break, but not all.

Not something to worry about in the first iteration, to be sure.

>> Reading through the document, I'm not sure it's explicitly stated
>> anywhere it is the platform dynamic linker that is being used here.
>> Doing something different would be pretty crazy though...
>
> Using the system dynamic linker is the intent, yes.

I'm glad to hear that :-)

Cheers,
mwh

[Michael Hudson-Doyle]

Ian Lance Taylor <ia...@golang.org> writes:

I did have a sudden worry about this sort of thing: a shared library
will have to provide this sort of data (and also things like the data
the gc uses to scan types provided by the library). But of course, go
already supports separate compilation and so the existing go linker is
clearly able to take care of this. Do you forsee any difficulty
combining this data at dynamic link time rather than build time? I
don't know anything like enough about how this works today to be able to
guess.

Cheers,
mwh

[Ian Lance Taylor]

I don't think the dynamic linker will have to worry about this. The
information will already have been examined by the regular linker.
All we need to do at run time is ensure that the shared libraries are
compatible, and I believe we can do that using Go code that runs when
the program starts.

Ian

[Ian Lance Taylor]

On Tue, Sep 23, 2014 at 8:30 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:

I added a note about this to the execution modes doc.

Ian