plugins (Go 1.8) and packages

[Basile Starynkevitch]

Hello all,

Here are some thoughts and questions about plugins in Go 1.8 (which I use on Linux/x86-64).

Plugins in other languages

In C and C++ code (on Linux), plugins are practically not exactly the same as e.g. shared objects (a good reference would be Drepper's How To Write Shared Libraries paper). In practice, a plugin is a (dlopen-ed) shared object with references to symbols in the main program. For example, a GCC plugin (this is an example that both Ian Taylor & me know well) will call functions from GCC itself (such as register_callback or many others, e.g. gimple_block etc etc...). Likewise, a GEDIT plugin (for example the wordcompletion one) will call GEDIT or GTK functions like gedit_debug or gtk_text_view_get_buffer etc...). Notice that to make that possible the main program (cc1plus for GCC
or gedit) has to be linked with the -rdynamic flag (to make the symbols of the main program visible from the plugin). Even if that is in theory principle, it is

uncommon to dlopen a shared object which is not a plugin, designed and coded to call symbols from the main program (a program might in theory dlopen some 
plain shared library  like /usr/lib/x86_64-linux-gnu/libgdbm.so.3 but in practice this is never done; program loading plugins are dlopen-ing some 
shared object -the plugin- specifically designed for them, and calling functions from the main program, and the main program uses dlsym to find some specific 
symbols such as plugin_init for GCC obeying some signature convention). I deliberately ignore plugins loading other plugins (but I know about RTLD_GLOBAL flag to dlopen)
and I know about constructor function attributes in plugins.

In JAVA, there are class loaders with a sophisticated protocol for loading them.

In Ocaml, you have the dynlink library. The Dynlink.loadfile function is similar to dlopen (and will run the plugin initialization code).
There is no equivalent of dlsym: the plugin is supposed to e.g.  register function values at initialization time, perhaps by passing them to 
some function of the main program. Of course the plugin can call functions from the main program.

Plugins and packages in Go

The package concept is a core concept of Go since every source file belongs to some package (with main being a special case) and often imports several other ones.
Practically speaking, a Go plugin is likely to call functions from some package defined by the main program and having itself (the plugin) some packages. So the
 tiny example in plugin documentation is a bit too naive (even if it is calling fmt.Printf). A realistic example would be a program having not only some main function 
in its main package, but also defining some purple package having some Foo public function called as purple.Foo from main and having a Bar public function
(with purple.Bar called from the plugin). The plugin would have not only some public F function in its main package (which should call purple.Bar from the main 
plugin-loading program) but also some plugin specific yellow package with a public Y function called (as yellow.Y) from F. I hope that such a realistic example will be 
given in the documentation of future Go 1.9.

Unfortunately, I know no public mechanism to say, within the source code of a plugin, that some imported package is (and has to be) provided by the main loading program.
I would suggest to re-use and extend the import comment convention (with the path "*" being a notation to say import that package from the main loading program). For example, our plugin.go source code might have import "purple" // import "*" on a single line, and then the purple package would be externally linked (not incorporated in the plugin; exactly like in some GCC plugin calling gimple_block does not add all the existing GIMPLE processing code into the plugin).

Currently, a plugin which uses some package (even a standard one like fmt; but I am thinking of a program-supplied one like purple)  is linking it twice (once in the main program, and once in the plugin). This is inefficient (compile time of plugins -even a tiny one- is huge, and their shared object size is much too big) and error prone (it is not defined what happens if the version of that package is different, and there might be -in weird cases- infelicities with multiple initialization of the same package)


A use case

In my monimelt program (which is now buildable with go tool and don't use gb anymore, since gb does not support plugins yet) commit 6bb8b56160749
the test-plugins/makename.go is a simplistic plugin which just calls a few functions from the objvalmo & payloadmo packages defined in the main program. But its compilation
time with build-plugin-monimelt.sh script is ridiculously big, since all the packages of the main program (and even external ones like go-sqlite) get linked both in the main program and the plugin. To run that test, do ./monimelt -run-plugin test-plugins/makename.go which will compile test-plugins/makename.go into makename.so using build-plugin-monimelt.sh script and open that plugin. The whole point of monimelt will be to generate Go source code (of plugins) at runtime and compile and load them (I followed the same idea in GCC MELT).

BTW, I am interested in reading some white paper about the precise semantics of packages, and how the Go compiler works internally.

Cheers
--
Basile Starynkevitch (France)

[Basile Starynkevitch]

On Monday, March 20, 2017 at 8:36:04 AM UTC+1, Basile Starynkevitch wrote:

Plugins and packages in Go

The package concept is a core concept of Go since every source file belongs to some package (with main being a special case) and often imports several other ones.
Practically speaking, a Go plugin is likely to call functions from some package defined by the main program and having itself (the plugin) some packages. So the
 tiny example in plugin documentation is a bit too naive (even if it is calling fmt.Printf). A realistic example would be a program having not only some main function 
in its main package, but also defining some purple package having some Foo public function called as purple.Foo from main and having a Bar public function
(with purple.Bar called from the plugin). The plugin would have not only some public F function in its main package (which should call purple.Bar from the main 
plugin-loading program) but also some plugin specific yellow package with a public Y function called (as yellow.Y) from F. I hope that such a realistic example will be 
given in the documentation of future Go 1.9.

To be more concrete, Here is the scenario I am thinking of:

the main (plugin loading) program has some purple package in a purple.go file (probably in some purple/ directory):

/// file purple.go of the main program
package purple
import "fmt"

func Foo(x int) { // called from main
  fmt.Printf("purple.Foo has x=%d\n", x)
}

func Bar(s string) { // called from plugin
  fmt.Printf("purple.Bar has s=%q\n", s)
}

/// eof purple.go

Let's assume we have a /tmp/plugin.so binary plugin. Is is made from plugintop.go and pluginyellow.go and it has a yellow package and it calls purple.Bar. So here is pluginyellow.go :

//file pluginyellow.go of the plugin
package yellow // the yellow package is in the plugin
import "fmt"

// we import the purple package from the main program
import "purple"

// in real life, yellow is importing a lot more of existing packages (e.g. sql)....

function F() { // this function is called from plugintop.go
  fmt.Printf("in yellow.F\n")
  purple.Bar("from yellow.F")
  fmt.Printf("ending yellow.F\n")
}

function Gee () {
  fmt.Printf("in yellow.Gee calling Foo with 345\n")
  purple.Foo(345);
  fmt.Printf("end yellow.Gee")
}

function GG() {  // this ls looked up by the main program
  fmt.Printf("in yellow.GG\n")
}

// eof pluginyellow.go

And here is the top file of the plugin, plugintop.go (I don't know in what directory it should go exactly):

// file plugintop.go
package main

import "fmt"
import "yellow"
func init() {
  fmt.Printf("init of plugintop\n")
  yellow.F()
  fmt.Printf("end of init of plugintop\n")
}

func Pub() { // this is looked up by the main program
  fmt.Printf("in Pub of plugintop calling GG\n")
  yellow.GG()
  fmt.Printf("in Pub of plugintop ending\n")
}
// eof plugintop.go

Our main program (which is loading the plugin) defines a purple package in some purpleprog.go file

// file purpleprog.go
package purple
import "fmt"
func Foo(x int) {
  fmt.Printf("in purple.Foo x=%d\n", x)
}

func Bar(m string) {
  fmt.Printf("in purple.Bar m=%q\n", m)
}
// eof purpleprog.go

Of course we need a main in our program, file mainprog.go is loading the /tmp/plugin.so plugin

// file mainprog.go
package main
import "fmt"
import "plugin"
import "purple"

func main() {
  fmt.Printf("start of main in mainprog.go\n")
  plug, err := plugin.Open("/tmp/plugin.so")
  fmt.Printf("in mainprog plug=%v err=%v\n", plug, err)
  if err != nil {
    panic(fmt.Errorf("plugin.Open /tmp/plugin.so failed in mainprog with %v", err))
  }
  fmt.Printf("mainprog before call purple.Foo with 12751\n")
  purple.Foo(12751)
  fmt.Printf("mainprog after call purple.Foo with 12751\n")
  symbPub, err := plug.Lookup("Pub")
  fmt.Printf("in mainprog symbPub=%v err=%v\n", symbPub, err)
  if err != nil {
    panic(fmt.Errorf("Lookup of pub failed with %v", err))
  }
  funpub := symbPub.(func())
  fmt.Printf("mainprog before calling funpub=%v\n", funpub)
  funpub()
  symbGG, err := plug.Lookup("yellow.GG")
  fmt.Printf("in mainprog symbGG=%v err=%v\n", symbGG, err)
  if err != nil {
    panic(fmt.Errorf("Lookup of yellow.GG failed with %v", err))
  }
  funGG := symbPub.(func())
  fmt.Printf("mainprog before calling funGG=%v\n", funGG)
  funGG()
  fmt.Printf("end of mainprog\n")
}
// end of file mainprog.go

In real life, the purple package inside the main program which is also used by the plugin is actually using a lot of code in many other packages (e.g. go-sqlite, os/exec, and so on).

So here are my questions:

What is (or are) the exact build command of the main program (we need to compile both mainprog.go & purpleprog.go files...). I suspect that -buildmode=shared would be useful when compiling purpleprog.go but I am not sure. I suspect that -linkshared should be used when compiling mainprog.go

What is (or are) the exact build command of the plugin (we need to compile both pluginyellow.go & plugintop.go files...). I expect the plugin to be compiled failrly quickly so I would like the purple package (and all its dependencies) to be external to it, and not be compiled with the plugin (just linked to it). I really want to avoid compiling twice the purple package (which actually is quite big and using many other packages).


I might have misunderstood how plugins work. IMHO there should be some way to indicate to the compiler -when compiling the plugin- that the purple package is available in the main program. Perhaps the -buildmode=plugin mode might also accept a -loaded-from-program argument giving the main executable program that is loading it (so that the compiler can reuse and inspect packages available in the main program, like purple). Perhaps there should be some way (I was suggesting import "purple" // import "*" in my previous message) to indicate inside the source code of a plugin that a package should be imported and available from the main program.

(perhaps I might have identified an issue in the current Go1.8 implementation of plugins; if I did, please tell, and say me if I could hope that issue to be covered in Go1.9)

Regards, thanks for reading!

Basile Starynkevitch (France); my email is bas...@starynkevitch.net if you need to answer me privately.

PS. I really hope that some Go plugin guru would answer my messages!
 

[Ian Lance Taylor]

Well, there are no Go plugin gurus, and to be honest I'm starting to
think it was a mistake to let the plugin package into 1.8. I know
that is not what you want to hear. It's in because the API seems OK,
but it has become clear that there are many related issues that we
haven't thought about.

If I understand your example, your plugin imports a package, and your
main program also imports the same package, and it's unfortunate that
the package gets linked into both the plugin and the main program.
That is true. I think the only way to deal with is going to be to
compile the shared package with -buildmode=shared, and build both the
plugin and the main program with -linkshared. In fact that is what
you suggest. Does it work? I don't know whether it does or not.

Ian

[Basile Starynkevitch]

On Tuesday, March 21, 2017 at 5:59:11 PM UTC+1, Ian Lance Taylor wrote:

Well, there are no Go plugin gurus, and to be honest I'm starting to
think it was a mistake to let the plugin package into 1.8.  I know
that is not what you want to hear.  It's in because the API seems OK,
but it has become clear that there are many related issues that we
haven't thought about.

Well, certainly you Ian Lance Taylor are infinitely more a Go plugin guru than I (Basile) am today, and I thank you for spending some time to answer my newbie questions.

If I understand your example, your plugin imports a package, and your
main program also imports the same package, and it's unfortunate that
the package gets linked into both the plugin and the main program.
That is true. 

Thanks for the insight. I'm still not at ease with go build command, and I am a bit stressed by lack of build directives (e.g. some Makefile like thing instructing the build automation tool for particular options,
such as -buildmode=shared & -linkshared etc...)

Bear in mind that I am a Go newbie and the plugin abilities of Go1.8 are absolutely essential to me, since I want to generate Go code at runtime, compile it as a plugin, and load it as a plugin
(all that in the same process, with the compilation step  using some external command, e.g. forking some go build process, probably thru my -buggy- build-plugin-monimelt.sh shell script).
BTW, I am not alone in wanting this: Sandeep Kalra is asking something quite similar in his []byte to func question (even if he formulates his question very differently).
I'm coding monimelt (on github) and I just realize that my build procedure is badly broken (in today's commit 974dda17acc7)

First, is Svetlin's golang-sharing-libraries tutorial still exactly correct for Go1.8 specifically? Its title  Sharing Golang packages to C and Go is a bit misleading. I just want all my non-main packages to be "shared objects" in Linux parlance
(because I really don't want them to be compiled twice, once for the main program and once for the plugin). I don't care if each of my package has its own *.so file, or if all of them are agglomerated in one single *.so file. BTW, I would
prefer to avoid having tons of *.so files (what I would probably prefer is to have all the non-main code of my thing as a single *.so file shared between main program & plugins).

In particular, is issue 12236 still relevant for Go 1.8 ?

I think the only way to deal with is going to be to
compile the shared package with -buildmode=shared, and build both the
plugin and the main program with -linkshared.  In fact that is what
you suggest.  Does it work?  I don't know whether it does or not.

 My main concern is that I don't want to compilation of a small plugin taking a lot of time to recompile all the code used by that particular plugin and already provided & linked in the main program loading it.
I expect (like GCC MELT did for C and C++ code) that, when generating only a few hundred lines of Go source lines (the simplest case is a few hundred lines in a single /tmp/generated.go file with package main
as source code of a plugin which is importing many packages built in the main program, such as go-sqlite3 which contains sqlite3 agglomeration C code so takes many seconds to be compiled),
its compilation time as a plugin should be quick (probably a fraction of a second).
 

I am not sure to understand well how -buildmode=shared & -linkshared should be used. Is there an up to date tutorial in how to use them precisely?
Is there some "bullet-proof" (or at least "newbie-proof") way to ensure that a given set of packages (in my current monimelt, it would include my packages objvalmo  payloadmo  serialmo and most importantly all their dependencies
which includes, in addition of Go standard packages, jason, go-sqlite3 (which sadly takes a lot of time to be compiled, because it does not use the system's libsqlite3, but recompile its own Sqlite amalgamation in C code), & rbt ....
And I expect to use more external libraries in the future.

So how to force Go to systematically use -buildmode=shared? Is there some comment directive for that? Or should I provide my own shell script to build (I do know that the current one build-monimelt.sh is very buggy).
Is -buildmode=pie useful for me (I believe that not)?

I'm sorry for still being a newbie regarding Go. I understand that my use case is not usual. I have very hard time understanding the precise relation between Go source files and Go packages and their build procedure (notably for plugins).
Any link to a white paper describing them (in details) is welcome. The documentation of -buildmode is not precise enough (and does not gives hints of which one to choose).   The documentation of -linkshared  is saying just:

-linkshared
	link against shared libraries previously created with
	-buildmode=shared.

and I find that explanation too short (what happens if by mistake a previous package was compiled twice, both with -buildmode=shared & -buildmode=default and what happens if two weeks ago I have compiled my purple package -in some older version- with -buildmode=default and today I am compiling it -an improved version- with -buildmode=shared) ?

Regards

--
Basile Starynkevitch (France) - email bas...@starynkevitch.net

[Ian Lance Taylor]

On Wed, Mar 22, 2017 at 3:06 AM, Basile Starynkevitch
<bas...@starynkevitch.net> wrote:
>
> First, is Svetlin's golang-sharing-libraries tutorial still exactly correct
> for Go1.8 specifically? Its title Sharing Golang packages to C and Go is a
> bit misleading. I just want all my non-main packages to be "shared objects"
> in Linux parlance
> (because I really don't want them to be compiled twice, once for the main
> program and once for the plugin). I don't care if each of my package has its
> own *.so file, or if all of them are agglomerated in one single *.so file.
> BTW, I would
> prefer to avoid having tons of *.so files (what I would probably prefer is
> to have all the non-main code of my thing as a single *.so file shared
> between main program & plugins).

As far as I know that is correct.

> In particular, is issue 12236 still relevant for Go 1.8 ?

It is fixed in 1.8.

You may find it helpful to read https://golang.org/s/execmodes .

> I am not sure to understand well how -buildmode=shared & -linkshared should
> be used. Is there an up to date tutorial in how to use them precisely?

Not that I know of.

> So how to force Go to systematically use -buildmode=shared? Is there some
> comment directive for that? Or should I provide my own shell script to build
> (I do know that the current one build-monimelt.sh is very buggy).

You would have to use a shell script or something. There is no way to
make it the default, and really that wouldn't even make sense: you
would not to use -buildmode=shared when building your main executable,
only when building the packages that it imports.

> Is -buildmode=pie useful for me (I believe that not)?

I doubt that it is useful.

> The documentation of -linkshared is saying just:
>
>> -linkshared
>> link against shared libraries previously created with
>> -buildmode=shared.
>
>
> and I find that explanation too short (what happens if by mistake a previous
> package was compiled twice, both with -buildmode=shared & -buildmode=default
> and what happens if two weeks ago I have compiled my purple package -in some
> older version- with -buildmode=default and today I am compiling it -an
> improved version- with -buildmode=shared) ?

-linkshared is going to find the last package you built and installed
using `go install -buildmode=shared ...`.

Ian

[Basile Starynkevitch]

Thanks for the help. Using plugins is really difficult in practice.
 https://blog.ksub.org/bytes/2017/02/12/exploring-shared-objects-in-go/ is very helpful.

To Go implementors: the multiple roots error message is really cryptic. It should be a lot more explicit.

Thanks for reading

--
Basile Starynkevitch (France)


[nikl...@gmail.com]

It looks like an issue or bug that GoLang includes the runtime in plugins so that they cannot be shared. I did

go install -buildmode=shared std

And then I try to compile my plugin as shared:

 go build -buildmode=plugin -linkshared /tmp/code_SUM.go
# command-line-arguments
runtime.islibrary: missing Go type information for global symbol: size 1

My code

# cat /tmp/code_SUM.go
package main
import (
        "fmt"
)
func SUM(x int, y int) int { fmt.Println("")
return x+y}

Is there any workaround?