[go-nuts] Dynamically loaded binaries

[Stefan Midjich]

This has been posted to the list before.

http://groups.google.com/group/golang-nuts/browse_thread/thread/3e2a0f662ba7f3ff/f062a0e0853f4871?lnk=gst&q=dlopen#f062a0e0853f4871

It is a concern of mine however so i'd very much like to know if work is progressing or if it is planned at all.

People have mentioned using exec and netchan, which i found an attractive idea because i've been eager to try netchans.

But right now i'm porting an application from C that does use dlopen and dlsym to implement modules loaded at runtime. I want to port its functions to Golang but i'm not sure how to approach the modules.

Other languages have this, for example in Perl you can store a list of modules and call a pre-defined function in each module.

So it would be very great if Golang develops something like that too with packages. Perhaps using eval.

I know your slogan is share information and not memory but in this case i would actually like to load an external application into my own and have it share some memory or at least communicate as a goroutine.


Med vänliga hälsningar / With kind regards

Stefan Midjich

Telefon: 0707 54 62 02
E-post: swe...@gmail.com
Webb: http://swehack.se

Spara miljön, skriv ej ut det här brevet!

[Gustavo Hexsel]

  This is even harder if you want, say a webserver which handles multiple apps through a single port.

  Doing a process-per-application, would mean you have to transit all the info twice through the network...

[Stefan M]

Yes, Apache for example uses just that, dlopen, to implement runtime
loaded modules.

I think this is essential for any system language.

On May 16, 7:47 pm, Gustavo Hexsel <ghex...@gmail.com> wrote:
>   This is even harder if you want, say a webserver which handles multiple
> apps through a single port.
>
>   Doing a process-per-application, would mean you have to transit all the
> info twice through the network...
>
>
>
> On Sun, May 16, 2010 at 10:38 AM, Stefan Midjich <sweh...@gmail.com> wrote:
> > This has been posted to the list before.
>

> >http://groups.google.com/group/golang-nuts/browse_thread/thread/3e2a0...

>
> > It is a concern of mine however so i'd very much like to know if work is
> > progressing or if it is planned at all.
>
> > People have mentioned using exec and netchan, which i found an attractive
> > idea because i've been eager to try netchans.
>
> > But right now i'm porting an application from C that does use dlopen and
> > dlsym to implement modules loaded at runtime. I want to port its functions
> > to Golang but i'm not sure how to approach the modules.
>
> > Other languages have this, for example in Perl you can store a list of
> > modules and call a pre-defined function in each module.
>
> > So it would be very great if Golang develops something like that too with
> > packages. Perhaps using eval.
>
> > I know your slogan is share information and not memory but in this case i
> > would actually like to load an external application into my own and have it
> > share some memory or at least communicate as a goroutine.
>
> > Med vänliga hälsningar / With kind regards
>
> > Stefan Midjich
>
> > Telefon: 0707 54 62 02

> > E-post: sweh...@gmail.com

[roger peppe]

dynamically loaded code would be lovely to have.
and go already does dynamic typing really well,
so it's a long way along the road.

tricky bits:

- types

- global state

- runtime

i'm only going to talk about the first of these in this post.

suppose one could define a dynamic module like this:

package main
func dynmain() string {
   return "hello, world"
}

and you could load it like this:

   x, err := runtime.Load("/path/to/dynamiclibrary")
   fmt.Printf("got value from dynamic module: %s\n", x.(string))

so far so good.
but what if the module looked like this:

package main
import "os"
func dynmain() *os.File {
   f, _ := os.Open("/dev/null", os.O_WRONLY, 0)
   return f
}

and the loading code looked like this:

   x, err := runtime.Load("/path/to/dynamiclibrary")
   switch x := x.(type) {
   case *os.File:
       x.Write([]byte("hello, world"))
   default:
       panic("not a file")
   }

will "hello, world" get written to /dev/null, or will the code panic?

intuitively i'd prefer the former, but i don't think it's a good

idea. the dynamically loaded module has been linked against

some version of the library code. to make the dynamic code

loading safe, you'd have to ensure that the loading code 

has been linked against exactly the same version of the

code, which makes it extremely fragile.

one solution could be to agree on some set

of types that both the loading and the loaded module

will agree on - all other types test as different. call this

the set of shared types, S

A type t is a member of S if all the following conditions hold:

a) all its members are public

b) it has no concrete methods (*)

c) all component types of the members of t are members of S

(*) by "concrete method", i mean a method defined on t

with a func declaration.

then we can refine the notion of type identity:

Two named types are identical if they originate in the

same type declaration, or are both members of S,

otherwise identical, and declared in packages with identical

path names.

these rules means you can pass values back and forth between dynamic

modules, but no private state gets transferred, unless it's

inside an interface (and therefore safely encapsulated).

the compiler could check that the type returned by dynmain()

is actually a shared type.

the upshot of this proposal is that you'd have be careful how

you defined the interface to dynamically loaded modules,

but that's probably a good thing.

most current interfaces are allowed through under these

rules (e.g. io.Reader, image.Image), but some expectations

won't be fulfilled (e.g. image.AlphaColor is not a shared type)

perhaps this is too restrictive? thoughts?

[Scott Pakin]

On May 17, 3:43 am, roger peppe <rogpe...@gmail.com> wrote:
> perhaps this is too restrictive? thoughts?

I agree that dynamically loaded code would be awfully nice, but why
not make it resemble static Go packages as much as possible?

dynimport "mypackage"
...
mypackage, err = runtime.Load("/path/to/dynamiclibrary")
...
mypackage.DoSomething()

That is, the only difference between a static and a dynamic package is
that the latter has to be loaded before its contents are referenced.

> the dynamically loaded module has been linked against
> some version of the library code. to make the dynamic code
> loading safe, you'd have to ensure that the loading code
> has been linked against exactly the same version of the
> code, which makes it extremely fragile.

I think that's easy enough to handle. Just hash mypackage's public
interface at (static) compile/link time and verify at dynamic-load
time that the public interface hasn't changed.

-- Scott

[roger peppe]

On 17 May 2010 20:35, Scott Pakin <scot...@pakin.org> wrote:
> On May 17, 3:43 am, roger peppe <rogpe...@gmail.com> wrote:
>> perhaps this is too restrictive? thoughts?
>
> I agree that dynamically loaded code would be awfully nice, but why
> not make it resemble static Go packages as much as possible?
>
>    dynimport "mypackage"
>    ...
>    mypackage, err = runtime.Load("/path/to/dynamiclibrary")

what's the type of mypackage here?

[roger peppe]

On 17 May 2010 20:35, Scott Pakin <scot...@pakin.org> wrote:

the problem is that the public interface isn't all that needs checking.
two dynamically loaded packages can have identical public interfaces
but very different internals. their types should not be interchangeable.

once you start including private data in the hash, that's when things
start to become more fragile.

[Scott Pakin]

It'd have to be some new type as yet not defined by Go.

-- Scott

[Scott Pakin]

roger peppe wrote:
> the problem is that the public interface isn't all that needs checking.
> two dynamically loaded packages can have identical public interfaces
> but very different internals. their types should not be interchangeable.
>
> once you start including private data in the hash, that's when things
> start to become more fragile.

Okay, so let's say that the hash also covers the size and location of
any private data exposed via a public structure. That is,

type MyStruct struct {
Foo int
bar int
Baz string
}

shouldn't match

type MyStruct struct {
Foo int
Baz string
}

or

type MyStruct struct {
Foo int
Baz string
bar int
}

because those structs have different representations. However, a
top-level

var mySecretData string = "JOSHUA"

has no bearing on the package's public interface and therefore, I
argue, should be interchangeable with an implementation that omits or
replaces mySecretData.

-- Scott

[roger peppe]

On 18 May 2010 16:48, Scott Pakin <scot...@pakin.org> wrote:
> Okay, so let's say that the hash also covers the size and location of
> any private data exposed via a public structure.  That is,

unfortunately i don't think that's enough.

suppose you do this.
when then happens about packages that are used by
the dynamically loaded module?

i can see three alternatives:
1) use the code that already exists in the loading binary.
2) use the code from the loaded binary
3) merge the two

the problem with 1) is that suddenly your binaries
become huge because you don't know which
code might be used by the package you're loading.
currently the compiler can do a static check and
eliminate unused functions, but this stops it doing that.

the problem with 3) is that it does require private
variables to be taken into account. hence extreme
fragility.

option 2) is the approach i envisaged - a dynamically
loaded binary comes with all the code it needs.
the dynamic linker can potentially merge identical code segments,
so the runtime size doesn't necessarily blow up.

i envisage that 'import "fmt"' in the dynamically loaded
package refers to an entirely different format package
(with distinct package-scoped variables) from the
fmt package in the loading code.

yes, this makes a dynamically loaded package
something that's more at arm's length from the
static code, but i think perhaps that could be a good thing.

[Scott Pakin]

That's a good point about dynamically loaded modules themselves using
other modules. I don't think it would be unreasonable to say that a
dynamically loaded module are loaded only once into memory, but
statically loaded modules are replicated. That's basically how things
work in the C world.

I don't believe option 3 does require private variables to be taken
into account, other than how they affect structure sizes. Would your
C programs notice if a new version of libc added or removed static
variables somewhere in the code?

If Go were to hash public names (variables, types, constants,
functions, etc.) plus initial values of public variables and
constants into a signature representing a package's interface, I
claim this would work pretty well on its own. If a programmer knew he
was breaking backwards compatibility with a new release of the module
he could easily alter the interface explicitly by sticking in a "const
Version = 2.0" or somesuch.

[roger peppe]

On 19 May 2010 02:52, Scott Pakin <scot...@pakin.org> wrote:
> I don't believe option 3 does require private variables to be taken
> into account, other than how they affect structure sizes.  Would your
> C programs notice if a new version of libc added or removed static
> variables somewhere in the code?

they would if you had two different portions of libc both
using differently named or typed static variables.

here's an example:

// foo version 1
package foo
import "os"
var resource map[int]privateResource
var counter int
type Foo struct {
n int
}
type privateResource struct {
f *os.File
}
func Open() Foo {
counter++
resource[counter], _ = privateResource{os.Open("some resource",
os.O_RDONLY, 0)}
return Foo{counter}
}
func (f Foo) Close() {
resource[f.n].f.Close()
}

// dynamically loaded module
package mymod
import "foo"
func GetFoo() foo.Foo {
return foo.Open()
}

# loading module
package main
import "foo"
dynimport "mymod"

func main() {
mymod, _ := runtime.Load("/path/to/module")
x := mymod.GetFoo()
x.Close()
}

when main is linked, the binary will contain code
for foo.Foo.Close(), because it's called directly, but not
code for foo.Open(), because it doesn't know that
the dynamic module will call it.

when the dynamic module is linked, it will include code
for foo.Open() but not foo.Foo.Close().

but what happens if they're both linked with different
versions of foo, each with different types for the
privateResource struct? or what if one version
doesn't declare the "resource" variable at all?

you could get a run time crash, or memory corruption.

> If Go were to hash public names (variables, types, constants,
> functions, etc.) plus initial values of public variables and
> constants  into a signature representing a package's interface, I
> claim this would work pretty well on its own.  If a programmer knew he
> was breaking backwards compatibility with a new release of the module
> he could easily alter the interface explicitly by sticking in a "const
> Version = 2.0" or somesuch.

go is supposed to be an inherently safe language. relying on the programmer
to preserve program integrity in this way isn't a good thing.

[Stefan Midjich]

2010/5/19 roger peppe <rogp...@gmail.com>

I agree with you that the code demonstrated would be the best way to have dynamically loaded packages in Golang.

However i don't think a safe language is a realistic goal when you're dealing with system programming. If a programmer doesn't want to touch pandoras box, then she doesn't have to load the module that handles dynamic objects. It's as simple as that.

Please think of all the other programmers out there that want a full featured system language.

This hand holding in programming sometimes goes too far.


--

[roger peppe]

On 19 May 2010 11:50, Stefan Midjich <stefan....@gmail.com> wrote:
> However i don't think a safe language is a realistic goal when you're
> dealing with system programming. If a programmer doesn't want to touch
> pandoras box, then she doesn't have to load the module that handles dynamic
> objects. It's as simple as that.

i think a safe language *is* a realistic goal when you're dealing
with system programming, and go is largely (but not quite
- see http://research.swtch.com/2010/02/off-to-races.html) a safe language.

i also think it's perfectly possible to have safely loadable dynamic objects,
and my earlier sketch outlines one way of doing it.

what kind of thing are you wanting to do that my original suggestion would
rule out?

[Scott Pakin]

On Wed, 19 May 2010 10:39:18 +0100, roger peppe <rogp...@gmail.com> wrote:
> but what happens if they're both linked with different
> versions of foo, each with different types for the
> privateResource struct? or what if one version
> doesn't declare the "resource" variable at all?
>
> you could get a run time crash, or memory corruption.

That's why the foo programmer can add a public Version variable to
indicate an invisible but incompatible change, as I suggested in
my previous message.

> go is supposed to be an inherently safe language. relying on the programmer
> to preserve program integrity in this way isn't a good thing.

At some point you have to draw the line, though. Suppose the Go
developers decided that fmt.Printf() should divide all your numbers
into 1000000 before outputting them? If you relink any code that does
an fmt.Printf() of 0, your program could get a run-time crash. Is
that your fault for not knowing that fmt.Printf() changed or the
developers' fault for changing it in an incompatible way? Package
signatures at least *provide* a way for a developer to indicate
explicitly "not backwards compatible".

-- Scott

[Benny Siegert]

Hello,

On Mon, May 17, 2010 at 21:35, Scott Pakin <scot...@pakin.org> wrote:

> I agree that dynamically loaded code would be awfully nice, but why
> not make it resemble static Go packages as much as possible?
>
>    dynimport "mypackage"
>    ...
>    mypackage, err = runtime.Load("/path/to/dynamiclibrary")
>    ...
>    mypackage.DoSomething()
>
> That is, the only difference between a static and a dynamic package is
> that the latter has to be loaded before its contents are referenced.

So essentially, you want something like in Limbo. To load, say, the
"Math" module at runtime, you declare a variable of type Math (i.e.
equivalent to the name of the module):

m: Math;

And then load it by saying

m := load Math Math->PATH;

Functions are then called as m->sin(x), for example. Actually, dynamic
loading is the _only_ way of loading code in the Limbo environment.

However, dynloading would be more interesting if you can load an
arbitrary package, whose name is perhaps read from a configuration
file, and query the loaded package for the existence of an arbitrary
function, like you do with dlopen() and dlsym() in POSIX. However, I
have no idea how to handle new types introduced by the dynamically
loaded module.

--Benny.


--
The first essential in chemistry is that you should perform practical
work and conduct experiments, for he who performs not practical work
nor makes experiments will never attain the least degree of mastery.
-- Abu Musa Jabir ibn Hayyan (721-815)

[roger peppe]

On 19 May 2010 16:09, Benny Siegert <bsie...@gmail.com> wrote:
> So essentially, you want something like in Limbo. To load, say, the
> "Math" module at runtime, you declare a variable of type Math (i.e.
> equivalent to the name of the module):
>
> m: Math;
>
> And then load it by saying
>
> m := load Math Math->PATH;

i'm familiar with inferno, and that's precisely the model i had in mind.

where things become difficult in the inferno world is when you
start passing around modules and adts. usually, a module
can implement a superset of the required functions,
but the moment it's passed around, the entire interface
is set in stone, because you cannot tell which functions
in the module might be called.

adts are a problem because, even though there are no private
adt members, an adt created by one module can freely be
passed to another, which can break things.

> However, dynloading would be more interesting if you can load an
> arbitrary package, whose name is perhaps read from a configuration
> file, and query the loaded package for the existence of an arbitrary
> function, like you do with dlopen() and dlsym() in POSIX. However, I
> have no idea how to handle new types introduced by the dynamically
> loaded module.

this is exactly what go's interfaces provide.

which is why i proposed that runtime.Load could return an interface{}
which could then be queried for its type, the same as any interface
in go.

this raises the question of type compatibility, which i was trying
to address in my original post to this thread.

i haven't yet seen a good objection to that proposal.

[Beoran]

This is becoming a very long discussion, so how about trying to use
CGO to wrap dlopen, dlsym, dlclose and friends on Linux to see how
this could be done? A bit like how libffi was implemented in Ruby or
in Java. If there are no takers I'll try it, but much, much later.

Kind Regards,

B.

[Vivek Khurana]

I dont think wrapper with CGO is a good idea. We do need dynamic code
loading in GO. Without dynamic loading, it will be very difficult to
build plugin/module based applications.
I will suggest taht we write some kind of RFP and have a debate on
the RFP instead of long running email thread.

regards
Vivek

[chris dollin]

On 23 May 2010 17:56, Vivek Khurana <hidden...@gmail.com> wrote:

Without dynamic loading, it will be very difficult to
build plugin/module based applications.

Go plugins might be based on channel communications
(with something like, possibly eq to, netchan for the tween-
process links). Depends how big the unit of work is.

Chris

--
Chris "speculating" Dollin

[roger peppe]

that's pretty heavyweight for a lot of the uses that
plugins might be put to. it's nice to be able to
share memory with your plugin.

and if you do that,avoiding marshalling and unmarshalling, then it's
helpful if you
both agree on types for the communication.

hence my earlier proposal.

[Vivek Khurana]

On May 23, 10:53 pm, chris dollin <ehog.he...@googlemail.com> wrote:
>
> Go plugins might be based on channel communications
> (with something like, possibly eq to, netchan for the tween-
> process links). Depends how big the unit of work is.

May or may not. While channel communication provides a nice option,
it is suitable for small unit of work. If you are building a large
system or framework/metaframework, then channel based communication
might not be of much use.


regards
Vivek

[chris dollin]

On 23 May 2010 19:27, Vivek Khurana <hidden...@gmail.com> wrote:

On May 23, 10:53 pm, chris dollin <ehog.he...@googlemail.com> wrote:
>
> Go plugins might be based on channel communications
> (with something like, possibly eq to, netchan for the tween-
> process links). Depends how big the unit of work is.

 May or may not. While channel communication provides a nice option,
it is suitable for small unit of work.

That's the "Depends" bit. Is it suitable for a small unit of
work? (By "unit of work", I mean how much gets done per
value down the channel. The smaller the unit, the worse
using channels is. The bigger the unit, the less likely it
is that it's the right unit for a specific use.)
 

If you are building a large
system or framework/metaframework, then channel based communication might not be of much use

What sort of considerations had you in mind?

Chris

--
Chris "considering considerations" Dollin

[chris dollin]

On 23 May 2010 19:21, roger peppe <rogp...@gmail.com> wrote:

that's pretty heavyweight for a lot of the uses that
plugins might be put to.

Maybeso.
 

it's nice to be able to
share memory with your plugin.

Can that not be done without dynamically loading stuff into
your codespace?
 

and if you do that,avoiding marshalling and unmarshalling, then it's helpful if you both agree on types for the communication.

Concur.

But I was wondering if plugins-via-channels might be the Go
way. Perhaps with loadable code, but with the only interface
via channels -- then it's only the channel types that have to
match up across the plugin boundaries. (And their components
by recursion ...)
 

Chris

--
Chris "allusive" Dollin

[Camilo Aguilar]

I'm very interested in seeing some kind of actionable items coming out of this discussion. It has been 4 years, are we set with a strategy here?

[ad...@ecm.ub.edu]

This is a really interesting and needed feature. Is there any news on this?

[Ian Lance Taylor]

On Wed, Jul 2, 2014 at 2:49 AM, <ad...@ecm.ub.edu> wrote:
> This is a really interesting and needed feature. Is there any news on this?

Not yet. I hope to have a plan in the next couple of weeks. I don't
know when we will have an implementation.

Ian

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.