Dynamic library loading feasible?

[Steven Degutis]

One of the key features of languages like Ruby and Python are native
extensions. That is, extensions that are written in C, compiled into a
library, and loaded into the the language at runtime dynamically,
probably using dlopen().

This currently doesn't seem possible with Go. If another dynamic
language were written in Go, then anyone who wants to use libxml2 with
it must recompile the language to have libxml support built-in. That's
obviously not feasible.

Is it feasible to add some kind of dynamic library loading to Go? I
know there are some who disagree with the concept in the first place.
But is that the only reason it's not available, or is there technical
obstacle to implementing it?

Thanks,

-Steven Degutis

[Ian Lance Taylor]

Steven Degutis <sdeg...@8thlight.com> writes:

> One of the key features of languages like Ruby and Python are native
> extensions. That is, extensions that are written in C, compiled into a
> library, and loaded into the the language at runtime dynamically,
> probably using dlopen().

Both cgo and SWIG support this approach (it's probably a bit easier with
SWIG at the moment). You can only dynamically load code written in
C/C++, though. There is, ironically, no way to dynamically load code
written in Go. That is a known limitation which will be addressed at
some point, but not before Go 1.

Ian

[Steven Degutis]

So the main problem isn't loading C code, but interoperability between
the dynamically loaded C code and the currently running Go code. The C
code has no way of using non-primitive types that the Go code can
recognize.

Unless... is there some way to know precisely what kind of a C struct
would map to a given Go struct, in a way where I can call C functions
from Go, passing in the Go types and the C function getting the
corresponding C struct it expects and understands?

-Steven


On Feb 12, 2:04 pm, Ian Lance Taylor <i...@google.com> wrote:

[Ian Lance Taylor]

Steven Degutis <sdeg...@8thlight.com> writes:

> So the main problem isn't loading C code, but interoperability between
> the dynamically loaded C code and the currently running Go code. The C
> code has no way of using non-primitive types that the Go code can
> recognize.
>
> Unless... is there some way to know precisely what kind of a C struct
> would map to a given Go struct, in a way where I can call C functions
> from Go, passing in the Go types and the C function getting the
> corresponding C struct it expects and understands?

This is hard to answer in the abstract. Both cgo and SWIG are about
interoperability between Go and C/C++. Take a look at them. cgo is
part of the regular Go distribution.

Ian

[brainman]

All Go syscalls are implemented by calling "dynamic library". namely dlls
I can't see why something similar could not be implemented for similar technology on linux.

Alex

[Paulo Pinto]

What I miss is some mechanism similar to what Delphi, Ada, D, .NET,
among
other languages have, where there are special languages constructs to
bind with
foreign code, including DLLs.

Currently, the way syscalls work is a bit hackish with Perl scripts
going through
comments and generating the corresponding syscall calls.

It works pretty well I would say, but I would rather have a language
support feature,
based on my experience with other languages.

--
Paulo

[Ian Lance Taylor]

Paulo Pinto <paulo....@gmail.com> writes:

> What I miss is some mechanism similar to what Delphi, Ada, D, .NET,
> among
> other languages have, where there are special languages constructs to
> bind with
> foreign code, including DLLs.
>
> Currently, the way syscalls work is a bit hackish with Perl scripts
> going through
> comments and generating the corresponding syscall calls.

That is true but you should not confuse that Perl script with binding
with DLLs. That Perl script is used by the Go developers to build the
syscall library in the first place. It is not run when building a Go
program, and it is not even run when building a Go distribution.

Go programs are normally statically linked and do not link with DLLs at
all. You can use programs like cgo and SWIG to link with DLLs. cgo and
SWIG do not use Perl scripts.

The Go tools actually do have language constructs similar to, e.g., the
Ada bindings. They just aren't advertised, but are instead used
internally by cgo and SWIG. The constructs are expressed as #pragma's
in .c files generated by cgo and SWIG.

Ian

[Paulo Pinto]

Thanks for replying Ian.

My comment was based on my short experience with Go, when I did the
inital
contribution to os.user and the windows draw package.

--
Paulo

On Feb 13, 6:28 pm, Ian Lance Taylor <i...@google.com> wrote:

[Aram Hăvărneanu]

Paulo Pinto wrote:
> You really did not understood my statement.
>
> I would like to use higher level mechanism that the mentioned languages
> provide, without the
> need to have a C compiler installed.

You said nothing about the requirement of not having a C compiler in
your original post.

Please note that Go allows arbitrary mixing of C and Go while Delphi
and .NET only allow calling stdcall functions from DLLs. Go's
mechanism is richer, also portable and it doesn't require specifying
function signatures.

If all you want is to call stdcall functions from DLLs, why don't you
just call LoadLibrary() from your Go code and call the function
directly? It requires package unsafe, it doesn't require cgo or C
compilers.

> 1 - Make sure Perl, Bash and make are available
> 2 - Add the imports in the form of comments like
>   //sys   GetModuleHandle(modname *uint16) (handle syscall.Handle, err
> error) = GetModuleHandleW
> 3 - Create the following make targets in the make file
>
> target-api.go: source-api.go
>         (echo '// +build windows'; \
>         $(GOROOT)/src/pkg/syscall/mksyscall_windows.pl $<) \
>                 | gofmt \
>                 > $@

I don't understand why you posted this.

> 1 - Make sure a cgo compatible C compiler is available
> 2 - Use cgo

As I said, if all you want is call stdcall functions from DLLs call
LoadLibary() directly.

kernel32, _ = syscall.LoadLibrary("kernel32.dll")
getModuleHandle, _ = syscall.GetProcAddress(kernel32, "GetModuleHandleW")

> But I should refrain myself to give opinions on this forum as Go is not
> really for me.

This attitude can't help you.

--
Aram Hăvărneanu

[Paulo Pinto]

You really did not understood my statement.

I would like to use higher level mechanism that the mentioned languages provide, without the
need to have a C compiler installed.

With Delphi:
----------------
procedure DllMessage; external 'SimpleMessageDLL.dll'

With D:
-----------
extern (C) int strcmp(char* string1, char* string2);

With Ada:
--------------
with Interfaces.C.Strings;
package API is
    use Interfaces;
    
    Some_Var : C.int;
    function Get (Str : C.Strings.Chars_Ptr) return C.int;
    
 private
    pragma Import (C, Get);
    pragma Import (DLL, Some_Var);
end API;

With .NET using C#:
[DllImport("msvcrt.dll")]
public static extern int puts(string c);    


With Go:
------------

1 - Make sure Perl, Bash and make are available
2 - Add the imports in the form of comments like
  //sys   GetModuleHandle(modname *uint16) (handle syscall.Handle, err error) = GetModuleHandleW
3 - Create the following make targets in the make file

target-api.go: source-api.go
        (echo '// +build windows'; \
        $(GOROOT)/src/pkg/syscall/mksyscall_windows.pl $<) \
                | gofmt \
                > $@

or,

1 - Make sure a cgo compatible C compiler is available
2 - Use cgo

Hence I disagree with you that the way Go does it now is the way to "go", specially on
platforms like Windows.

But I should refrain myself to give opinions on this forum as Go is not really for me.

--
Paulo

On Mon, Feb 13, 2012 at 2:06 PM, Aram Hăvărneanu <ara...@mgk.ro> wrote:

Paulo Pinto wrote:
> What I miss is some mechanism similar to what Delphi, Ada, D, .NET,
> among
> other languages have, where there are special languages constructs to
> bind with
> foreign code, including DLLs.

Eh? D can call C, Ada can call C. There's no portable way for D or Ada
to call C++, though it is possible in restricted circumstances with
some pain. Shared libraries are usually out of bounds of those
restricted circumstances. There's no language feature involved in any
of this.

Of course Go can call C just as easy as D or Ada, and of course it can
link with shared libraries.

Delphi can link with C with much pain, it forces the programmer to
explicitly state call conventions and functions prototypes. Delphi can
link with C++ only if the programmer manages the ABI conversion
himself. It significantly more difficult to use C code from Delphi
compared with Go.

.NET stuff can call C, not C++ code with a horrible mechanism called
P/Invoke which is even more difficult to use than the Delphi
mechanism.

.NET however, can natively call into COM objects, the COM objects can
be written in any language, most commonly C++. I believe Delphi can do
this as well. COM is a very specific technology.

I believe your statements are unfounded, Go can call C code from
shared libraries just as easy as D or Ada. The other languages
mentioned make calling foreign code much harder.

> Currently, the way syscalls work is a bit hackish with Perl scripts
> going through
> comments and generating the corresponding syscall calls.

Syscalls work by having the Go runtime linked in the binary issue the
syscalls directly, as opposed to calling through a C library (except
Windows maybe, I don't know). It's essential the runtime and the
programmer know details about the kernel interface and these details
are encoded in C headers. Everything is already documented, redoing
the work would be wasteful and extracting what Go needs automatically,
with scripts is easy and ensures some degree of compatibility in the
future.

> It works pretty well I would say, but I would rather have a language
> support feature

Language support feature to guess the kernel interface?

--
Aram Hăvărneanu

[Ian Lance Taylor]

Paulo Pinto <paulo....@gmail.com> writes:

> I would like to use higher level mechanism that the mentioned languages
> provide, without the
> need to have a C compiler installed.

Perhaps we will implement a mechanism for that some day. You can do it
in principle, but you have to calculate various things yourself.
Currently cgo uses gcc to perform that calculation for you.

> With Go:
> ------------
>
> 1 - Make sure Perl, Bash and make are available
> 2 - Add the imports in the form of comments like
> //sys GetModuleHandle(modname *uint16) (handle syscall.Handle, err
> error) = GetModuleHandleW
> 3 - Create the following make targets in the make file
>
> target-api.go: source-api.go
> (echo '// +build windows'; \
> $(GOROOT)/src/pkg/syscall/mksyscall_windows.pl $<) \
> | gofmt \
> > $@

Please never use this approach.

> 1 - Make sure a cgo compatible C compiler is available
> 2 - Use cgo

Yes, that works.

Another approach: use SWIG.

Ian

[Guillermo Estrada]

I know that D language has a project I used a lot, with the ability to load shared libraries at runtime and you only had to create the bindings to the library in D (No need for C compilers, toolchains, c code, libraries, etc...) on Windows you just need the dll to run it, but you don't need anything but your bindings to compile it. It's called Derelict (http://www.dsource.org/projects/derelict). That truly made my life in D a bliss, I wrote bindings for SQLite, GLFW, BASS, and so on by just reading header files in C of those libraries and applying some rules to port them to D. Now I don't use D anymore in favor of Go, but that was an awesome way I could get going with most C libs. I wish that I can make something about that, but it would be awesome to have something to just load the shared library (.dll, .so, etc...) at runtime, then any .dll binary I download for windows can work without having to build every library myself.

[minux]

On Fri, Apr 13, 2012 at 11:29 PM, Guillermo Estrada <phro...@gmail.com> wrote:

I know that D language has a project I used a lot, with the ability to load shared libraries at runtime and you only had to create the bindings to the library in D (No need for C compilers, toolchains, c code, libraries, etc...) on Windows you just need the dll to run it, but you don't need anything but your bindings to compile it. It's called Derelict (http://www.dsource.org/projects/derelict). That truly made my life in D a bliss, I wrote bindings for SQLite, GLFW, BASS, and so on by just reading header files in C of those libraries and applying some rules to port them to D. Now I don't use D anymore in favor of Go, but that was an awesome way I could get going with most C libs. I wish that I can make something about that, but it would be awesome to have something to just load the shared library (.dll, .so, etc...) at runtime, then any .dll binary I download for windows can work without having to build every library myself.

I've sketched a proposal for the ffi interface in go, modeled after LuaJIT's ffi interface:

One example:

package main

import "ffi"

var printf ffi.Function // something like: func (args ...interface{}) (ffi.RetVal)

func init() {

   ffi.Declare(`int printf(const char *, ...);`)

   printf = ffi.F("printf")

}

func main() {

   printf("%d\n", printf("hello world\n").Int())

}

If there is sufficient interest, I will try to implement a prototype (it will use dynamic code generation so

I think it could be as efficient as using cgo).

[Guillermo Estrada]

I've sketched a proposal for the ffi interface in go, modeled after LuaJIT's ffi interface:

One example:

package main

import "ffi"

var printf ffi.Function // something like: func (args ...interface{}) (ffi.RetVal)

func init() {

   ffi.Declare(`int printf(const char *, ...);`)

   printf = ffi.F("printf")

}

func main() {

   printf("%d\n", printf("hello world\n").Int())

}

If there is sufficient interest, I will try to implement a prototype (it will use dynamic code generation so

I think it could be as efficient as using cgo).

Well at least every Windows dev that uses Go will love you. Not having to compile every library we want to use would be awesome, i can get rid of my mingw msys installation and go get will work with any dll we can download. LuaJit is astonishing well performant and if the binding procedure is as easy as it seems, lots of developers will start making bindings to dynamically load every single C shared library out there (me included).

[Sebastien Binet]

minux <minu...@gmail.com> writes:

> On Fri, Apr 13, 2012 at 11:29 PM, Guillermo Estrada <phro...@gmail.com>wrote:
>
>> I know that D language has a project I used a lot, with the ability to
>> load shared libraries at runtime and you only had to create the bindings to
>> the library in D (No need for C compilers, toolchains, c code, libraries,
>> etc...) on Windows you just need the dll to run it, but you don't need
>> anything but your bindings to compile it. It's called Derelict (

>> http://www.dsource.org/**projects/derelict<http://www.dsource.org/projects/derelict>).

>> That truly made my life in D a bliss, I wrote bindings for SQLite, GLFW,
>> BASS, and so on by just reading header files in C of those libraries and
>> applying some rules to port them to D. Now I don't use D anymore in favor
>> of Go, but that was an awesome way I could get going with most C libs. I
>> wish that I can make something about that, but it would be awesome to have
>> something to just load the shared library (.dll, .so, etc...) at runtime,
>> then any .dll binary I download for windows can work without having to
>> build every library myself.
>
> I've sketched a proposal for the ffi interface in go, modeled after
> LuaJIT's ffi interface:
> One example:
>
> package main
> import "ffi"
>
> var printf ffi.Function // something like: func (args ...interface{})
> (ffi.RetVal)
> func init() {
> ffi.Declare(`int printf(const char *, ...);`)
> printf = ffi.F("printf")
> }
>
> func main() {
> printf("%d\n", printf("hello world\n").Int())
> }
>
> If there is sufficient interest, I will try to implement a prototype (it
> will use dynamic code generation so
> I think it could be as efficient as using cgo).

you'd be my hero.

-s

[Andrew Wilkins]

On Friday, 13 April 2012 23:44:03 UTC+8, minux wrote:

I've sketched a proposal for the ffi interface in go, modeled after LuaJIT's ffi interface:

...

If there is sufficient interest, I will try to implement a prototype (it will use dynamic code generation so

I think it could be as efficient as using cgo).

I'm interested, though not 100% sure I'll use it in the short term. I want to remove the direct

dependency on libLLVM from llgo, to make distribution easier. I was considering wrapping libltdl, but

your proposal would be much friendlier.

Cheers,

Andrew

[LRN]

gccgo already supports dynamic linking. [1]

[1] http://gcc.gnu.org/ml/gcc-help/2011-10/msg00233.html

[Sebastien Binet]

LRN <lrn...@gmail.com> writes:

> gccgo already supports dynamic linking. [1]
>
> [1] http://gcc.gnu.org/ml/gcc-help/2011-10/msg00233.html

nice.
but I am myself more interested in dynamic code loading rather than
dynamic linking.

IIUC, the api minux was devising is hinted at dynamic code loading for
C, which, I suppose, is the first wedge to get the same for Go :}

-s

PS: minux, please CC-me when/if you progress on the ffi github repo,
golang-dev or whatever.

[Sebastien Binet]

one more thing which could be nice to consider providing:
being able to automatically or at least rather seamlessly wrap
C-callback functions would be really nifty.

the python ctypes module does this and this has been really convenient.

right now, with cgo, the situation is a bit keyboard heavy, especially
if one wants to implement the callback in go (most probably via a
closure) and pass it down to C.
it is doable, but a lot of scaffolding and boilerplate code is needed.

-s