An Introduction and a Question

[Chris Wailes]

Hello ruby-llvm group,

I thought I would stop by and introduce myself and then ask a quick question.

My name is Chris and I'm a graduate student at University of Colorado at Boulder.  My research is focused on programming languages and parallel computing, hence my interest in the ruby-llvm bindings!  Lately I've been working on a project called the Ruby Language Toolkit that includes a lexer generator, parser generator, and AST node baseclass.  I plan on using this in conjunction with ruby-llvm for my research, so I'm very interested in contributing to the project.

On to my questions:  I've been translating the LLVM Kaleidoscope tutorial into a Rubified version using RLTK and the ruby-llvm bindings.  So far it has been going very well.  You can see the results so far here.  I am, however, having trouble getting the custom 'extern' calls working.  Specifically I'm talking about the example that can be found here, where a function called 'putchard' is defined in a C file and then linked into the executable, allowing the JIT to call said function.

I attempted to achieve the same effect in Ruby by compiling the C file into a shared library and then using FFI::Library::ffi_lib to load it.  This didn't work, and when I tried to call 'putchard' (after doing a 'extern putchard(char)') I received the following error:

LLVM ERROR: Program used external function 'putchard' which could not be resolved!

Any thoughts on how to get this working properly?

- Chris

P.S.  The full code for the tutorial is located here.  It is still a work in progress, so there might be some differences in the files between chapters.

[Jeremy Voorhis]

Hi Chris,

Welcome to the group! First, I'm very happy to read about RLTK as well as the Kaleidoscope port. I've been wanting to create an example language for ruby-llvm and simply haven't had the time to get started. I'm happy to help.

Making calls to library code that is loaded at runtime or statically linked with LLVM is very easy, but perhaps unintuitive at first. LLVM will only allow you to call functions that are declared within the current module, even if they are not defined within that module. Here is an example from my audio synthesis library: https://github.com/jvoorhis/Siren/blob/master/lib/siren/code_generator.rb#L7-10. Once these functions are declared within the module with the proper type, you can access them with the call instruction as if you defined them yourself. My advice is to use FFI for loading dynamic libs via dlopen(), and use the ruby-llvm api to inform LLVM about the functions you wish to call (or variables you wish to access). This should work equally well for libc (which needn't be loaded), 3rd party libs, or custom runtimes (which are easy to package as dynamic libs).

Hope that helps,

Jeremy

[Chris Wailes]

Currently the tutorial code can make calls to functions in the standard C library after declaring them like so: extern cos(x).  This will cause the function to be declared in the module in a similar way to how you define cosf in your code.  What I currently can't do is write custom C functions and call them from LLVM.  Here is the example function I'm trying to call, and how I compile it.

double putchard(double x) {
    putchar((char) x);
    return 0;
}

gcc -fPIC -c -o kazoo.o kazoo.c
gcc -shared -Wl,-soname,libkazoo.so -o libkazoo.so kazoo.o

I then attempt to load the shared library like so:

module KazooLib
    extend FFI::Library
    ffi_lib 'kazoo'
end

This returns successfully, but when I then try and declare and call the putchard function like so:

extern putchard(char);
putchard(120);

I get this error:  LLVM ERROR: Program used external function 'putchard' which could not be resolved!

It seems to me that the problem is somehow related to how I'm loading the shared library.  Do I need to inform LLVM that a new library has been loaded?

- Chris

 

[Jeremy Voorhis]

Chris,

Can you post the generated LLVM IR for the module you are generating? I find that to be very helpful. You can get at it with Module#dump method. If the solution isn't obvious, then we can dive into it further.

Best,

Jeremy

[Chris Wailes]

Generated IR:

; ModuleID = 'Kazoo JIT'

declare double @putchard(double)

define double @0() {
entry:
  %calltmp = call double @putchard(double 1.200000e+02)
  ret double %calltmp

}
LLVM ERROR: Program used external function 'putchard' which could not be resolved!

- Chris

[Chris Wailes]

Any more thoughts on this?

- Chris

[Jeremy Voorhis]

Hi Chris,

Unfortunately I haven't had time to dig deeper yet. I will post an update as I get the chance.

Best,

Jeremy

[Chris Wailes]

I found the problem here: you need to use LLVM's llvm::sys::DynamicLibrary::LoadLibraryPermanently function to let LLVM know where to look for functions.  There isn't an equivalent C binding for this function, so I wrote my own and integrated it into the latest commit to my branch of ruby-llvm.

- Chris

[Jeremy Voorhis]

Chris,

Thanks for digging in and figuring this out. It looks good. I've deferred creating a support lib for ruby-llvm, so thank you for taking care of all of the boilerplate.

Best,

Jeremy

[Chris Wailes]

Jeremy,

The Makefile is currently static, and is based on values for my machine.  I'd like to get something like Autoconf working to help compile the native library on different machines, if you knew of any Ruby equivalent.  Also, I think there needs to be some changes to the GemSpec, but I don't know what they might be.  If you could point me in the right direction I'd be happy to take care of these issues.  If not, I'll get around to it eventually.

- Chris

[Jeremy Voorhis]

The Ruby equivalent for autotools is a library called mkmf. It's used by writing a file named extconf.rb inside of the ext/ directory. This article is a bit old, but not much has changed: http://www.rubyinside.com/how-to-create-a-ruby-extension-in-c-in-under-5-minutes-100.html. After extconf.rb is created, you add its relative path to the gemspec, as in  http://docs.rubygems.org/read/chapter/20#extensions. I'll be happy to help with this as I get the time.

Best,

Jeremy

[Chris Wailes]

Well, that looks like what I need.  I've got it generating the Makefile now and it can compile the code into a shared library correctly, but I don't know how to check for the presence of the correct version of LLVM or locate and add the llvm archives (such as libLLVMSupport.a) to the shared library.  Any pointers on how to accomplish that would great!

Thanks in advance,
Chris

[Jeremy Voorhis]

You only need to link libLLVM-2.9.dylib (or .so on Linux). It includes everything you get by linking the more granular archive libs.

The mkmf library is badly documented. Here's an extconf.rb that I've written to bind to an OS X framework. https://github.com/jvoorhis/music_player/blob/master/ext/music_player/extconf.rb. You can probably get away with $LDFLAGS = '-l LLVM-2.9'.

HTH,

Jeremy

[Chris Wailes]

It sure did.  I've committed my latest efforts.  If you have any suggestions or comments on any of the work, I'd love to hear them.

- Chris

[Ary Borenszweig]

Nice work!! :-)

Please, can we have this in the original branch? I so want to be able
to make external calls to C functions...

On Jun 15, 4:08 am, Chris Wailes <chris.wai...@gmail.com> wrote:
> It sure did.  I've committed my latest efforts.  If you have any suggestions
> or comments on any of the work, I'd love to hear them.
>
> - Chris
>
>
>
>
>
>
>
> On Thu, Jun 9, 2011 at 5:46 PM, Jeremy Voorhis <jvoor...@gmail.com> wrote:
> > You only need to link libLLVM-2.9.dylib (or .so on Linux). It includes
> > everything you get by linking the more granular archive libs.
>
> > The mkmf library is badly documented. Here's an extconf.rb that I've
> > written to bind to an OS X framework.

> >https://github.com/jvoorhis/music_player/blob/master/ext/music_player....

> > You can probably get away with $LDFLAGS = '-l LLVM-2.9'.
>
> > HTH,
>
> > Jeremy
>
> > On Thursday, June 9, 2011 at 4:25 PM, Chris Wailes wrote:
>
> > Well, that looks like what I need.  I've got it generating the Makefile now
> > and it can compile the code into a shared library correctly, but I don't
> > know how to check for the presence of the correct version of LLVM or locate
> > and add the llvm archives (such as libLLVMSupport.a) to the shared library.
> > Any pointers on how to accomplish that would great!
>
> > Thanks in advance,
> > Chris
>

> > On Fri, Jun 3, 2011 at 3:13 PM, Jeremy Voorhis <jvoor...@gmail.com> wrote:
>
> >  The Ruby equivalent for autotools is a library called mkmf. It's used by
> > writing a file named extconf.rb inside of the ext/ directory. This article
> > is a bit old, but not much has changed:

> >http://www.rubyinside.com/how-to-create-a-ruby-extension-in-c-in-unde....

> > After extconf.rb is created, you add its relative path to the gemspec, as in
> >  http://docs.rubygems.org/read/chapter/20#extensions. I'll be happy to
> > help with this as I get the time.
>
> > Best,
>
> > Jeremy
>
> > On Friday, June 3, 2011 at 2:02 PM, Chris Wailes wrote:
>
> > Jeremy,
>
> > The Makefile is currently static, and is based on values for my machine.
> > I'd like to get something like Autoconf working to help compile the native
> > library on different machines, if you knew of any Ruby equivalent.  Also, I
> > think there needs to be some changes to the GemSpec, but I don't know what
> > they might be.  If you could point me in the right direction I'd be happy to
> > take care of these issues.  If not, I'll get around to it eventually.
>
> > - Chris
>

> > On Fri, Jun 3, 2011 at 2:54 PM, Jeremy Voorhis <jvoor...@gmail.com> wrote:
>
> >  Chris,
>
> > Thanks for digging in and figuring this out. It looks good. I've deferred
> > creating a support lib for ruby-llvm, so thank you for taking care of all of
> > the boilerplate.
>
> > Best,
>
> > Jeremy
>
> > On Friday, June 3, 2011 at 1:01 PM, Chris Wailes wrote:
>
> > I found the problem here: you need to use LLVM's
> > llvm::sys::DynamicLibrary::LoadLibraryPermanently function to let LLVM know
> > where to look for functions.  There isn't an equivalent C binding for this
> > function, so I wrote my own and integrated it into the latest commit to my
> > branch of ruby-llvm.
>
> > - Chris
>

> > On Thu, Jun 2, 2011 at 12:12 PM, Jeremy Voorhis <jvoor...@gmail.com>wrote:
>
> >  Hi Chris,
>
> > Unfortunately I haven't had time to dig deeper yet. I will post an update
> > as I get the chance.
>
> > Best,
>
> > Jeremy
>
> > On Thursday, June 2, 2011 at 9:19 AM, Chris Wailes wrote:
>
> > Any more thoughts on this?
>
> > - Chris
>

> > On Fri, May 27, 2011 at 12:55 PM, Chris Wailes <chris.wai...@gmail.com>wrote:
>
> > Generated IR:
>
> > ; ModuleID = 'Kazoo JIT'
>
> > declare double @putchard(double)
>
> > define double @0() {
> > entry:
> >   %calltmp = call double @putchard(double 1.200000e+02)
> >   ret double %calltmp
>
> > }
> > LLVM ERROR: Program used external function 'putchard' which could not be
> > resolved!
>
> > - Chris
>

> > On Thu, May 26, 2011 at 11:09 PM, Jeremy Voorhis <jvoor...@gmail.com>wrote:
>
> >  Hi Chris,
>
> > Welcome to the group! First, I'm very happy to read about RLTK as well as
> > the Kaleidoscope port. I've been wanting to create an example language for
> > ruby-llvm and simply haven't had the time to get started. I'm happy to help.
>
> > Making calls to library code that is loaded at runtime or statically linked
> > with LLVM is very easy, but perhaps unintuitive at first. LLVM will only
> > allow you to call functions that are declared within the current module,
> > even if they are not defined within that module. Here is an example from my
> > audio synthesis library:

> >https://github.com/jvoorhis/Siren/blob/master/lib/siren/code_generato....

> > Once these functions are declared within the module with the proper type,
> > you can access them with the call instruction as if you defined them
> > yourself. My advice is to use FFI for loading dynamic libs via dlopen(), and
> > use the ruby-llvm api to inform LLVM about the functions you wish to call
> > (or variables you wish to access). This should work equally well for libc
> > (which needn't be loaded), 3rd party libs, or custom runtimes (which are
> > easy to package as dynamic libs).
>
> > Hope that helps,
>
> > Jeremy
>
> > On Thursday, May 26, 2011 at 4:07 PM, Chris Wailes wrote:
>
> > Hello ruby-llvm group,
>
> > I thought I would stop by and introduce myself and then ask a quick
> > question.
>
> > My name is Chris and I'm a graduate student at University of Colorado at
> > Boulder.  My research is focused on programming languages and parallel
> > computing, hence my interest in the ruby-llvm bindings!  Lately I've been

> > working on a project called the Ruby Language Toolkit<http://chris.wailes.name/?page_id=51>that includes a lexer generator, parser generator, and AST node baseclass.

> > I plan on using this in conjunction with ruby-llvm for my research, so I'm
> > very interested in contributing to the project.
>
> > On to my questions:  I've been translating the LLVM Kaleidoscope tutorial
> > into a Rubified version using RLTK and the ruby-llvm bindings.  So far it

> > has been going very well.  You can see the results so far here<http://chris.wailes.name/?page_id=97>.

> > I am, however, having trouble getting the custom 'extern' calls working.

> > Specifically I'm talking about the example that can be found here<http://llvm.org/docs/tutorial/OCamlLangImpl4.html>,

> > where a function called 'putchard' is defined in a C file and then linked
> > into the executable, allowing the JIT to call said function.
>
> > I attempted to achieve the same effect in Ruby by compiling the C file into
> > a shared library and then using FFI::Library::ffi_lib to load it.  This
> > didn't work, and when I tried to call 'putchard' (after doing a 'extern
> > putchard(char)') I received the following error:
>
> > LLVM ERROR: Program used external function 'putchard' which could not be
> > resolved!
>
> > Any thoughts on how to get this working properly?
>
> > - Chris
>

> > P.S.  The full code for the tutorial is located here<https://code.launchpad.net/%7Echris-wailes/rltk/llvm-example>.