Re: libtcc, FFI, C::Dynalib, and friends!

116 views

Skip to first unread message

Reini Urban

unread,

Apr 2, 2012, 1:27:22 PM4/2/12

to David Mertens, perl-c...@googlegroups.com, Ryan Jendoubi, Gaal Yahas, Anatoly Vorobey, shl...@shlomifish.org

On Mon, Apr 2, 2012 at 11:15 AM, David Mertens <dcmerte...@gmail.com> wrote:

On Mon, Apr 2, 2012 at 10:59 AM, Ryan Jendoubi <ryan.j...@gmail.com> wrote:

On 02/04/12 16:16, David Mertens wrote:

To clarify: I discovered libffi two days ago and started playing with the FFI module yesterday. I discovered CTypes this morning. I communicated with Gaal and Antoly about FFI yesterday, but my recent discovery of CTypes and Ryan's recent rekindling of interest in the project has lead me to this discussion.

But you did not discover http://search.cpan.org/dist/Jit yet :)

For the two of you who are not aware of me until now, my name is David Mertens. I'm a PDL core dev and I've recently taken an interest in writing a Perl wrapper for libtcc. TCC is the Tiny C Compiler, and its lib interface can take a string of code and compile it to machine code on x86, x86-64, and ARM processors. It was designed and optimized to compile quickly, which is ideal for a C jit compiler. It is very close to supporting multiple compiler contexts, which would be ideal for a Perl C-jit compiler, and I have offered to help out on that.

Sounds cool!

The problem, as I'm sure you can guess, is that calling the compiled C functions from Perl is tricky. I had considered writing some sort of function call wrapper, which is a somewhat acceptable solution since the C source code is jit-compiled, and can be tweaked to match whatever interface I impose. However, the C::DynaLib, FFI, and CTypes modules all provide a potentially more elegant means for handling C function calls.

Please, do not use a FFI for a simple jit. It makes matters much more complicated.

Look at B::C, B::CC and Jit instead.

These are the grounds to produce a perl jitter.

Calling a jitted code is easy:

#ifdef HAS_MPROTECT

if (mprotect(code,size*sizeof(char),PROT_EXEC|PROT_READ|PROT_WRITE) < 0)

croak ("mprotect code=0x%x for size=%u failed", code, size);

#endif

For windows it's a similar VirtualAlloc call. Look at various jitters.

Producing the intermediate c code for a jitter is not easy.

Look at Runops::Switch (best my github fork), which would be the best start.

The goal is to eliminate the runloop overhead and maybe avoid the 'heap-stack'

pp API for simple ops.

A FFI can be used to replace the XS API in the longer term, to go back from the heap stack to a C stack. Since we already count our recursion depth in CX there's no other technical problem.

See http://corehackers.perl.org/wiki/index.php5?title=C_stack

Being able to strictly type our data would also help for faster execution (jitted or B::CC), but I'm still working on that.

See http://blogs.perl.org/users/rurban/2011/02/use-types.html

I am writing in the hopes of getting everybody together who cares about this so that we can all converge our tuits upon one project. I noticed that Ryan has picked up work again (very recently) on CTypes, which as we all know uses libffi rather than libffcall. *Just Yesterday* I had worked on some tweaks to FFI.xs to use libffi instead of libffcall, but I have not fully merged them into the original code and now wonder if my interests would be better served helping Ryan on CTypes.

So, these are my questions: (1) What modules are you currently interested in supporting? (2) Is there a way to work together on one module, and turn the others into backwards-compatible wrappers for the agreed-upon module? (3) Is there any sort of community for this work? A mailing list? An IRC channel?

My tentative opinion is that we should throw our weight behind CTypes, since it has not yet hit CPAN and is therefore more flexible than the others. However, I am certainly open to arguments for either of the others.

The languishing of Ctypes in its disgusting, never-quite-usable state is a source of continued shame for me. I never lost 'interest' in it per se, I just started studying a law degree :-/

I want to see it come to fruition. I'm currently staring down the muzzle of my finals in June, but have been trying to tidy up Ctypes to relax where I can, keeping pretty much under the radar, hoping to eventually have something functional and well-documented enough to at least publicise again and ask for help with.

1) As I say, I want to see Ctypes working. It's also the only thing I'd be placed to help with, in all likelihood. I remember getting my head around how libffcall worked when researching for the Ctypes project, and came to the conclusion that in comparison to libffi its capabilities were limited - can't remember why at this point though.

2) I very much hope so. That said, I cringe horribly at the idea of other people looking at the Ctypes code. Reini will attest to its inherent ugliness (when you get a hold of him - I don't have another address, sorry). I guess the best thing I could do is dramatically improve the documentation, and remove the tens of thousands of useless tests.

3) Not that I know of. perl-xs would be the closest I guess, but doesn't seem the place. Let's start one? :-)

I've CC'd Shlomi Fish, as he recently expressed an interest in Ctypes (specifically how broken the test suite is).

For your interest, in addition to what you mentioned above I looked at the dyncall C library (dyncall.org; dismissed on the basis of negligible uptake elsewhere) and chromatic's P5NCI Perl module (cool API, but inherently limited and explicitly a proof-of-concept).

So, those are my few cents. Main practical takeaway is my lack of tuits I guess. What do you make of Gaal's suggestion that FFI.pm might meet your needs without you having to go on a mission? :-)

Bests,

-r

Ah. I had assumed that ffcall and ffi were the same libraries (one being the predecessor of the other), but now I realize they're distinct. In that case, another issue worth exploring is ease of installing the base C library with Alien::Base.

I thought of storing those kind of discussions on the perl-c...@googlegroups.com maling list archive since p5p is technically challenged in this regard, and usually comes up with incredible stupid remarks.

I used to put my ideas here:

http://corehackers.perl.org/wiki/index.php5?title=Main_Page#Performance_Hacks

Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

Reini Urban

unread,

Apr 3, 2012, 11:07:52 AM4/3/12

to David Mertens, perl-c...@googlegroups.com, Ryan Jendoubi, Gaal Yahas, Anatoly Vorobey, shl...@shlomifish.org

On Mon, Apr 2, 2012 at 1:27 PM, David Mertens <dcmerte...@gmail.com> wrote:

On Mon, Apr 2, 2012 at 12:27 PM, Reini Urban <rur...@x-ray.at> wrote:

On Mon, Apr 2, 2012 at 11:15 AM, David Mertens <dcmerte...@gmail.com> wrote:

On Mon, Apr 2, 2012 at 10:59 AM, Ryan Jendoubi <ryan.j...@gmail.com> wrote:

On 02/04/12 16:16, David Mertens wrote:

To clarify: I discovered libffi two days ago and started playing with the FFI module yesterday. I discovered CTypes this morning. I communicated with Gaal and Antoly about FFI yesterday, but my recent discovery of CTypes and Ryan's recent rekindling of interest in the project has lead me to this discussion.

But you did not discover http://search.cpan.org/dist/Jit yet :)

Yes, I have discovered that, but I have no idea how to use it, and I think you and I are tackling different beasts. Here, for example, is some working code that I wrote this morning that uses TCC and Ctypes:

__TEST.PL__
use strict;
use warnings;
use TCC;
my $context = TCC->new();
$context->code('Body') .= q{
    int triple(int input) {
        return input * 3;
    }
};
$context->compile;

my $p_func = $context->get_symbol('triple');

use blib;
use Ctypes;
my $result = Ctypes::call($p_func, 'cii', 42);
print "Got $result, expect ", 42*3, "\n";
__END__

See? I wish to create C functions at run time, not Perl-to-C-code at compile time.

Oh, I see. A fast Inline::C.

Yes, Ctypes seems to best.

I have time now as I am stuck with the compiler and the latest release is pretty stable.

I will try to fix and release Ctypes on CPAN.

I also just released my updated Jit-0.05 yesterday.

Back to perl-TCC:

I tried perl-TCC from github and tinycc from debian, github master and github mob.

I guess it only works with a static libperl.a, not shared.

BTW the libtcc _relocate API changed from debian to git/master. You are using the latest/

I tried linking to libtcc.a and libtcc.so.1.0

I always got tcc_relocate errors:

t/111-compile-include.t (Wstat: 2048 Tests: 9 Failed: 8)

Failed tests: 1-2, 4-9

t/112-compile-define.t (Wstat: 3840 Tests: 17 Failed: 15)

Failed tests: 3-17

but the other tests do work fine.

Should we use libtcc.a or shared?

Could you describe your env and versions?

For the two of you who are not aware of me until now, my name is David Mertens. I'm a PDL core dev and I've recently taken an interest in writing a Perl wrapper for libtcc. TCC is the Tiny C Compiler, and its lib interface can take a string of code and compile it to machine code on x86, x86-64, and ARM processors. It was designed and optimized to compile quickly, which is ideal for a C jit compiler. It is very close to supporting multiple compiler contexts, which would be ideal for a Perl C-jit compiler, and I have offered to help out on that.

Sounds cool!

The problem, as I'm sure you can guess, is that calling the compiled C functions from Perl is tricky. I had considered writing some sort of function call wrapper, which is a somewhat acceptable solution since the C source code is jit-compiled, and can be tweaked to match whatever interface I impose. However, the C::DynaLib, FFI, and CTypes modules all provide a potentially more elegant means for handling C function calls.

Please, do not use a FFI for a simple jit. It makes matters much more complicated.
Look at B::C, B::CC and Jit instead.

These are the grounds to produce a perl jitter.

Calling a jitted code is easy:
#ifdef HAS_MPROTECT
if (mprotect(code,size*sizeof(char),PROT_EXEC|PROT_READ|PROT_WRITE) < 0)

croak ("mprotect code=0x%x for size=%u failed", code, size);
#endif
For windows it's a similar VirtualAlloc call. Look at various jitters.

Producing the intermediate c code for a jitter is not easy.

Look at Runops::Switch (best my github fork), which would be the best start.

The goal is to eliminate the runloop overhead and maybe avoid the 'heap-stack'
pp API for simple ops.

A FFI can be used to replace the XS API in the longer term, to go back from the heap stack to a C stack. Since we already count our recursion depth in CX there's no other technical problem.
See http://corehackers.perl.org/wiki/index.php5?title=C_stack

Being able to strictly type our data would also help for faster execution (jitted or B::CC), but I'm still working on that.

See http://blogs.perl.org/users/rurban/2011/02/use-types.html

Yeah, I'm somewhat familiar with this work, and see it as a complement to what I'm trying to do.

I am writing in the hopes of getting everybody together who cares about this so that we can all converge our tuits upon one project. I noticed that Ryan has picked up work again (very recently) on CTypes, which as we all know uses libffi rather than libffcall. *Just Yesterday* I had worked on some tweaks to FFI.xs to use libffi instead of libffcall, but I have not fully merged them into the original code and now wonder if my interests would be better served helping Ryan on CTypes.

So, these are my questions: (1) What modules are you currently interested in supporting? (2) Is there a way to work together on one module, and turn the others into backwards-compatible wrappers for the agreed-upon module? (3) Is there any sort of community for this work? A mailing list? An IRC channel?

My tentative opinion is that we should throw our weight behind CTypes, since it has not yet hit CPAN and is therefore more flexible than the others. However, I am certainly open to arguments for either of the others.

The languishing of Ctypes in its disgusting, never-quite-usable state is a source of continued shame for me. I never lost 'interest' in it per se, I just started studying a law degree :-/

I want to see it come to fruition. I'm currently staring down the muzzle of my finals in June, but have been trying to tidy up Ctypes to relax where I can, keeping pretty much under the radar, hoping to eventually have something functional and well-documented enough to at least publicise again and ask for help with.

1) As I say, I want to see Ctypes working. It's also the only thing I'd be placed to help with, in all likelihood. I remember getting my head around how libffcall worked when researching for the Ctypes project, and came to the conclusion that in comparison to libffi its capabilities were limited - can't remember why at this point though.

2) I very much hope so. That said, I cringe horribly at the idea of other people looking at the Ctypes code. Reini will attest to its inherent ugliness (when you get a hold of him - I don't have another address, sorry). I guess the best thing I could do is dramatically improve the documentation, and remove the tens of thousands of useless tests.

3) Not that I know of. perl-xs would be the closest I guess, but doesn't seem the place. Let's start one? :-)

I've CC'd Shlomi Fish, as he recently expressed an interest in Ctypes (specifically how broken the test suite is).

For your interest, in addition to what you mentioned above I looked at the dyncall C library (dyncall.org; dismissed on the basis of negligible uptake elsewhere) and chromatic's P5NCI Perl module (cool API, but inherently limited and explicitly a proof-of-concept).

So, those are my few cents. Main practical takeaway is my lack of tuits I guess. What do you make of Gaal's suggestion that FFI.pm might meet your needs without you having to go on a mission? :-)

Bests,

-r

Ah. I had assumed that ffcall and ffi were the same libraries (one being the predecessor of the other), but now I realize they're distinct. In that case, another issue worth exploring is ease of installing the base C library with Alien::Base.

I thought of storing those kind of discussions on the perl-c...@googlegroups.com maling list archive since p5p is technically challenged in this regard, and usually comes up with incredible stupid remarks.

I used to put my ideas here:
http://corehackers.perl.org/wiki/index.php5?title=Main_Page#Performance_Hacks

--
Reini Urban
http://cpanel.net/ http://www.perl-compiler.org/

I see. In light of what I'm trying to do---compile and call C code from Perl, without using Inline::C---do you still think that perl-c...@googlegroups.com is the appropriate forum?

At least better than nothing at all.

There's also the inline group: inline-s...@perl.org

http://www.nntp.perl.org/group/perl.inline/

but there are only talking about Inline itself, not about the architecture.

David Mertens

unread,

Apr 3, 2012, 12:18:55 PM4/3/12

to Reini Urban, perl-c...@googlegroups.com, Ryan Jendoubi, Gaal Yahas, Anatoly Vorobey, shl...@shlomifish.org

On Tue, Apr 3, 2012 at 10:07 AM, Reini Urban <rur...@x-ray.at> wrote:

I tried perl-TCC from github and tinycc from debian, github master and github mob.
I guess it only works with a static libperl.a, not shared.

I am fairly certain that it does not require a static perl, but I'm not 100% certain about all of it.

BTW the libtcc _relocate API changed from debian to git/master. You are using the latest/

Yes. I should note that in the README. See the instructions below, and and if they work I'll add them there.

I tried linking to libtcc.a and libtcc.so.1.0
I always got tcc_relocate errors:
t/111-compile-include.t (Wstat: 2048 Tests: 9 Failed: 8)
Failed tests: 1-2, 4-9

t/112-compile-define.t (Wstat: 3840 Tests: 17 Failed: 15)
Failed tests: 3-17
but the other tests do work fine.

Should we use libtcc.a or shared?

Could you describe your env and versions?

Right now I'm working from Ubuntu 10.10. Here are (what I hope are) full instructions for getting libtcc installed in such a way that it'll work with these bindings. I have only gotten this working on Linux. I have not tested on Windows, and Mac uses dynalib, which is not supported by TCC's build setup. (I'm considering wrapping libtcc into a Perl distribution, since that knows how to create the shared library on Mac as well. But I've not gotten there yet.)

1) Check out the latest mob branch of TCC from git:

git clone git://repo.or.cz/tinycc.git
cd tinycc
git checkout mob

2) Configure, build, install (optionally set the PREFIX)

./configure --disable-static --prefix=$HOME
make
make install

3) Get my latest Perl/TCC bindings:

cd ..
git clone git://github.com/run4flat/perl-TCC.git
cd perl-TCC

4) Slightly modified incantation for build and test (pulls in dependencies)

perl Build.PL
./Build installdeps
./Build
./Build test

That should work (crossing fingers). Let me know either way.

Thanks, Reini!
David

--
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." -- Brian Kernighan

Reply all

Reply to author

Forward

0 new messages