Haxe FFI Library

[TopHattedCoder]

Hey Haxers,
I've been working on an up-to-date transparent FFI library for Haxe using LibFFI and it works well on my Linux 64-bit laptop with the math library and stuff like that. For reference, here's what the FFI for the math library looks like:

@:lib("m")
class NativeMath extends ffi.lib.EasyLibrary {
    public function cos(x:Float):Float;
    public function tan(x:Float):Float;
    public function sin(x:Float):Float;
    public function acos(x:Float):Float;
    public function atan(x:Float):Float;
    public function asin(x:Float):Float;
    public function sqrt(x:Float):Float;
    public function cbrt(x:Float):Float;
    public function pow(x:Float, y:Float):Float;
    public function round(x:Float):Float;
    public function llround(x:Float):Int64;
}

This works on Java and on Neko for me. Does anyone have any suggestions for how Structs should look, etc?
Currently, they look like this:

@:struct(Person => {
    var name:String;
    var age:UInt16;
})
@:lib("test/test.so")
class Tests extends ffi.lib.EasyLibrary {
    public function make_person(name:String, age:UInt16):Pointer;
}
...
var person:Person = t.make_person("Tom", 234).get(Person.TYPE);
new Person("Bobaffet", 23);

I'm considering adding a ffi.lib.Struct auto-built class since structs are generated into abstracts anyway. Structs aren't supported on Java yet, but nevertheless the rest of the API is supported. Any feedback, builds, suggestions etc would be appreciated.

[Cauê Waneck]

Wow!! I've been meaning to find time to work on a similar project for some time now. I'm really happy to see it working, and with such a nice syntax!

Thanks for the project. I'll try to test it soon :)

2013/9/22 TopHattedCoder <tophatt...@gmail.com>

--
To post to this group haxe...@googlegroups.com
http://groups.google.com/group/haxelang?hl=en
---
You received this message because you are subscribed to the Google Groups "Haxe" group.
For more options, visit https://groups.google.com/groups/opt_out.

[Cauê Waneck]

By the way, in terms of building, building for Windows with MSVC is quite the challenge. I've made a "libffi-mini" version, which includes the build files that work on linux, mac, windows and ios: https://github.com/waneck/hxffi/tree/master/haxelib/project

Cheers!
Cauê

2013/9/22 Cauê Waneck <wan...@gmail.com>

[Juraj Kirchheim]

This looks awesome, congratulations!

Regarding structs, I'm not sure how you want turn an abstract
definition into a class definition through a build macro. Also having
the structs and functions attached to one single definition is
probably useful. Splitting it could lead to weird edge cases with
unexpected macro execution order.

I think the current syntax does the job pretty well, although it is a
little cheeky to use an EBlock as a way to denote types ;)
There's one limitation that comes to my mind: nested structs. With the
current approach, you'd have to parse the struct from two different
syntactical structures. Any of the two struct "clauses" might do the
trick:

@:struct(var Person:{ name:String, age: UInt16 })
class Tests extends ffi.lib.EasyLibrary {
@:struct var Person:{ name:String, age: UInt16 };
}

Also, I'm wondering what Pointer.getValue does. Could it be Pointer<T>
and have Pointer.getValue be Void->T?

Regards,
Juraj

[TopHattedCoder]

@back2dos
I got curious into how HxSL did it. It's hacky, but pretty.
Also, that's what 'get' does, but 'getValue' is meant to get the pointer as a Neko value. I'll rename it to getAsNekoValue to make it clearer. It's only really there because it can be and it will be useful for the future.

@caea
Thanks, I'll look into it.

[Cauê Waneck]

2013/9/22 TopHattedCoder <tophatt...@gmail.com>

@back2dos
I got curious into how HxSL did it. It's hacky, but pretty.
Also, that's what 'get' does, but 'getValue' is meant to get the pointer as a Neko value. I'll rename it to getAsNekoValue to make it clearer. It's only really there because it can be and it will be useful for the future.

I also think that it's pretty. :)

One thing that would be nice is to support haxe.io.BytesData conversion as well, which would be necessary for e.g. OpenGL buffers. Also do you intend to support closures / callbacks as well?

I might add the raw libffi lib interface for haxe --interp as well, so this lib (for example) can make use of libffi on haxe interp mode as well. This should make us able to use --interp without being restricted to the current native API

All in all, congrats. It's really a very useful project. When the api is more stable I can try to add support for C# as well!

Cheers

Cauê

[Cauê Waneck]

One more thing: It would be useful to make some haxe CFFI-only functions available through haxe code (e.g. for hxcpp, the GC_ENTER_BLOCKING and alikes).

2013/9/23 Cauê Waneck <wan...@gmail.com>

[TopHattedCoder]

How goes Pointer.getBytes(?length:Int):haxe.io.BytesData sound? Length is nullable so that null-terminated things can work too.

I'll try to get around to closure support as soon as I understand the use of them enough to write a clean wrapper, and I'll try and make a ffi.lib.HaxeLibrary that provides some Haxe CFFI-only functions, like the one you mentioned, then add a @:haxe metadata to the EasyLibrary. I'll probably clean up EasyLibrary too eventually - it is currently quite unorganised.

And a Haxe interp FFI interface would be awesome!

[TopHattedCoder]

Is there any documentation on the underlying Haxe CFFI interface?

On Sunday, September 22, 2013 6:15:43 PM UTC+1, TopHattedCoder wrote:

[Cauê Waneck]

Here:

http://haxe.org/doc/cpp/ffi

2013/9/23 TopHattedCoder <tophatt...@gmail.com>

--

[Marcelo Serpa]

All this FFI stuff sounds really interesting! However the information is spread around, and I can't quite make sense of what's different (or great) about this library, whether it supports C++ [the target], etc, and most importantly, the practical use cases for it.

For example -- @Cauê -- could this be used to make to port the native webview for Croxit desktop across Windows/Mac/Linux?

Cheers,

-- 

Marcelo

[TopHattedCoder]

This library is different because it allows you to run native code from shared libraries at runtime - there's no middle man. This is especially great for people wanting to develop interpreters, VMs etc on Haxe that require a way of loading a library on-the-fly. It should support C++, but I can't confirm this because I'm missing a library or something needed by HxCPP

It's still very premature and not stable for usage just yet, but Java and NodeJS bindings are in the works.

Practical use cases (when it becomes stable):

• Python interpreter with ctypes.
• Neko interpreter with loader.
  
• SDL bindings that work on most sys platforms.

[Cauê Waneck]

Hey!!

2013/9/23 Marcelo Serpa <fullofc...@gmail.com>

For example -- @Cauê -- could this be used to make to port the native webview for Croxit desktop across Windows/Mac/Linux?

Yes - This is actually probably what I'm going to use to accomplish that btw ;)

It'll greatly simplify the need the development of native extensions, as they will all be able to be developed inside the Haxe ecosystem !

Cheers!
Cauê

[TopHattedCoder]

If anyone wants to contribute to the repo, just say and I'll add you as a contributor.

On Sunday, September 22, 2013 6:15:43 PM UTC+1, TopHattedCoder wrote:

[Vujadin Krtolica]

Would it be possible to use ImageMagick^®  or GraphicsMagick ( http://www.graphicsmagick.org/ ) with this lib, in an OpenFL based Windows app ?

[David Peek]

You're in luck:

https://code.google.com/p/haxe-imagemagick/

I even got it working on OS X the other day. I'm sure you could rewrite with this library though :)

[Heinz Hölzer]

It would be nice if we had a nice syntax for haxe.macro.MacroType because this is a perfect use case for it.

But at the moment the Syntax would be very ugly (imho the reason why MacroType is not really used).

typedef Person = haxe.macro.MacroType<[ffi.lib.EasyStruct.build({

var name : String;

var age : UInt16;

})]>;

maybe something like this would be great:

// definition of a MacroType

macro typedef Struct<Args> = haxe.macro.MacroType<[ffi.lib.EasyStruct.build(Args)]>; // Where each Type parameter could be a Type or an Expression

// usage:

import Struct;

typedef Person = Struct<{

var name : String;

var age : UInt16;

}>;

but i don't think that something like this would be easy to implement ;)

[Heinz Hölzer]

or alternative syntax, maybe easier to implement:

// define your build function (must be uppercase (should make parsing easier),

class Build {

    macro public static function Struct(fields:Expr):Type {...}

}

// usage (like a function but arguments are surrounded by <> instead of () )

import Build.Struct;

typedef MyStruct = Struct<{

   var name : String;

   var age : UInt16;

}>;

just some random thoughts ;)

best,

h

[TopHattedCoder]

I'd try implementing the Struct<{name: String, age:UInt16}> syntax, but it might not work in typedefs because of the unknown int types... how about extra syntax for structs with unnamed fields like this:

Struct<String, UInt16>

then for structs with named fields there's the older way and this way:

@:structs([
    CString => String,
    Person => {
        var name:CString;
        var age:UInt16;
    },
    Player => {
        var nick:CString;
        var score:UInt;
    }
])

On Sunday, September 22, 2013 6:15:43 PM UTC+1, TopHattedCoder wrote:

[dlots]

How does waneck/hxffi/libffi.cpp relate to thc/hx-ffi/common/ExternalInterface.cpp. They accomplish the same thing, exposing libffi to haxe. Have you guys agreed upon which version to standardize on?

[dlots]

Please change getAsNekoValue() back to getValue. getAsNekoValue makes no sense in the context of a generic Pointer and it is quite obvious what the point of getValue is. getValue is a critical function.

[dlots]

Any reason why array of Array of Type isn't supported? It would appear a memcpy routine similar to the struct to_pointer code would facilitate this.

On Sunday, September 22, 2013 1:15:43 PM UTC-4, TopHattedCoder wrote:

[dlots]

I am reliant on this library and this feature is broken. If anyone has any idea on what is causing this, please, please, please specify. Relevent ExternalInterface.ccp portion:

case FFI_TYPE_STRUCT: {
	const unsigned int size = t -> size;
	uintptr_t v = (uintptr_t) malloc(size);
	ffi_type** elem = t -> elements;
	unsigned int i = 0;
	while(*elem != NULL) {
		const ffi_type* curr = *elem;
		const size_t size = curr -> size;
		uintptr_t nptr = (uintptr_t) to_pointer(val_array_i(val, i), *elem);
		memcpy((void*) v, (void*) nptr, size);
		v += size;
		elem++;
		i++;
	}
	return (void*) v;
}

As you can imagine all of the segfault have been quite vexxing. 

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct {
    int i;
} St_n;
char* test(St_n st) {
    char res[1024];
    sprintf(res,"st %d",st.i);
    return res;
}

package;
import ffi.lib.EasyLibrary;
@:struct(St_n=>{
    var i:Int;
})
@:lib("libntv.so")
class Ntv extends EasyLibrary {
    public static function main():Void {
        var lib:Ntv=new Ntv();
        trace(lib.test(new St_n(100)));
    }
    public function test(st:St_n):String;
}

Results in:
Ntv.hx:10: st 32735

On Sunday, September 22, 2013 1:15:43 PM UTC-4, TopHattedCoder wrote:

[dlots]

Ugh, and people question why development takes time:

The pointer is not reset to the malloc location:

uintptr_t v_start=v;
...as now
return (void*) v_start;

Ps, in addition, a char*/haxe.std.String in a struct is at this time causing a segmentation fault.

[dlots]

https://github.com/TopHattedCoder/hx-ffi/issues/5

The memcpy loop causes segfaults because struct memory allocation/padding does not necessarily conform the actual sizeof members. It's essentially a requirement to pad the struct correctly:

http://stackoverflow.com/a/5397638/832565