node-ffi with a function that returns a pointer and a length

[Ryan Schmidt]

This is my first day of trying to use node-ffi, and I'm going mad trying to get it to work with a function that has this signature:

int makeData(char **data, unsigned int *length);

This function will return a pointer to some data and an unsigned int saying how many bytes long the data is. The data might be text or binary.

How do I define this function in node-ffi? What kind of variables do I need to declare and pass to it? And how do I get the data out in the end? I have spent hours on this already, finding very little information on Google and only a single example in the node-ffi repository (which doesn't cover this use case), and sparse documentation.

What I have at the moment is:

var mylib = ffi.Library('/path/to/mylib', {
makeData: ['int', ['char **', 'uint *']],
});

var lengthpointer = ref.alloc('uint *');
var datapointer = ref.alloc('char *');
mylib.makeData(datapointer, lengthpointer);
var length = lengthpointer.deref();
var data = ref.reinterpret(datapointer.deref(), length);

The problem is that length is always NaN. If I set length to some integer instead, then the reinterpreting of the buffer works and I get that amount of data.

A surprise is that if I alloc() datapointer before lengthpointer instead of after, deref()ing lengthpointer causes a segmentation fault. I really don't understand why the order in which I declare some variables would have this effect.

If I declare datapointer as alloc('char **') (to match the function's signature) instead of 'char *' I also get a segmentation fault when deref()ing lengthpointer.

This is on OS X 10.8.4 with node 0.10.12 and node-ffi 1.2.5.

[Nathan Rajlich]

I think you're close here... So one of the slightly confusing points is that Buffer instances already have 1 level of indirection by nature (they're a pointer to some process memory), so when you do:

var lengthpointer = ref.alloc('uint *');

You're really creating a Buffer that "can store a 'uint *'", so it itself is a "uint **" Buffer. Does that make sense? You're probably getting NaN for the length since when you call .deref() you're actually dereffing to a "uint *" Buffer, rather than a "uint" Number value.

So on that note, try something like this (there's basically just 1 level of indirection removed from the variables):

  var mylib = ffi.Library('/path/to/mylib', {
    makeData: ['int', ['char **', 'uint *']],
  });

  var lengthpointer = ref.alloc('uint');

  var datapointer = ref.alloc('char *');
  mylib.makeData(datapointer, lengthpointer);
  var length = lengthpointer.deref();
  var data = ref.reinterpret(datapointer.deref(), length);

Warning, untested! But let me know if that works out for you. FWIW I think you want a `datapointer.readCString()` call somewhere in there as well (instead of ref.reinterpret() perhaps?).

Finally, this code might be able to help you out since it seems similar: https://github.com/TooTallNate/NodObjC/blob/78197b330f1c8ca42f9185c298f346f4314e5099/lib/core.js#L150-L160

Cheers!

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Ryan Schmidt]

On Jun 19, 2013, at 14:38, Nathan Rajlich wrote:

> I think you're close here... So one of the slightly confusing points is that Buffer instances already have 1 level of indirection by nature (they're a pointer to some process memory), so when you do:
>
> var lengthpointer = ref.alloc('uint *');
>
> You're really creating a Buffer that "can store a 'uint *'", so it itself is a "uint **" Buffer. Does that make sense? You're probably getting NaN for the length since when you call .deref() you're actually dereffing to a "uint *" Buffer, rather than a "uint" Number value.
>
> So on that note, try something like this (there's basically just 1 level of indirection removed from the variables):
>
> var mylib = ffi.Library('/path/to/mylib', {
> makeData: ['int', ['char **', 'uint *']],
> });
>
> var lengthpointer = ref.alloc('uint');
> var datapointer = ref.alloc('char *');
> mylib.makeData(datapointer, lengthpointer);
> var length = lengthpointer.deref();
> var data = ref.reinterpret(datapointer.deref(), length);
>
> Warning, untested! But let me know if that works out for you.

ref.alloc('uint') instead of ref.alloc('uint *') works. That's even what the Output Parameters section of the Tutorial wiki page says. I swear to you I spent hours yesterday trying every possible permutation without success. But thank you, that's clearly what I needed to do.


I think I got confused by the lone example, which seems to get it wrong:

https://github.com/rbranson/node-ffi/blob/master/example/sqlite.js

It declares the function as:

'sqlite3_open': [ 'int', [ 'string', sqlite3PtrPtr ] ],

Then allocates:

var db = ref.alloc(sqlite3PtrPtr)

Then calls:

SQLite3.sqlite3_open(dbName, db)

By what you wrote, shouldn't it instead allocate:

var db = ref.alloc(sqlite3Ptr)


A lot of my confusion was because:

* the Tutorial uses types like 'pointer'
* the only other wiki page, API Changes from v0.x to v1.x, says you should never use 'pointer' anymore and should always declare your own custom pointer types
* installing 'node-ffi' with npm gets you version 0.x that doesn't support custom pointer types; you have to install 'ffi' to get version 1.x

> FWIW I think you want a `datapointer.readCString()` call somewhere in there as well

I had been using data.toString() which worked. data.readCString() ended early with binary data. datapointer.readCString() didn't return anything.

> (instead of ref.reinterpret() perhaps?).

Not reinterpreting gave me only the first 8 bytes of data. I don't see how node could possibly know how much data is in the memory location being pointed to, unless I tell it by using reinterpret.

> Finally, this code might be able to help you out since it seems similar: https://github.com/TooTallNate/NodObjC/blob/78197b330f1c8ca42f9185c298f346f4314e5099/lib/core.js#L150-L160

Thanks, I'm not sure I understand it though. But I think I'm all set for now.

Now I just need to make a test to see what the speed difference is between accessing the library via ffi and just running its command line program with spawn. My hope is that the pain of dealing with ffi yields a nice speed boost.

[Nathan Rajlich]

This is very valuable feedback Ryan, thank you!

And ya, ignore most of my comments except for the beginning part :P

I was assuming that `char ** data` argument somehow got passed an array of C string pointers, but that wouldn't really make any sense actually. Now I realize that your function is outputting only a single C string (which appears to *not* be NUL terminated since you also are given a `length` output parameter, which is also why .readCString() wouldn't work in your case).

Anyways, glad you got it figured out. Cheers!

[Ryan Schmidt]

On Jun 19, 2013, at 17:30, Ryan Schmidt wrote:

> Now I just need to make a test to see what the speed difference is between accessing the library via ffi and just running its command line program with spawn. My hope is that the pain of dealing with ffi yields a nice speed boost.

It does. In my test, running the function 1000 times with node-ffi is 6 times faster than running the command line program 1000 times to do the same thing.