Re: [julia-dev] Passing and array of strings to C

[Jameson Nash]

An int in C is an Int32 in Julia. A formula for calling main can be found slightly further down on that page.

ccall(:foo, Void, (Int32, Ptr{Ptr{Uint8}}), length(argv), argv)

On Nov 18, 2012, at 1:03 AM, lgautier wrote:

Hi,

I seem unable to pass an array of strings successfully, despite trying to replicate what the documentation is telling at

http://docs.julialang.org/en/latest/manual/calling-c-and-fortran-code/#type-correspondences

and I did not find reports of anyone with a similar issue.

The C code is in a file test.c:

#include <stdio.h>

void foo(const int argc, const char **argv) {

  int i;

  const char *string;

  for (i = 0; i < argc; i++) {

    string = argv[i];

    printf("  %i: %s\n", i, string);

  }

}

The Julia code is:

lib = dlopen("test")

argv = ["abc", "def"] # type is Array{ASCIIString,1}

res = ccall(dlsym(lib, :foo), Void,

            (Int, Ptr{Ptr{Uint8}}), length(argv), argv)

No error occurs, but the outcome on the console is:

  0: �;W

  1: �;W

(There char** is not received properly on the C end, although the int is).

The Julia version is the latest release: Version 0.0.0+99218429.r0431, commit 04315b1d1e (2012-10-14 14:19:35).

Any hint or pointer would be greatly appreciated.

Laurent

--
 
 
 

[lgautier]

Same issue with Int32 (I started with that, and moved to Int while trying to guess if 32/64 bit issues).

[Jameson Nash]

Ah yes, I see the real problem now. argv is a pointer to julia strings, not cstrings. We actually want something like:

argv = map((x)->pointer(x.data), ["abc", "def"])

but in that case, the data in argv is no longer protected from garbage collection (although it will happen to work most of the time).

see also process.jl:97 :: Base._jl_pre_exec(args)

--
 
 
 

[lgautier]

Thanks for the help. That's helpful.

Then I have two comments:

- The documentation page should be fixed accordingly.

- The bit about possible dangling pointers hints at rough edges in the design (and to particularly nasty bugs). A way to ensure protection from garbage collection can be:

argv = ["abc", "def"]

argv_p = map((x)->pointer(x.data), argv)

res = ccall(dlsym(lib, :foo), Void,

            (Int, Ptr{Ptr{Uint8}}), length(argv), argv_p)

...but in a sense that leaves the responsibility of taking care of some of the memory management to a Julia programmer. Not ideal. An alternative would be

to let one do this at the C level, but that would make the interfacing with C code require the writing of C code (an interface layer with the C library).

May be other FFIs (for example "ctypes" for Python) have a way to handle this ?

Laurent

[John Cowan]

On Sun, Nov 18, 2012 at 10:00 AM, lgautier <lgau...@gmail.com> wrote:

...but in a sense that leaves the responsibility of taking care of some of the memory management to a Julia programmer. Not ideal. An alternative would be to let one do this at the C level, but that would make the interfacing with C code require the writing of C code (an interface layer with the C library).

This is pretty much a universal in all foreign-function interfaces (FFIs) designed to allow a dynamic language D to interface with a static language S (often C).  One ends up writing ugly, funky, tricky glue code to combine the two.  Depending on the nature of the FFI, the code in question may be written in D (in which case it looks like a literal transcription of S into D) or in S (in which case it looks somewhat more normal, but is usually extremely error-prone, as S's normal type checking probably does no real good).

The good news is that one only has to write such code once per S function.  The bad news is that one has to write such code once per S function.  Glue generators like Swig try to solve this problem, but they end up very complex and brittle, and eventually languages move away from them.

-- 

Being understandable rather than obscurantist poses certain

risks, in that one's opinions are clear and therefore     | John Cowan

falsifiable in the light of new data, but it has the      | co...@ccil.org

advantage of encouraging feedback from others.  --James A. Matisoff

[Jeff Bezanson]

Hmm, I thought I implemented this. I would not have put it in the doc otherwise. I'll take a look.

--
 
 
 

[lgautier]

I just tried with a recent snapshot of the repository -Version 0.0.0+102207829.r30cf, commit 30cf3e5af6 (2012-11-18 03:42:55)-.

The problem is still there.

I believe that the boilerplate code John gave could be wrapped somewhere within `ccall()` (so a Julia programmer does not have to do it).

L.

[Jeff Bezanson]

As I thought, there is actually code inside ccall to handle this case. The problem is that it's subverted by the somewhat-recent change to have ccall automatically call convert() for each argument. One would want to neatly fix this by defining

convert{T}(::Type{Ptr{Ptr{T}}}, a::Array) = pointer([convert(Ptr{T},x) for x in a])

except that doesn't work, since it immediately drops the only reference to the newly-allocated pointer array.

A workaround is to pass convert(Array{Ptr{Uint8}}, argv). I could allow Array{Ptr{Uint8}} as the argument type in ccall, but it wouldn't be obvious that one needs to use that instead of Ptr{Ptr{Uint8}}.

Another approach is to take advantage of prefix "&", which disables the normal call to convert(). With this patch:

--- a/base/base.jl
+++ b/base/base.jl
@@ -8,6 +8,7 @@ convert(T, x)               = convert_default(T, x, convert)
 convert(T::Tuple, x::Tuple) = convert_tuple(T, x, convert)
 
 ptr_arg_convert{T}(::Type{Ptr{T}}, x) = convert(T, x)
+ptr_arg_convert{T}(::Type{Ptr{T}}, x::Array) = x
 
and passing "&argv" instead of "argv", my old ccall machinery takes effect and the call works. The only problem is that it doesn't make sense, since argv by itself is the logical equivalent of the char**.

So this is not a great situation, and any ideas are appreciated. Probably the best of my above options is to use Array{Ptr{Uint8}} as the type of char** arguments; then that would just have to be documented.

--
 
 
 

[Stefan Karpinski]

We could automatically convert Array{T} to Ptr{T}. That avoids people needing to even know about Ptr for most things (there's still occasional need for pointer arithmetic), and ccall could maybe ensure that gc breakage doesn't happen.

--
 
 
 

[Jameson Nash]

This sounds simpler to me. The trouble would arise if I wanted to pass a literal julia array reference to C, since now that would get auto-converted.

--
 
 
 

[Jeff Bezanson]

You can always pass a literal julia reference as "Any".