cffi Segmentation fault

[Sean McCully]

Hello,

I have a bit of a perplexing problem when trying to use CFFI to create an array of character arrays and pass to C function call that will iterate over each character array.

The python code looks something like this:

 

          lib = ffi.verify(....)  

          strings = []

for i in lst:

    s = ffi.new('char []', i)

    strings.append(s)

arr = ffi.new("char *[]", strings)

          lib.c_function(arr, len(strings))

 int  c_function(char **array, int len) {

      // do something

     for (int i=0;i<len;i++) {

         do_something(array[i]);  

    }

              return 0; 

}

         int do_something(const char *str) {

               const char *input;

               input = str;

               if (*input != ' ') {  <--- SEGV bad memory address

                    do_something_else(input); 

               }  

         }

The problem when do_something tries to access the character at the address referenced in array, nothing is there it appears to have moved or something. 

[Armin Rigo]

Hi,

On 19 November 2014 01:18, Sean McCully <seanm...@gmail.com> wrote:
> I have a bit of a perplexing problem when trying to use CFFI to create an
> array of character arrays and pass to C function call that will iterate over
> each character array.

I see no problem. I tried to write a runnable example but it works
fine for me. Please modify this example until it crashes:
http://cobra.cs.uni-duesseldorf.de/~buildmaster/paste/9k7Zq.txt


A bientôt,

Armin.

[Sean McCully]

Thanks Armin,

I'll see if I can take your paste and provide a more concise example. But this is the elongated, version

I have this python script, which looks something like this.

import pygit2

repo = pygit2.Repository('/path/to/torvalds/linux/git/repo/linux')

mk_refspecs = lambda r: ["+%s:%s" % (rf, rf) for rf in r]

refspecs = mk_refspecs(repo.listall_references())

remote = list(repo.remotes)[0]

remote.fetch_refspecs = refspecs

So pygit2 is using cffi to make C API Functions calls to libgit2, and when pygit2.remote.Remote.fetch_refspecs.setter is called it takes a Python list converts to a git_strarray and calls a function git_remote_set_fetch_refspecs. 

https://github.com/libgit2/pygit2/blob/master/pygit2/remote.py#L288

Simple enough the problem is, and it appears to be somewhat non-deterministic while a C Function in libgit2 iterates over the git_strarray struct one of the character array pointers no longer points to an accessible memory address. 

https://github.com/libgit2/libgit2/blob/master/src/remote.c#L1842

https://github.com/libgit2/libgit2/blob/master/src/remote.c#L1834

Making the assumption maybe CPython/Interpreter is doing some memory management and has moved the PyObject(s) around. So before calling C Function should cffi memcpy the original PyObject/CData into destination C type.

So I started to look at the cffi generated code, which looks something like  this.

       .....

       .....

returns 0 -->  datasize = _cffi_prepare_pointer_call_argument(

      _cffi_type(52), arg1, (char **)&x1);

  if (datasize != 0) {

    if (datasize < 0)

      return NULL;

    x1 = alloca(datasize);

    memset((void *)x1, 0, datasize);

    if (_cffi_convert_array_from_object((char *)x1, _cffi_type(52), arg1) < 0)

      return NULL;

  }

       ....

       ....

So I am trying to wrap my head around _cffi_prepare_pointer_call_argument, which appears to take Py_ssize_t of the intended c type, and arg1. Which would then allocate new memory for the second argument to C Function. Which has a signature as follows,

int git_remote_set_fetch_refspecs(git_remote *remote, git_strarray *array)

the question I have then, is it intended that _cffi_prepare_pointer_call_argument should not return 0 in this type of instance but instead the size of the intended target and allocate new memory or not?  

Sean

[Armin Rigo]

Hi Sean,

The problem is in strings_to_strarray(). It contains the line

strings = ffi.new('char *[]', len(l))

but the 'strings' object dies at the end of the function. You need to
carefully keep it alive, like you do with all the 's' objects.


A bientôt,

Armin.

[Sean McCully]

Hi Armin,

I noticed that in the docs as well,

https://cffi.readthedocs.org/en/release-0.8/#working-with-pointers-structures-and-arrays

So I had previously rewritten the strings_to_strarray function as follows,

https://github.com/seanmccully/pygit2/blob/master/pygit2/utils.py#L65

Note, the previous messages still apply.

Sean

--
-- python-cffi: To unsubscribe from this group, send email to python-cffi+unsubscribe@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
---
You received this message because you are subscribed to the Google Groups "python-cffi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-cffi+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Armin Rigo]

Hi Sean,

On 21 November 2014 19:26, Sean McCully <seanm...@gmail.com> wrote:
> So I had previously rewritten the strings_to_strarray function as follows,

It's still not correct. The following line:

arr.strings = ffi.new("char *[]", strings)

is always bogus, by itself, if "arr" is a cdata struct. Let me try
again to explain why: this line allocates a "char *[]" whose lifetime
is based on the cdata object that it returns; then "arr.strings = "
will take this new cdata object, get the pointer stored in it, and
copy this pointer into the memory area owned by "arr". Once it is
done, the intermediate cdata object is not needed any more, so it is
deallocated. This causes the "char *[]" array to be deallocated as
well. As a result, after running this line, you have in "arr" a cdata
struct, and in this struct, there is a bogus pointer to freed memory.


A bientôt,

Armin.

[Sean McCully]

Yep, make sense now.

https://github.com/seanmccully/pygit2/blob/master/pygit2/utils.py#L74

Thanks for the help!! It works now