C Code works, Python CFFI code does not

[Israel Brewster]

I have a C library that I have compiled that provides functions for connecting to my 4D database (http://freecode.com/projects/lib4d_sql). I initially compiled and used it via python/cffi on my Mac OS X system, and everything worked as desired. I then moved over to a CentOS 6.5 box. I set up the box myself, however I don't recall explicitly installing cffi on it, so I don't know how it got installed, or if it is just part of the base package. In any event, I was able to compile the library again without difficulty, and run the following C code to test it successfully:

#include "fourd.h"
#include <stdio.h>

int main(int argc, const char * argv[]) {
        char *host="<host>";
        char *user="<username>";
        char *pass="<password>";
        char *base="<base>";


        FOURD*connptr=fourd_init();
        int result=fourd_connect(connptr,host,user,pass,base,19812);
        printf("The result is: %i\n",result);


        return 0;
}

The output of that code looks like the following:

[israel@follow lib4d_sql]$ ./a.out
About to log in with user: <username> password: <password>
Running login function
Login function ran with result: 0
The result is: 0

which is exactly what I want. I then tried to run this from python using cffi and the following python code (which is the exact code that worked fine from my mac):

from cffi import FFI
from dateutil import parser
from datetime import datetime, timedelta

ffi = FFI()
ffi.cdef("""<bunch of stuff from the fourd.h header here>""")
C = ffi.dlopen("./lib4d_sql.so")

connptr = C.fourd_init()

host = ffi.new("char[]", "<host>")
user = ffi.new("char[]", "<user>")
password = ffi.new("char[]", "<password>")
base =  ffi.new("char[]", "<base>")

result = C.fourd_connect(connptr, host, user, password, base, 19812)

if result != 0:
    error=C.fourd_error(connptr)
    print ffi.string(error)
    print "Unable to connect! error:", result
    exit(0)

Which gives the output:

[israel@follow lib4d_sql]$ python python4d.py
About to log in with user:<username> password:<password>
Login function ran with result: -1
Error during login
Unable to connect! error: 1

Indicating that it did NOT work as expected. Notice the lack of the "running login function" line, which is a printf that I added to the C code for debugging purposes, as well as the -1 return value from the login function that didn't run, rather than the 0 (actually, the login function returns either 0 or 1, so I don't even know how it could give a -1 there). What might be different about my CentOS install such that the C code works but the python code does not, while the same python code does work under Mac?

[Armin Rigo]

Hi Israel,

On 18 November 2014 21:48, Israel Brewster <macav...@gmail.com> wrote:
> What might be different about my CentOS install such that the C code works but
> the python code does not, while the same python code does work under Mac?

You're using dlopen() instead of verify(), so you need to check that
all declarations in the cdef() are very exactly the same ones as the
ones used to compile the original C code of lib4d_sdl.so. It is
possible to get strange errors when porting to another architecture if
the cdef() constains "compatible enough" declarations on one
architecture but not on another.

No clue if it's the source of the problem here. I need to reproduce
the problem to tell more.


A bientôt,

Armin.

[Israel Brewster]

So the easiest solution may be to use verify()? I tried that initially, but didn’t get it to work. I’ll take another stab at it.

I did copy all the lines in the cdef straight from the header file, although I had to do some tweaking since #defines didn’t work. That could be the whole problem though.
————
Israel Brewster

> --
> -- python-cffi: To unsubscribe from this group, send email to python-cffi...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/python-cffi?hl=en
> ---
> You received this message because you are subscribed to a topic in the Google Groups "python-cffi" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/python-cffi/GJgl_8z18aQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to python-cffi...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Israel Brewster]

Ok, so I did manage to get verify working. So my python code now looks like this:

from cffi import FFI
from dateutil import parser
from datetime import datetime, timedelta

ffi = FFI()
ffi.cdef("""<bunch of stuff from the fourd.h header here>""")

C=ffi.verify("""
#include "/home/israel/lib4d_sql/fourd.h"


#define BUFFER_LENGTH 131072
#define ERROR_STRING_LENGTH 2048


#define MAX_HEADER_SIZE 2048
#define DEFAULT_IMAGE_TYPE "jpg"
#define MAX_LENGTH_COLUMN_NAME 255


#define FOURD_OK 0
#define FOURD_ERROR 1


""",libraries=["4d_sql"],library_dirs=["/home/israel/lib4d_sql"])

connptr = C.fourd_init()

host = ffi.new("char[]", "<host>")
user = ffi.new("char[]", "<user>")
password = ffi.new("char[]", "<password>")
base =  ffi.new("char[]", "<base>")

result = C.fourd_connect(connptr, host, user, password, base, 19812)

if result != 0:
    error=C.fourd_error(connptr)
    print ffi.string(error)
    print "Unable to connect! error:", result
    exit(0)

Unfortunately, the end result is identical - the printf that I put at the beginning of the c login function never runs, and I get the -1 return code from a function that should never return anything other than 0 or 1. BTW, It seems a bit odd to me that I have to both #include the header as well as replicating it in the cdef block, but whatever - it works on the Mac, at least.

[Israel Brewster]

A Little more information:

The C function I am calling is (obviously) fourd_connect. This function itself (apparently) runs fine, for the most part. It first checks to make sure the connection object is properly initialized and the user not already connected to the 4D server, then establishes a socket connection to the 4D server. This much appears to work properly, in that it doesn't return any errors. It then calls the login() function, which is where things get weird. As mentioned, from straight C code, or from a Mac running python/CFFI this works, however from my CentOS box the login function never gets called, and the variable that it should be setting gets set to -1. That is, the C code looks something like this (roughly, this isn't an exact representation):

int fourd_connect(<args>){
        //bunch of code that runs as expected
        printf("About to log in with user:%s password:%s",user,password);
        int loginsuccess=login(cnx,1,user,password,cnx->preferred_image_types);
        printf("Login function ran with result: %i",loginsuccess);
        //other code that responds to the non-zero loginsuccess as expected
 }

//login is defined in a separate c file, with a header included here.
int login(FOURD *cnx,unsigned short int id_cnx,const char *user,const char*pwd,const char*image_type)
{
    printf("Entering login function\n"); //this is never printed when run through python CFFI
    //other code to actually login, which never runs
    if(<check for error on login>)
        return 1;
    return 0;
}

Note a couple of things here:

- The first line in the login function is a printf. This line never gets printed when calling from python CFFI on CentOS, which seems to indicate that the function never even executes the first line

- The only two possible return values from this function are 1 for error or 0 for success: yet the variable I set to the return of this function gets a value of -1.

Any ideas here would be greatly appreciated. Are there any steps I can take to troubleshoot this? If it was pure C code I could trace through the execution and get an idea of what is happening, but in pure C it works as expected, so that doesn't help.

[Israel Brewster]

Ok, solution, sort of: if I rename the inner C function from login to anything else (I used dblogin), then it works properly under CentOS as well as Mac. Which is totally weird, especially given that login is never called directly - it is strictly used within the C library. So while I would still like to know why this is happening, at least I have a path to move forward with my development.

[Armin Rigo]

Hi Israel,

On 19 November 2014 22:16, Israel Brewster <macav...@gmail.com> wrote:
> Ok, solution, sort of: if I rename the inner C function from login to
> anything else (I used dblogin), then it works properly under CentOS as well
> as Mac.

Ah, I see. I know what is going on then: login() is a function that
exists in the C standard library (actually in "-lutil", see "man 3
login"). When you compile a small standalone C program, it doesn't
matter. But when you compile for CFFI, with "-lpython2.7" or
whatever, it includes a lot of other libraries and the function name
"login" is overridden...

I'm vaguely confused about why you don't get any error or warning
about it, but well, the linking logic on Linux is rather complicated
in the bad sense (imho). A quick fix is either (as you did) to rename
the function, or (if it is only used inside one C file) to set it
"static".


A bientôt,

Armin.

[Israel Brewster]

That makes sense, and certainly explains the behavior I was seeing. I would imagine that the lack of warnings was due to not linking in the problematic libraries at compile time, but rather at run time. Essentially, the compiler/ verify() didn't know there was a conflict, since none of the relevant header files showed the conflict. It was only when the additional library was linked in at run time that the wrong function got executed.

In any case, hopefully this bug will be fixed quickly. For me, yes, I can simply rename the conflicting function. I would imagine, however, that there are dozens of libraries that would have a "login" or "logout" function in which renaming said function is either problematic (many uses, complicated code or whatever) or simply not an option, such as in the case of any library that is not open source. Maybe not though - maybe I am the only one who has ever wanted to use a C function named "login" or "logout" :-)

[Armin Rigo]

Hi Israel,

On 20 November 2014 18:08, Israel Brewster <macav...@gmail.com> wrote:
> In any case, hopefully this bug will be fixed quickly. For me, yes, I can
> simply rename the conflicting function. I would imagine, however, that there
> are dozens of libraries that would have a "login" or "logout" function

In any case, name conflicts do occur. A program cannot run with two
libraries that have non-static, exported functions called "login",
unless it does crazy tricks with dlopen(). The name "login" is bad,
both because it's a common name and because, sadly, it's also the name
of a function in the relatively standard library "util". This library
also contains openpty() --- which means that CPython and PyPy link to
it.


A bientôt,

Armin.