create shared library

[Eric Frederich]

Is it possible to create a shared library using Cython?

I know that you can create functions with "cdef api" but that just makes those functions available through a .h file.

Those functions aren't directly available (like a normal C function would be) from within the shared library you create out of the generated .c file... you have to go through the .h file.

[Stefan Behnel]

Eric Frederich, 16.01.2014 19:33:

It's not clear to me what you mean by not having to go through a .h file.
That's how interfaces work in C.

Could you explain your use case? What so you want to achieve? How do you
want to call the function?

Stefan

[Eric Frederich]

Stefan,

Sorry for not explaining clearly.

I was trying to see if you can create a "normal" .so file where the function symbols are exported in the .so file itself not called through some macro defined in a .h file.

... so that you could call it through something like libffi, or ctypes just by having the .so and knowing the signatures of the functions.

As it is now, to use the resulting .so file from C you have to run Py_Initialize(), and call import_modulename() while making sure to set PYTHONPATH to the location of the .so file.

Stefan

--

---
You received this message because you are subscribed to the Google Groups "cython-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Sturla Molden]

> Den 19. jan. 2014 kl. 02:20 skrev Eric Frederich <eric.fr...@gmail.com>:
>
> Stefan,
>
> Sorry for not explaining clearly.
> I was trying to see if you can create a "normal" .so file where the function symbols are exported in the .so file itself not called through some macro defined in a .h file.
> ... so that you could call it through something like libffi, or ctypes just by having the .so and knowing the signatures of the functions.

That is how C works. To use ctypes you need to use the information in the header file.

Cython does not generste an .so file, it generates C. The .so file is generated by the C compiler, so obviously it is a normal .so file you can call with ctypes.

> As it is now, to use the resulting .so file from C you have to run Py_Initialize(), and call import_modulename() while making sure to set PYTHONPATH to the location of the .so file.
>

If you use the Python C API you must do this.

If you just want plain C, you know where to find it.

But if you are using ctypes, I am quite sure Py_initialize has already been called for you...



Sturla

[Eric Frederich]

If I have a function in foo.pyx file.

cdef api int add_two_numbers(int x, int y):
    return x + y

I get a foo.c and a foo_api.h
I compile the foo.c file into a .so file.

I cannot use this from the Go programming language.  For example here is a test.go

package main

import "fmt"

// #cgo CFLAGS: -I. -I/usr/include/python2.7
// #cgo LDFLAGS: -lpython2.7
// #include <foo_api.h>
import "C"

func main() {
    fmt.Println("Hello, 世界")
    C.Py_Initialize()
    C.import_foo()
    tmp := C.add_two_numbers(1, 2)
}

When I try to compile it I get the error:
./test.go:14:12: call of non-function C.add_two_numbers

For some reason Go doesn't consider add_two_numbers a full fledged C function.

Either because it cannot find it defined within the .so file, or because of the magic in the api.h file.

static int (*__pyx_f_3foo_add_two_numbers)(int, int) = 0;
#define add_two_numbers __pyx_f_3foo_add_two_numbers

.... it seems to be a preprocessor macro, not a real function.

[Stefan Behnel]

Eric Frederich, 20.01.2014 03:50:

> If I have a function in foo.pyx file.
>
> cdef api int add_two_numbers(int x, int y):
> return x + y
>
> I get a foo.c and a foo_api.h
> I compile the foo.c file into a .so file.
>
> I cannot use this from the Go programming language. For example here is a
> test.go
>
> package main
>
> import "fmt"
>
> // #cgo CFLAGS: -I. -I/usr/include/python2.7
> // #cgo LDFLAGS: -lpython2.7
> // #include <foo_api.h>
> import "C"
>
> func main() {
> fmt.Println("Hello, 世界")
> C.Py_Initialize()
> C.import_foo()
> tmp := C.add_two_numbers(1, 2)
> }
>
>
> When I try to compile it I get the error:
> ./test.go:14:12: call of non-function C.add_two_numbers
>
> For some reason Go doesn't consider add_two_numbers a full fledged C
> function.

You need to declare it "public" for that, i.e.

cdef public int add_two_numbers(int x, int y):
...

Essentially, it's the opposite of C's "static" declaration, which is the
default for non-public C functions.

The "api" declaration generates a C-API that can be used by other Cython
(or C) code due to the 'magic' in the header file. It's not meant to be
used through an FFI (which Go might be doing here).

Stefan

[Eric Frederich]

Stefan,

cdef public worked like a charm and is exactly what I wanted.

Sorry it took a while to explain what I wanted.

~Eric

[Eric Frederich]

Okay... the "cdef public" seems to work with very simple functions like adding two numbers.
But... when I try to print within the function or return len(os.listdir('/tmp')) I get a segfault.

Below are foo.pyx, test.c, the Makefile, and the output of running "make run"

=== foo.pyx ===
cdef public int add_two_numbers(int a, int b):
    print 'hello'
    return a + b

=== test.c ===
#include <Python.h>
#include "foo.h"
int main(int argc, char* argv[]){
    printf("HERE %d\n", __LINE__); fflush(stdout);
    int x = add_two_numbers(5, 6);
    printf("HERE %d\n", __LINE__); fflush(stdout);
    printf("Got %d\n", x);
    printf("HERE %d\n", __LINE__); fflush(stdout);
    return 0;
}

=== Makefile ===
all: foo.so test
foo.h foo.c: foo.pyx
    cython foo.pyx
foo.so: foo.c
    gcc -I/usr/include/python2.7 -Wall -Werror -fPIC -shared -o libfoo.so foo.c
test: foo.so
    gcc -I/usr/include/python2.7 -Wall -Werror -o test test.c -L. -lfoo -lpython2.7
run: test
    LD_LIBRARY_PATH="." ./test
clean:
    rm -f foo.c foo.h libfoo.so test

=== make run output ===
$ make run
cython foo.pyx
gcc -I/usr/include/python2.7 -Wall -Werror -fPIC -shared -o libfoo.so foo.c
gcc -I/usr/include/python2.7 -Wall -Werror -o test test.c -L. -lfoo -lpython2.7
LD_LIBRARY_PATH="." ./test
HERE 4
Segmentation fault
make: *** [run] Error 139

[Eric Moore]

Why do you expect that to work? Cython creates _python_ extensions not fully independent shared libraries.  Your simple add two numbers example works because it results in a function that doesn't depend on calling out to python.  Look at the generated C. Calling print or using a module function depends on your compiled extension being loaded into a python interpreter which you aren't doing in your example. 

Using Cython to write a shared library and not a python module is going to cause nothing but pain even if it could work in certain circumstances if you are careful. 

Eric 

[Eric Frederich]

Maybe I need to look at cffi.

I know that cffi can create real valid C functions, and it can even do so at runtime dynamically (to use as C callbacks).

To answer your question...
We're looking at re-writing the core of an application in a high level language but still want to have bindings available for Python, Perl, TCL, Java, etc.
That is the reason I want to have a generic shared library.

I was trying to see if I could implement it in Cython under the hood

[Chris Barker - NOAA Federal]

On Jan 22, 2014, at 6:56 AM, Eric Frederich <eric.fr...@gmail.com> wrote:

> Maybe I need to look at cffi.
> I know that cffi can create real valid C functions, and it can even do so at runtime dynamically (to use as C callbacks

I'm not familiar with cffi, but I doubt it. It is designed to CALL
c--not generate it. It does look like it generates c interface code,
but that's a pretty limited use case.

> To answer your question...
> We're looking at re-writing the core of an application in a high level language but still want to have bindings available for Python, Perl, TCL, Java, etc.
> That is the reason I want to have a generic shared library.
> I was trying to see if I could implement it in Cython under the hood

Well, if you want to write your core in Python, then Cython may be a
good way to make it callable from c ( and thus other languages ). But
you'll need to have a python interpreter running and properly
initialized, even when you are using it from some other language.

You may similarly be able to use cffi in that way. But it will
certainly require the python interpreter at run time as well. And
cython allows you to have some stuff compiled to pure c (like your
simple add function) which I don't think cffi would give you.

if you restrict yourself to that subset of cython, you may be able to
run without an interpreter, but then you are not using a dynamic
language anymore, but essentially c with a different syntax. Not much
point in that.

And it would take some care to make sure you never call into the
python interpreter.

( though I do like the cython syntax better, so there may be some
point to a "pure C" version of cython....)

-Chris

[Robert Bradshaw]

You need to call Py_initialize and import_modulename(). At that point
you can safely call into the Python interpreter and do stuff like
print len(os.listdir('/tmp'))). It should be noted that Cython
provides a C API not ABI.

- Robert

[Eric Frederich]

Okay guys... thanks for the help so far.

I was wondering whether to create a new thread or continue on this one... for now I'll just continue here.

I am still trying to create an independent shared library which uses Python underneath but has a plain C API.

I think I have succeeded in that my library works from a C main program and a Go main program but I cannot get it to work from a Python main program through cffi.

This is what I did for my example.

I wrote a core in pure python with core.py

I created the C interfaces to core.py in cython with ccore.pyx

I wrote a shared library libcore.so in C (core.c, core.h) which calls Py_Initialize() and import_ccore().

My example C and Go programs work, but not my Python program.

I put a non-python function in my libcore.so which works from Python via cffi, its the functions which call into Python that don't work.

I put some print statements in the generated ccore_api.h file and found the place it fails is within __Pyx_ImportModule.

Specifically during PyImport_Import(py_name) where the name is ccore.

Its interesting that I was successful in creating a shared library that hides the fact that it uses Python under the hood and it works from C and Go but not from Python.

Any ideas what could be going on here?

=== core.py ===

class Session(object):

    def __init__(self, name):

        print 'creating session', name

        self.name = name

main_session = Session('main')

=== core.pyx ===

import core

cdef api _get_main_session():

    return core.main_session

cdef api _new_session(char* name):

    return core.Session(name)

cdef api int _is_main_session(s):

    return s == core.main_session

=== core.h ===

#ifndef __CORE__H

#define __CORE__H

void* get_main_session();

void* new_session(char* name);

int is_main_session(void*);

int non_python_function(int a, int b);

#endif

=== core.c ===

#include <Python.h>

#include "ccore_api.h"

void init_python_stuff(){

    Py_Initialize();

    int tmp = import_ccore();

    if(tmp){

        printf("ERROR IMPORTING CCORE\n");

        exit(1);

    }

}

void* get_main_session(){

    init_python_stuff();

    return _get_main_session();

}

void* new_session(char* name){

    init_python_stuff();

    return _new_session(name);

}

int is_main_session(void* s){

    init_python_stuff();

    return _is_main_session(s);

}

int non_python_function(int a, int b){

    return a + b;

}

=== test.c ===

#include "core.h"

#include <stdio.h>

int main(int argc, char* argv[]){

    void* session1 = get_main_session();

    void* session2 = new_session("foo");

    int answer;

    answer = is_main_session(session1);

    printf("Session 1 is main? %d\n", answer);

    answer = is_main_session(session2);

    printf("Session 2 is main? %d\n", answer);

    return 0;

}

=== test_go.go ===

package main

import "fmt"

// #cgo LDFLAGS: -L. -lcore

// #include "core.h"

import "C"

func main() {

    session1 := C.get_main_session()

    session2 := C.new_session(C.CString("BLAH"))

    

    verdict := C.is_main_session(session1);

    fmt.Println("Go session1 is main?:", verdict);

    verdict = C.is_main_session(session2);

    fmt.Println("Go session2 is main?:", verdict);

}

=== test.py ===

#!/usr/bin/env python

from cffi import FFI

ffi = FFI()

ffi.cdef('''\

void* get_main_session();

int non_python_function(int a, int b);

''')

C = ffi.dlopen('libcore.so')

print 'hi 1'

print C.non_python_function(4, 5)

print 'hi 2'

C.get_main_session()

print 'hi 3'

[Chris Barker]

On Fri, Jan 24, 2014 at 10:58 AM, Eric Frederich <eric.fr...@gmail.com> wrote:

 

I think I have succeeded in that my library works from a C main program and a Go main program but I cannot get it to work from a Python main program through cffi.

 

I wrote a core in pure python with core.py

I created the C interfaces to core.py in cython with ccore.pyx

I wrote a shared library libcore.so in C (core.c, core.h) which calls Py_Initialize() and import_ccore().

My example C and Go programs work, but not my Python program.

I haven't used cffi, but presumably it calls into a shared lib with a C interface frmo a running pyton process.

So in this case, the main python process that is using cffi has initialized python already. Then you load a shared lib that calls Py_Initialize(). I've never hacked at this level, but I suspect that you can't call Py_Initialize() twice in the same process, nor have two different instances of the interpreter running in the same process at once.

So you'll either need a different libcore for pyton use, or have some way to check if pyton is already running in libcore. But why do you need libcore at all when running from Python -- couldn't you simply call into ccore directly in that case?

-Chris

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris....@noaa.gov

[Eric Frederich]

You can call Py_Initialize over and over again from within the same process... maybe the problem comes from doing it both from the main process and a shared lib.

I could take some shortcuts with Python since the core is implemented in it, but I didn't want to give special treatment to it and treat all eventual extension languages (Perl, TCL, etc) the same.