Kinect Python binding and API design

[Kelvie Wong]

I'd like to start a Python binding for libfreenect, and started a
prototype here:

https://github.com/kelvie/libfreenect/commit/7221fa0adab12a44f4fc01a8bf19055406cd80f8

I know I'm pretty bad at API design (especially pythonic API design),
and perhaps I shouldn't be using ctypes and just write it using the C
api.

Anyone have any ideas how this is improved? Is any one else working
on something similar?

--
Kelvie Wong

[Thomas Hansen]

check out cython.  its very easy to write native c extension using it.  You basically get to use python syntax, but can apply static typing to your varibles.  Cython will convert the code to C before its compiled to a python extension.  It therefore also allows you to make straight C calls from within cython as long as you define the correct externs and header files.  Since there is no need to wrap all the calls and converting arguments for every C call like done by ctypes, performance is usually much better.

--

Thomas

[Nirav Patel]

I had started on a Python C API wrapper for libfreenect, which is also
fairly straightforward and efficient. My goal was to put a callbacks
thread in the module itself, to provide synchronous API calls for the
RGB and Depth images in Python. I think this makes it easier to use,
as the Python script/app would not need to deal with threads or
callbacks. I'll probably abadon my project though, now that your
bindings are there and work has started on a kernel module.

Nirav

[Ćharlęs]

Nirav, did you go to Lehigh Univ as undergrad

[Kelvie Wong]

On Thu, Nov 18, 2010 at 11:54 AM, Nirav Patel <n...@eclecti.cc> wrote:
> I had started on a Python C API wrapper for libfreenect, which is also
> fairly straightforward and efficient.  My goal was to put a callbacks
> thread in the module itself, to provide synchronous API calls for the
> RGB and Depth images in Python.  I think this makes it easier to use,
> as the Python script/app would not need to deal with threads or
> callbacks.  I'll probably abadon my project though, now that your
> bindings are there and work has started on a kernel module.
>
> Nirav

I was thinking of doing this as well, the threading thing. Of course,
this would have to happen inside a C API wrapper.

Or maybe we should use Cython, as suggested earlier.

--
Kelvie Wong

[Jesús Torres]

Another alternative to ctypes and cpython is to use swig. The main
advantage is it allows to make automatically bidings to many languages
at same time. I use it to make a java binding of libdc1394 two years
ago and I can say that it works fine.

Jesús

[Kelvie Wong]

On Thu, Nov 18, 2010 at 11:54 AM, Nirav Patel <n...@eclecti.cc> wrote:

> I had started on a Python C API wrapper for libfreenect, which is also
> fairly straightforward and efficient.  My goal was to put a callbacks
> thread in the module itself, to provide synchronous API calls for the
> RGB and Depth images in Python.  I think this makes it easier to use,
> as the Python script/app would not need to deal with threads or
> callbacks.  I'll probably abadon my project though, now that your
> bindings are there and work has started on a kernel module.
>
> Nirav
>

Could you actually publish that code? I think I may be interested in
writing exactly what you described now. I want to see how far in you
are before starting completely from scratch.

Although I just may do that tonight if I get bored.
--
Kelvie Wong

[Nirav Patel]

Sure. Completely untested (literally, didn't even get it to compile
properly), but here: https://github.com/nrpatel/pyfreenect

Nirav

[Kyle Machulis]

Kelvie: Should I hold off on your pull request?

[Kelvie Wong]

On Thu, Nov 18, 2010 at 11:30 PM, Kyle Machulis <ky...@nonpolynomial.com> wrote:
> Kelvie: Should I hold off on your pull request?
>

It wouldn't hurt to merge it now. I think it is still useful in its
current state (as a light wrapper around the C functions).

The other implementation will take some time.

--
Kelvie Wong

[ptone]

On Nov 18, 12:08 pm, Kelvie Wong <kel...@ieee.org> wrote:
> On Thu, Nov 18, 2010 at 11:54 AM, Nirav Patel <n...@eclecti.cc> wrote:

> > I had started on aPythonC API wrapper for libfreenect, which is also

> > fairly straightforward and efficient.  My goal was to put a callbacks
> > thread in the module itself, to provide synchronous API calls for the

> > RGB and Depth images inPython.  I think this makes it easier to use,
> > as thePythonscript/app would not need to deal with threads or

> > callbacks.  I'll probably abadon my project though, now that your
> > bindings are there and work has started on a kernel module.
>
> > Nirav
>
> I was thinking of doing this as well, the threading thing.  Of course,
> this would have to happen inside a C API wrapper.
>
> Or maybe we should use Cython, as suggested earlier.

I'm much more Python than C person - but let me say that my experience
with Cython has been great.

I've gotten libfreenect installed OK, but am hitting snags with this
python implementation.

first I'm on a Mac,which uses .dylib instead of .so - not sure what
the standard way of handling these differences in ctypes is. But I
was able to just change .so to .dylib.

The next error is I think related to the platform specific use of a
Mac specific IOkit. This might be called by libusb which is called by
libfreenect??


$ python __init__.py
Traceback (most recent call last):
File "__init__.py", line 53, in <module>
lib = CDLL("lib%s.dylib" % libname)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/
python2.6/ctypes/__init__.py", line 345, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(libfreenect.dylib, 6): Symbol not found:
_IOCreatePlugInInterfaceForService
Referenced from: /usr/local/lib/libusb-1.0.0.dylib
Expected in: flat namespace
in /usr/local/lib/libusb-1.0.0.dylib

I think, but am not sure that this sort of problem is less opaque with
cython. I'm highly motivated to get a python implementation working -
and like I said, I'm not much of a C person, but can help test and
debug and am happy to write the more pythonic higher level stuff,
examples, and docs.

-Preston

[Christopher Jowett]

This is from a guy in IRC, he doesn't have permissions to post in the group yet.

<brandyn> cryptk, Do you mind posting a link to my repo
https://github.com/bwhite/libfreenect/tree/master/c/python/ at this
thread http://is.gd/hu6Up? You can just paste this message verbatim

[ptone]

On Nov 20, 8:37 am, Christopher Jowett <cry...@gmail.com> wrote:
> This is from a guy in IRC, he doesn't have permissions to post in the group yet.

Here is the full message - he sent me as well (looks like only the
link got posted)


Hey guys I've been trying to post to the list but I'm still pending.
Check out my ctypes python library.
https://github.com/bwhite/libfreenect/tree/master/c/python/

The advantages are
1. No need to compile extra code. The Python module is harder to
maintain which is a problem for a quickly moving project. This has a
lower barrier to entry and doesn't require the python dev packages.
An argument for the module may be speed; however, very little time is
spent in the module as mine just passes pointers between the user's
python code and the C interface.
2. My runloop is the simplest you can get. The factories can be used
to take care of conversions (check out the opencv and matplotlib demos
I have in that repo).
3. The current interface that was checked in is limited. Mine mimics
the c version exactly, simplifying the semantics and again keeping
maintenance low.

Also can someone post this on my behalf on the forum as I have been
unable to (just copy the whole message).

-Brandyn

>
> <brandyn> cryptk, Do you mind posting a link to my repohttps://github.com/bwhite/libfreenect/tree/master/c/python/at this

> threadhttp://is.gd/hu6Up? You can just paste this message verbatim

[ptone]

I'm unable to get Brandyn's code to work either. I'm getting:


python demo_mp_depth_show.py

Traceback (most recent call last):

File "demo_mp_depth_show.py", line 1, in <module>
from freenect import *
File "/Users/preston/UNIX/src/kinect/libfreenect/c/python/
freenect.py", line 111, in <module>
_fn, freenect_depth_cb, freenect_rgb_cb = _setup_shared_library()
File "/Users/preston/UNIX/src/kinect/libfreenect/c/python/
freenect.py", line 61, in _setup_shared_library
fn = _np.ctypeslib.load_library('libfreenect', '../build/lib')
File "/System/Library/Frameworks/Python.framework/Versions/2.6/
Extras/lib/python/numpy/ctypeslib.py", line 111, in load_library
raise e
OSError: dlopen(/Users/preston/UNIX/src/kinect/libfreenect/c/build/lib/
libfreenect.pyd, 6): image not found


Looks like it is expecting a .pyd library file somewhere - but
libfreenect is a dylib??

-Preston

[Joshua Blake]

On Sat, Nov 20, 2010 at 12:14 PM, ptone <pre...@ptone.com> wrote:

On Nov 20, 8:37 am, Christopher Jowett <cry...@gmail.com> wrote:
> This is from a guy in IRC, he doesn't have permissions to post in the group yet.

Also can someone post this on my behalf on the forum as I have been
unable to (just copy the whole message).

-Brandyn

 

All mailing list apps are up to date. Sometimes we have to sleep. ;)

 

[Kelvie Wong]

This error just looks like a missing symbol. Is whatever library that
provides that IO function in your link path?

--
Kelvie Wong

[bwh...@dappervision.com]

The current version works in OSX if you use ipython and sudo (some
people are reporting they don't need to use sudo). It appears that
ipython adds more paths to search for libraries.

-Brandyn

On Nov 20, 12:45 pm, Kelvie Wong <kel...@ieee.org> wrote:

[bwh...@dappervision.com]

Some users wanted me to remove the numpy requirement so I did. Now it
just requires python to run the demo_tilt.py example and load the
library. There are still some lingering issues with some OSX users
but they appear to be due to non-standard build environments.

-Brandyn

On Nov 20, 12:56 pm, "bwh...@dappervision.com"

[Kelvie Wong]

On Sat, Nov 20, 2010 at 11:33 AM, bwh...@dappervision.com
<bwh...@dappervision.com> wrote:
> Some users wanted me to remove the numpy requirement so I did.  Now it
> just requires python to run the demo_tilt.py example and load the
> library.  There are still some lingering issues with some OSX users
> but they appear to be due to non-standard build environments.
>
> -Brandyn
>

Just wondering -- what is the performance of these bindings? In my
ctypes implementation, they perform quite poorly, just about 8.5
frames per second. And this is just running asserts and print
statements, with no actual processing.

--
Kelvie Wong

[Mathieu Virbel]

Ctypes is very very slow, but easy to use :)
Cython avoid a lot python overhead, so you'll always have more
performance from Python extension than ctypes.

2010/11/20 Kelvie Wong <kel...@ieee.org>:

[Kelvie Wong]

On Sat, Nov 20, 2010 at 11:49 AM, Mathieu Virbel <mat...@pymt.eu> wrote:
> Ctypes is very very slow, but easy to use :)
> Cython avoid a lot python overhead, so you'll always have more
> performance from Python extension than ctypes.
>

Cython would add another dependency, would it not? Or is it easy to
install in Windows/OS X, etc?

I was just thinking of writing a pure Python/C extension (which is not
hard at all), which is supported on all platforms Python is, and
distutils can build it for us.

--
Kelvie Wong

[Mathieu Virbel]

Cython generate C code. You don't need to own cython in order to
compile. Just for generating .pyx -> .c, that's can be done by
developper.
Note that numpy depend on cython, and numpy is widely used :)

2010/11/20 Kelvie Wong <kel...@ieee.org>:

[egradman]

I'm a huge fan of pure python/C modules. In fact, I wrote a pure C
module for flycap (pointgrey) cameras that exposes frames as numpy
arrays, python opencv objects, PIL objects, and cairo surfaces. I'm
going to drop the binding into that code this weekend.

I want to be using kinect with my large existing python-opencv
codebase by Sunday night.

On Nov 20, 12:16 pm, Kelvie Wong <kel...@ieee.org> wrote:

[Kelvie Wong]

On Sat, Nov 20, 2010 at 1:11 PM, Mathieu Virbel <mat...@pymt.eu> wrote:
> Cython generate C code. You don't need to own cython in order to
> compile. Just for generating .pyx -> .c, that's can be done by
> developper.
> Note that numpy depend on cython, and numpy is widely used :)
>

Interesting. Is anyone else working on a Cython binding right now?
Maybe one that produces a synchronous API by starting a thread in C
(as was suggested earlier).

Otherwise, I am going to start work on that today/tomorrow, either in
pure C or in Cython.
--
Kelvie Wong

[ptone]

On Nov 20, 11:33 am, "bwh...@dappervision.com"

<bwh...@dappervision.com> wrote:
> Some users wanted me to remove the numpy requirement so I did.  Now it
> just requires python to run the demo_tilt.py example and load the
> library.  There are still some lingering issues with some OSX users
> but they appear to be due to non-standard build environments.
>
> -Brandyn

The removal of the numpy flavor of dlopen and switch to vanilla ctypes
now leads to the same problem I was having with Kelvie's first take.
Somehow the linking of libfreenect with libusb on os x is causing a
problem.

dlopen(/usr/local/lib/libfreenect.dylib, 6): Symbol not found:

_IOCreatePlugInInterfaceForService
Referenced from: /usr/local/lib/libusb-1.0.0.dylib
Expected in: flat namespace
in /usr/local/lib/libusb-1.0.0.dylib

This has to do with the way libusb is built on darwin - that symbol is
in fact present in the libusb binaries - but its not being found when
the library is loaded this way. Problem didn't exist for me with
Brandyn's first version.

On the pure C vs cython - the main reason I like the idea of cython,
is that would allow python developers to add more preprocessing of the
data before it gets to python at runtime, but still write this in
cython/python.

The quest continues....

-Preston

[Kelvie Wong]

Can we not use both? :)

I tried googling Cython and threads, and that does not really turn up
any useful results (at first glance, anyways).

I was planning on writing a synchronous API, and am not sure if I am
able to thread in Cython.

--
Kelvie Wong

[bwh...@dappervision.com]

Yeah I'm in the middle of a Cython version and I'll compare it to what
is checked in now. I doubt there will be a noticeable impact as it is
really just passing pointers around (on my netbook it runs full
speed). However, the point about adding in more processing code makes
sense. I do think it will be slightly more annoying to build (people
complained about numpy as a req); however, if we can keep the same
interface then there is no harm just keeping both around.

Can I get some feedback on the actual interface to the module? That
you get different types of data inputs (string, numpy, array, opencv
(next push))? I kept all of the same functions as the C module and
added accelerometer helpers.

-Brandyn

On Nov 20, 8:03 pm, Kelvie Wong <kel...@ieee.org> wrote:

[bwh...@dappervision.com]

I allowed it to try using numpy now if it is available but it still
isn't required. On a simple test I just did the demo_cv_depth_show.py
example got 10FPS on my 1.66 Ghz atom netbook with no hardware
acceleration for the display. If I comment out the display but keep
the gamma correction (so it is doing something to the data) I get
26-27FPS consistently. So my point is that if the best we can get is
30 FPS then the performance argument isn't as strong and we should
consider maintainability and ease of use. That being said, I'll still
do a cython version and if I can get the interface similar to it is
now then I'll report back maybe tomorrow with it.

-Brandyn

[evershade]

I'm having trouble getting the demo code to run. Here's my output:

Traceback (most recent call last):
File "demo_mp_depth_show.py", line 1, in <module>
from freenect import *

File "/home/sean/programming/kinect/libfreenect/c/python/
freenect.py", line 291, in <module>
freenect_depth_cb, freenect_rgb_cb = _populate_namespace()
File "/home/sean/programming/kinect/libfreenect/c/python/
freenect.py", line 123, in _populate_namespace

_fn, freenect_depth_cb, freenect_rgb_cb = _setup_shared_library()

File "/home/sean/programming/kinect/libfreenect/c/python/
freenect.py", line 111, in _setup_shared_library
mk(c_int, fn.freenect_set_tilt_degs, [c_void_p, c_double])
File "/usr/lib/python2.6/ctypes/__init__.py", line 366, in
__getattr__
func = self.__getitem__(name)
File "/usr/lib/python2.6/ctypes/__init__.py", line 371, in
__getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/lib/libfreenect.so: undefined symbol:
freenect_set_tilt_degs

It seems to be missing the freenect_set_tilt_degs method. I'm not used
to following Python back into C code but as far as I can tell, this
method is declared properly in libfreenect.h and defined in tilt.c.

I haven't modified any files since pulling from the git repository,
besides building the c project and copying the libfreenect files over
to the python directory.

Anyone know what I'm doing wrong?

[ptone]

I think you have a mismatch between the vers of libfreenect you have
built, and the version of the python code you are using - there were
some changes today, but all should be in sync in the repo.

-Preston

[FreakTheMighty]

How about using pyusb? it seems like it shouldn't be too difficult to
replicate the c driver. The example in this post shows how to grab
accelerometer data in this fashion. http://ladyada.net/learn/diykinect/
I began copying the driver into python but got hung up after the
camsinit function. I think with some more research I can get past
it.

I like the idea of going this route because then we will have low
level access within python.

Check out this post.

Jesse

On Nov 18, 10:54 am, Kelvie Wong <kel...@ieee.org> wrote:
> I'd like to start a Python binding for libfreenect, and started a
> prototype here:
>
> https://github.com/kelvie/libfreenect/commit/7221fa0adab12a44f4fc01a8...
>
> I know I'm pretty bad at API design (especially pythonic API design),
> and perhaps I shouldn't be using ctypes and just write it using the C
> api.
>
> Anyone have any ideas how this is improved?  Is any one else working
> on something similar?
>
> --
> Kelvie Wong

[Joshua Blake]

Per https://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTinaApkBW%3D2Pgfeej1B4dY6AZ%3DzHGxpGYUm5MC5_%40mail.gmail.com&forum_name=pyusb-users, isochronous transfers are not yet implemented in pyusb. (We explored using pyusb the night Adafruit published the USB dumps but that was a no-go.)

 

Also it doesn't make sense to replicate the low-level cross-platform C driver and API in python or any other high-level language. Centralizing the core libfreenect code means that any bugs or enhancements only have to be done in one place and then all languages and platforms can use it.

 

The C API can expose anything that you need. What type of low-level data or control were you thinking about?

Josh

[evershade]

Thanks, I found the problem. I'm running Ubuntu and I had previously
installed the demo from the demo packages. Looks like the python
script was calling an old version of libfreenect. Oops...

Now my kinect is happily tilting away.

[FreakTheMighty]

I see your point about keeping the code base centralized, though it
doesn't seem to me like the driver is particularly large. From my
perspective if it were possible to replicate the driver in pure python
the benefit would be transparency. That is, the low level
communication to the usb driver would be easier to read and
understand. Sure there are a finite number of commands you might want
to send or receive, but for educational purposes and the possibility
that some guy sending random commands to the Kinect discovers it can
also cook turkey dinners, implementing lower level functions in python
could prove beneficial.

That said, if it is technically impossible to do with pyusb, that
would seem to be a nonstarter.

Thanks for the info.

Regards,

Jesse
On Nov 21, 9:38 am, Joshua Blake <joshbl...@gmail.com> wrote:
> Perhttps://sourceforge.net/mailarchive/forum.php?thread_name=AANLkTinaAp...,

> isochronous transfers are not yet implemented in pyusb. (We explored using
> pyusb the night Adafruit published the USB dumps but that was a no-go.)
>
> Also it doesn't make sense to replicate the low-level cross-platform C
> driver and API in python or any other high-level language. Centralizing the
> core libfreenect code means that any bugs or enhancements only have to be
> done in one place and then all languages and platforms can use it.
>
> The C API can expose anything that you need. What type of low-level data or
> control were you thinking about?
>
> Josh

> On Sun, Nov 21, 2010 at 12:23 PM, FreakTheMighty <jvanderd...@gmail.com>wrote:
>
>
>
>
>
>
>
> > How about using pyusb?  it seems like it shouldn't be too difficult to
> > replicate the c driver. The example in this post shows how to grab

> > accelerometer data in this fashion.http://ladyada.net/learn/diykinect/

[Andrew Miller]

I found a peculiar workaround for this problem. If you import
OpenGL.GL before importing freenect, then it works and you no longer
get that error. Before finding this solution, I've been able to use
freenect by running everything through IPython. My guess is that the
same thing is happening in either case... both IPython and pyopenGL
cause the right mac library to be loaded.

If you don't want to have a dependency to OpenGL or IPython, maybe
there are other libraries that solve the same problem. Try import
freenect last of all. Maybe this helps shed light on the problem as
well.

On Nov 20, 11:27 pm, ptone <pres...@ptone.com> wrote:

> > > > > Mac specificIOkit.  This might be called by libusb which is called by

[bwh...@dappervision.com]

Ok preliminary Cython is checked in for anyone that is interested.
Also changed the library interface a bit.

On Nov 20, 11:27 pm, ptone <pres...@ptone.com> wrote:

[sigmac]

I think I am seeing the same problem.

I installed on Mac OS X using Homebrew (based on the wiki
instructions). The glview app works but when trying to run any of the
python wrapper scripts I get the error:

Traceback (most recent call last):

File "demo_tilt.py", line 6, in <module>
import freenect
ImportError: dlopen(/usr/local/wrappers/python/freenect.so, 2): Symbol
not found: _IOCreatePlugInInterfaceForService
Referenced from: /usr/local/Cellar/libusb-freenect/
7da756e09fd97efad2b3/lib/libusb-1.0.0.dylib
Expected in: flat namespace
in /usr/local/Cellar/libusb-freenect/7da756e09fd97efad2b3/lib/
libusb-1.0.0.dylib

Any ideas?

On Nov 20, 11:27 pm, ptone <pres...@ptone.com> wrote:

[Andrew Miller]

This was mentioned in another thread on this list I think, but the solution to this problem on Mac has been to add ./configure options when we build libusb.

./configure LDFLAGS='-framework IOKit -framework CoreFoundation'

The current homebrew formula for libfreenect-libusb doesn't include these option.

https://github.com/OpenKinect/libfreenect/blob/master/platform/osx/homebrew/libusb-freenect.rb#L15

If you add the extra LDFLAGS option to the libfreenect-libusb formula, it should work.

--
Andrew Miller
Dapper Vision Inc.

[Andrew Miller]

Actually this fix is already in a pull request

https://github.com/OpenKinect/libfreenect/pull/146

Credit to emonti!

[sigmac]

Excellent, updated the formula with emonti's change and all is well.
Thanks for the pointer to the updated formula.


On Dec 23, 1:37 am, Andrew Miller <amil...@dappervision.com> wrote:
> Actually this fix is already in a pull requesthttps://github.com/OpenKinect/libfreenect/pull/146
>
> Credit to emonti!

>
> On Thu, Dec 23, 2010 at 12:37 AM, Andrew Miller <amil...@dappervision.com>wrote:
>
>
>
> > This was mentioned in another thread on this list I think, but the solution
> > to this problem on Mac has been to add ./configure options when we build
> > libusb.
>
> > ./configure LDFLAGS='-framework IOKit -framework CoreFoundation'
>
> > The current homebrew formula for libfreenect-libusb doesn't include these
> > option.
>

> >https://github.com/OpenKinect/libfreenect/blob/master/platform/osx/ho...

>
> > If you add the extra LDFLAGS option to the libfreenect-libusb formula, it
> > should work.
>