GSoC: OpenCV-Julia binding.

[Alexander Sun]

Hi theres,
My name is Sun Boxiang(Call me Sun, if you wish). I am a post-graduate and interested in the idea of port OpenCV API to Julia.

I am novice for Julia but had read some documents about Julia in these days. I am familiar with Python-C API and OpenCV itself. I try to make a simple OpenCV-Julia wrapper. It could pass some informations from OpenCV to Julia. But I am get stuck now.

Here is what I am done:

In OpenCV-Python port, developer write a wrapper with Python-C API and numpy C API. First, they map cv::Mat to Numpy ndarray. Then parse and wrap the OpenCV C++ functions. Here is an overview.

In Python context: Create a Numpy ndarray object as an image like cv::Mat in C++. Calling a OpenCV-Python function. It will call OpenCV Python wrapper function. In the wrapper function, first it will use py_from/to functions convert the ndarray to cv::Mat, then calling the true OpenCV C++ function. When get C++ return value, convert it to ndarry and return it as a PyObject pointer.

I try to write a wrapper in C. In Python, opencv developer use numpy C API to convert cv::Mat to numpy ndarray. But in Julia, the problem is there has no corresponding object like ndarry in Python. And map Julia's object to C struct look like have some problems, currently I use StrPack, but it also have some problems.

My approach is convert cv::Mat to a intermediate C struct--JMat, then map the C-JMat to Julia. Wrap the OpenCV C++ function to C function that corresponding the Julia calling conventions. Compile the wrap code to a shared lib. Then call it from Julia. The main task is write julia_from/julia_to functions to map C-Struct to Julia object. But I have trouble on it. Maybe it because that I don't familiar with Julia yet.

Here is a minimal prototype that I wrote:

===sample files begin===
**opencv-julia-test.h
#include "opencv2/core/core.hpp"
#include "opencv2/highgui/highgui.hpp"
#include <iostream>


extern "C" int getJMaxRows();
typedef struct JMat
{
    //! the number of rows and columns or (-1, -1) when the matrix has more than 2 dimensions
    int rows, cols;
    //! pointer to the data, but didn't know how to map it correctly, yet...
    uchar* data;
}JMat;

static JMat* tempMat = (struct JMat*)malloc(sizeof(struct JMat)); //when convert it Julia Mat, abandon it.
extern "C" void getMat(struct JMat* jmat_c);
extern "C" void myimread(char *filename);

**opencv-julia-test.cpp
#include "opencv-julia-test.h"
using namespace cv;
void myimread(char* filename)
{
    Mat image= imread(filename);
    tempMat->cols = image.cols;
    tempMat->rows = image.rows;
    tempMat->data = image.data;

    return;
}

**cvtest.jl
using StrPack
module cv2
using StrPack
export cv2

@struct type Mat   
    rows :: Int32
    cols :: Int32
    data :: Ptr{Uint8}
end
Mat(rows, cols,data) = Mat(convert(Int32, rows),
                convert(Int32, cols),
                convert(Ptr{Uint8}, data)
                )

function getMat()
    template = Mat(0, 0, convert(Ptr{Uint8}, ""))
    iostr = IOString()
    pack(iostr, template)
    seek(iostr, 0)
    data = unpack(iostr, Mat)

    return data
end

function imread(filename::String)
    ccall( (:myimread, "./libopj1"), Ptr{Void}, (Ptr{Uint8},), bytestring(filename))
println(3)
    mat = getMat()
   
    println(mat.cols)

    return
end
end

cv2.imread("/home/sun/workspace/julia/mars.jpg")

===sample files end===

If this approach is practical and the problem that stuck me could be solved. I will make a proposal in these day.

Thanks!
Any comments and suggestion will highly appreciated!

Sun

[Tim Holy]

Nice to see your interest in this problem.

On Sunday, March 16, 2014 05:07:27 AM Alexander Sun wrote:
> But in Julia, the problem is there has
> no corresponding object like ndarry in Python.

Why can't you use an Array? See
http://docs.julialang.org/en/latest/manual/arrays/
But it would probably be even better to use Images.jl, since that way you can
encode any metadata (like the colorspace, etc).

For a particularly-sophisticated example of how to design a wrapper around a
big C library, see Gtk.jl. There's a mix of automatic generation and hand-
written code.

--Tim

[Stefan Karpinski]

The PyCall package might be helpful to take a look at:

https://github.com/stevengj/PyCall.jl/blob/master/src/numpy.jl

It both proxies NumPy arrays as Python objects from Julia and supports zero-copy conversion of NumPy arrays to native Julia arrays by sharing data. I suspect that you might be able to call the entire OpenCV python library using PyCall, which seems like it could be a much easier approach to this. Of course, directly wrapping OpenCV would be better, albeit significantly more work because it's C++.

[Alexander Sun]

Hi theres,

It seems that submit a proposal need code portfolio and mentor. Here is my Julia-OpenCV binding prototype and the implementation idea.

The OpenCV-Julia wrapper project will have three parts: the layers, the API generate tools and the documentations.

The first thing are two layers.

The first layer is write in C, build on the top of OpenCV C++ API. And make sure that these C functions are corresponding the Julia calling conventions. The layer is responsible for convert data type between cv::Mat class and J_Mat struct( a cv::Mat data holder that in this C layer). Other objects are similar.

The second layer is write in Julia, this would be as a bridge between C and Julia context. The second layer used to convert data type between J_Mat and Image(in Images.jl).

The first layer written in C and will compile it to a shared library. The second layer will be a Julia package.

The second thing are generation tools.

When these two layers were completed. Write a parser and generator that base on the OpenCV-Python header file parser(hdr_parer.py and gen.py) in Julia. Parse the OpenCV exported API to Julia automatically.

The prototype could be seem as the "first layer". Here is the link:

https://github.com/Daetalus/Julia-OpenCV-binding

Any comments or suggestion are highly appreciated! 

Thanks!

Sun

在 2014年3月18日星期二UTC+8上午12时33分12秒，Stefan Karpinski写道：

[Tim Holy]

I don't have direct experience using OpenCV myself, but because of my role in
Images I'm a likely mentor for an OpenCV project. Others may chime in too.

As a prototype and proof-of-principle, this looks quite promising---nice work.
In the "real" version, I'm not quite sure I see the need for the JMat struct,
because the information that's there is also available in an Array{Uint8, 2}.
One good design might be to return a Mat* to Julia as an opaque type, and then
(if you can't work directly with the Mat struct from Julia) have ccall-able
utility functions that query the size, get the pointer address, etc. Then your
Julia wrapper can use pointer_to_array to access the buffer as a Julia Array.
There's an example of this kind of interface in base/stat.jl (for working with
the horribly non-portable filesystem stat struct) and the corresponding C stubs
in src/sys.c.

Best,
--Tim

> > On Sun, Mar 16, 2014 at 1:23 PM, Tim Holy <tim....@gmail.com <javascript:>

[Jameson Nash]

Unrelated, but that stat struct is now defined in uv.h, which eliminates all of the system dependency of the elements. 

[Kevin Squire]

I've been looking into OpenCV for a project recently.  While I don't have a deep understanding, I'm going to suggest it might be best, initially, to build off of the current python wrapper generator, which is written in Python and which is used as the basis for the Python and Java wrappers, and potentially the C wrappers in the future.  I feel this is the best way to keep the wrappers maintained in the future.  

These have been pointed out before, but you should definitely look at:

  http://answers.opencv.org/question/6618/how-python-api-is-generated/

  http://answers.opencv.org/question/27495/auto-generation-of-python-wrappers/

Cheers,

   Kevin

[Jake Bolewski]

Kevin is right. I said this before but anyone looking into this project should really look at the Matlab bindings in the OpenCV project.  They were written last year and are the third iteration of using OpenCV's C++ header parsing infrastructure to generate a language binding.  The way that they are designed, using a model decoupled from the template generation is the way to keep things sane and readable.  The auto generated Python bindings are a bit of a mess in this regard.

[Kevin Squire]

Thanks for mentioning that again, Jake--I had forgotten. 

Here is a link to the matlab bindings readme, which has some nice design details:

https://github.com/Itseez/opencv/blob/master/modules/matlab/README.md

Cheers, Kevin

[Alexander Sun]

Thanks for the comments!

To Tim:

For OpenCV 3.0, according the OpenCV-dev mail-list, it will introduce new matrix that called UMat, which could upload and download data to/from device(GPU) automatically. It means it will replace the Mat, oclMat, GpuMat. But it undocumented yet. Currently, I don't know how the UMat will impact to OpenCV-Julia binding. So here I made a JMat as a bridge，just in case. But in "real" version, I will try my best to remove it.

To other enthusiastic friends:

As I said in my second post, Julia-OpenCV has two major part: the layers and the generation tools.

Like OpenCV for Python, Java, Matlab bindings. Of course, we can use current python wrapper generator as second part. I didn't exclude it completely, my thought is make Julia-OpenCV binding has less dependency. If people just want to compile and use OpenCV in Julia, they just need install OpenCV and Julia, no need for Python.

If the dependency is not a problem. Use current python wrapper generator to parse OpenCV's C++ header file is the best idea I think. It could simplify the Julia-OpenCV binding.

TIA

Sun

[Joseph Curtin]

I have nothing technical to contribute, but socially: I can't wait to see what comes of this. Looking forward to the prototype. 

[Alexander Sun]

Proposal has submitted.

https://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2014/daetalus/5649050225344512

Any comments? Thanks!

Sun

[Ivar Nesje]

I get the error message:

You are not logged in as the user in the URL.

when I access the link. Maybe it will be visible for registered mentors?

Ivar

[Alexander Sun]

Sorry, I changed the visibility to public.

在 2014年3月21日星期五UTC+8下午5时23分12秒，Ivar Nesje写道：

[Mike Innes]

I get the same error - I think you'll need to repost the link now that you've changed the visibility.

You can try the link in incognito/private browsing mode if you want to check that it will be accessible by others.

[Mike Innes]

I get the same error - I think you'll need to repost the link now that you've changed the visibility.

You can try the link in incognito/private browsing mode if you want to check that it will be accessible by others.

On Friday, 21 March 2014 10:00:16 UTC, Alexander Sun wrote:

[Alexander Sun]

Sorry for the mistake. Here is the correct link:

http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/daetalus/5649050225344512