OpenCV.jl: A package for computer vision in Julia

[Max Suster]

Hi all, 

A few months ago I set out to learn Julia in an attempt to find an alternative to MATLAB for developing computer vision applications.

Given the interest (1,2,3,4) and wide application of OpenCV for fast real-time computer vision applications, I set myself to put together a simple interface for OpenCV in Julia.  Coding in Julia and developing the interface between C++ and Julia has been a lot of fun!

OpenCV.jl aims to provide an interface for OpenCV computer vision applications (C++) directly in Julia. It relies primarily on Keno´s amazing Cxx.jl, the Julia C++ foreign function interface (FFI).  You can find all the information on my package at https://github.com/maxruby/OpenCV.jl.

You can download and run the package as follows:

Pkg.clone("git://github.com/maxruby/OpenCV.jl.git")
using OpenCV

For MacOSX, OpenCV.jl comes with pre-compiled shared libraries, so it is extremely easy to run.  For Windows and Linux, you will need to first compile the OpenCV libraries, but this is well documented and links to the instructions for doing so are included in the README.md file.

The package currently supports most of OpenCV´s C++ API; however, at this point I have created custom wrappings for core, imgproc, videoio and highgui modules so that these are easy to use for anyone.

The package also demonstrates/contains 

• preliminary interface with the Qt GUI framework (see imread() and imwrite() functions)
• thin-wrappers for C++ objects such as std::vectors, std::strings 
  
• conversion from Julia arrays to C++ std::vector
• conversion of Julia images (Images.jl) to Mat (OpenCV) - though this has much room for improvement (i.e., color handling)

Please let me know if there are any features you would like to see added and I will try my best to integrate them. In the meantime, I will continue to integrate more advanced algorithms for computer vision and eventually extend the documentation as needed.

Cheers,

Max 

[Simon Danisch]

How very exciting!!!!!!!!!!!!

And its even over the C++ interface =)

I just recently checked out, how complicated it would be to do this, as I really want to have the GPU accelerated features from OpenCV...

Would be so cool to fuse this natively with my OpenGL library, to view and filter images completely on the GPU, enabling great real-time performance!

Did you manage to do the conversions without copying? 

[Simon Danisch]

Personal note:

I needed to do a lot of interactive 2D and 3D visualizations with results from OpenCV and it was all just very cumbersome...

This was actually one of the primers for me to start searching for a better language, in which you could do the 2D/3D visualizations, without performance penalty and with a high degree of interactivity.

Am Samstag, 6. Dezember 2014 11:44:45 UTC+1 schrieb Max Suster:

[Max Suster]

Glad to hear interest in this package :)   

I have indeed started to work on getting the CUDA features into OpenCV.jl (this was reorganized/relabelled from gpu to CUDA).

My understanding is that OpenCV CUDA algorithms can use only a single GPU, and to utilize multiple GPUs, its necessary to distribute the work between several GPUs manually.  I am experienced and not sure how to do this now with the Julia interface, but if you do know, I would be happy to collaborate on this.  My main goal is to use OpenCV for real-time tracking applications (e.g., principal skeleton tracking), and using GPU (with up to 30x the speed for acquisition) would be invaluable. 

I have tested OpenCV with both boost C++ (multithreading) and GPU-accelerated approaches, and it seems to me that the GPU approach is most promising. One challenge however is that I found it very messy to compile OpenCV 3.0 with CUDA on OSX 10.9.5 and it seems to me that a number of people have reported bugs with the v3.0 build itself (at least on OSX).  The second issue (as I am sure you know) is that for the GPU features to be worthwhile, you need a decent NVIDIA card and my GTX-Force 330M with a Computing Capability (CC) of 1.2 is not exactly amazing Hopefully this will change soon with a new Mac :) 

Since OpenCV is such a large API and it is used widely for so many applications, it will nice to hear from those interested here what features are worth expanding and which maybe less so.  

Max

[Tim Holy]

This looks really cool. Nice work!

As you know well, it looks like most or all of what's in your README is
covered with native Julia solutions (Images, VideoIO, and ImageView). But
clearly the fun is going to come via the "advanced applications," for which
there is very little publicly available in julia.

I'm really looking forward to playing with this!

Best,
--Tim

On Saturday, December 06, 2014 02:44:45 AM Max Suster wrote:
> Hi all,
>
> A few months ago I set out to learn Julia in an attempt to find an
> alternative to MATLAB for developing computer vision applications.
> Given the interest (1

> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-users/P
> jyfzxPt8Gk/SuwKtjTd9j4J> ,2
> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-users/8
> 1V5zSNJY3Q/DRUT0dR2qhQJ> ,3
> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-users
> /iUPqo8drYek/pUeHECk91AQJ> ,4
> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-users
> /6QunG66MfNs/C63pDfI-EMAJ> ) and wide application of OpenCV for fast

> real-time computer vision applications, I set myself to put together a
> simple interface for OpenCV in Julia. Coding in Julia and developing the
> interface between C++ and Julia has been a lot of fun!
>

> OpenCV.jl aims to provide an interface for OpenCV <http://opencv.org/>

> computer vision applications (C++) directly in Julia

> <http://julia.readthedocs.org/en/latest/manual/>. It relies primarily on
> Keno´s amazing Cxx.jl <https://github.com/Keno/Cxx.jl>, the Julia C++

> foreign function interface (FFI). You can find all the information on my
> package at https://github.com/maxruby/OpenCV.jl.
>
> You can download and run the package as follows:
>

> Pkg.clone("git://github.com/maxruby/OpenCV.jl.git")using OpenCV

>
>
> For MacOSX, OpenCV.jl comes with pre-compiled shared libraries, so it is
> extremely easy to run. For Windows and Linux, you will need to first
> compile the OpenCV libraries, but this is well documented and links to the
> instructions for doing so are included in the README.md file.
>
> The package currently supports most of OpenCV´s C++ API; however, at this
> point I have created custom wrappings for core, imgproc, videoio and
> highgui modules so that these are easy to use for anyone.
>
> The package also demonstrates/contains
>

> - preliminary interface with the Qt GUI framework (see imread() and
> imwrite() functions)
> - thin-wrappers for C++ objects such as std::vectors, std::strings
> - conversion from Julia arrays to C++ std::vector
> - conversion of Julia images (Images.jl) to Mat (OpenCV) - though this

[Simon Danisch]

Oh this is odd, why do they name the NVIDIA restricted version GPU-accelerated Computer Vision, and the hardware vendor independent version OpenCL-accelerated Computer Vision?

I personally would first go for the OpenCL module, as its supported by more GPU's (I don't even have an NVIDIA card)...

[Max Suster]

Hi Tim,

Thanks for your feedback. By no means, OpenCV.jl is meant to be a replacement of your outstanding contributions to Images.jl and Kevin´s wonderful VideoIO.jl package. As you may know, I spent some time recently contributing to VideoIO´s options API - so I realize that these packages are highly valuable :)

Having said that, I think you will see that already the README.md file describes the following which are not currently available at all in Julia (or to the same extent via C++):

• Highly efficient pixel scanning and manipulation using pointers in C++ (smooth interface between Julia and C++)
  
• Conversion between images in Images.jl and OpenCV Mat format (to take advantage of hundreds of algorithms readily available in C++)
• Convenient live image processing with adjustable trackbars (e.g., thresholding, and dozens of image processing filters possible without much effort), including options for Qt GUI support
• Object tracking (which is not currently available for Julia) is already demonstrated in test/cxx/demos.jl

There is clearly much work to be done on this package to get it to a mature state so that it supports great computer vision applications with Julia (especially for real-time tracking).   I am looking forward to suggestions for further development.

Cheers,

Max

[Max Suster]

Simon, 

Straight from the OpenCV 3.0 documentation site:

While in the future we hope to validate (and enable) the OCL module in all OpenCL capable devices, we currently develop and test on GPU devices only. This includes both discrete GPUs (NVidia, AMD), as well as integrated chips (AMD APU and Intel HD devices). Performance of any particular algorithm will depend on the particular platform characteristics and capabilities. However, currently, accuracy and mathematical correctness has been verified to be identical to that of the pure CPU implementation on all tested GPU devices and platforms (both Windows and Linux).

Does this mean that they have not tested on OSX or is this more likely due to hardware support?

Someone has encountered problems running OpenCL on OSX, but perhaps this is an isolated case (OpenCL example failed on OSX).

According to the OpenCV 3.0 documentation

Many kernels use 256 as its workgroup size if possible, so the max work group size of the device must larger than 256. All GPU devices we are aware of indeed support 256 workitems in a workgroup, however non GPU devices may not. This will be improved in the future.

However, there is a nice and useful link here for how to support OpenCV-OpenCL on OSX so it maybe worth trying:

https://fogofcode.wordpress.com/2014/02/23/running-opencv-on-osx-with-opencl-support/

Apparently, building OpenCV with OpenCL requires AMDBLAS and AMDFFT.  I guess you are using Linux or Windows, right?

clAmdFft and clAmdBlas are only built for Windows and Linux so I will have to build them first from source in OSX. 

Max

[Tim Holy]

Oh, I'm well aware of your great contributions to VideoIO. And OpenCV looks
like an even bigger one.

However, I think quite a bit of "live processing" is already supported via the
`map/MapInfo` framework. For example, below is a complete, high-performance
GUI for interactively setting a threshold. Because the whole MapInfo framework
is lazy, it can be used with both static images and video.

--Tim

using Base.Cartesian
using Images, Tk, ImageView, Color

import Base: map!

export threshold_slider

# Threshold pixels using a slider
type ThresholdSlider{To,A<:AbstractArray,From,M<:Images.MapInfo} <:
Images.MapInfo{To}
value::A
slider_min::From
slider_max::From
mapi0::M
slider
end

function ThresholdSlider(value::AbstractArray, mapi0, s,
extrma=extrema(value))
smin, smax = promote(extrma[1], extrma[2])
ThresholdSlider{Uint32,typeof(value),typeof(smin),typeof(mapi0)}(value,
smin, smax, mapi0, s)
end

function threshold_slider(img, value, extrma=extrema(value); mapi0 =
mapinfo(RGB24, img))
w = Toplevel()
f = Frame(w)
pack(f, expand=true, fill="both")
l = Label(f, "Threshold: $(extrma[1])")
pack(l, side="top")
s = Slider(f, extrma[1], extrma[2])
ts = ThresholdSlider(value, mapi0, s, extrma)
imgc, imsl = view(img, scalei=ts)
pack(s, side="top", expand=true, fill="both", anchor="w")
bind(s, "command", path -> (set_value(l, @sprintf("%4.2e", get_value(s)));
ImageView.redraw(imgc, imsl)))
end

function map!{T,N}(mapi::ThresholdSlider{Uint32}, dest::Array{Uint32,N},
src::AbstractImageDirect{T,N})
value = sub(mapi.value, data(src).indexes[1:N])
map_thresh!(mapi, dest, src, value, get_value(mapi.slider))
end

@ngenerate N typeof(dest) function map_thresh!{T,N}
(mapi::ThresholdSlider{Uint32}, dest::Array{Uint32,N},
src::AbstractImageDirect{T,N}, value, thresh)
size(dest) == size(src) || throw(DimensionMismatch())
mapi0 = mapi.mapi0
@nloops N i dest begin
c = map(mapi0, @nref N src i)
x = @nref N value i
(@nref N dest i) = x > thresh? convert(RGB24, c) : RGB24(0)
end
dest
end

On Saturday, December 06, 2014 04:34:55 AM Max Suster wrote:
> Hi Tim,
>
> Thanks for your feedback. By no means, OpenCV.jl is meant to be a
> replacement of your outstanding contributions to Images.jl and Kevin´s
> wonderful VideoIO.jl package. As you may know, I spent some time recently
> contributing to VideoIO´s options API - so I realize that these packages
> are highly valuable :)
>
> Having said that, I think you will see that already the README.md file
> describes the following which are not currently available at all in Julia
> (or to the same extent via C++):
>

> - Highly efficient pixel scanning and manipulation using pointers in C++

> (smooth interface between Julia and C++)

> - Conversion between images in Images.jl and OpenCV Mat format (to take

> advantage of hundreds of algorithms readily available in C++)

> - Convenient live image processing with adjustable trackbars (e.g.,

> thresholding, and dozens of image processing filters possible without
> much effort), including options for Qt GUI support

> - Object tracking (which is not currently available for Julia) is

[Tim Holy]

But I should say again: the "advanced" stuff looks great, and definitely there
is no julia analog that I know of. The tracking stuff in particular will be
very nice to have.

--Tim

On Saturday, December 06, 2014 04:34:55 AM Max Suster wrote:
> Hi Tim,
>
> Thanks for your feedback. By no means, OpenCV.jl is meant to be a
> replacement of your outstanding contributions to Images.jl and Kevin´s
> wonderful VideoIO.jl package. As you may know, I spent some time recently
> contributing to VideoIO´s options API - so I realize that these packages
> are highly valuable :)
>
> Having said that, I think you will see that already the README.md file
> describes the following which are not currently available at all in Julia
> (or to the same extent via C++):
>

> - Highly efficient pixel scanning and manipulation using pointers in C++

> (smooth interface between Julia and C++)

> - Conversion between images in Images.jl and OpenCV Mat format (to take

> advantage of hundreds of algorithms readily available in C++)

> - Convenient live image processing with adjustable trackbars (e.g.,

> thresholding, and dozens of image processing filters possible without
> much effort), including options for Qt GUI support

> - Object tracking (which is not currently available for Julia) is

> already demonstrated in test/cxx/demos.jl
>
> There is clearly much work to be done on this package to get it to a mature
> state so that it supports great computer vision applications with Julia
> (especially for real-time tracking). I am looking forward to suggestions
> for further development.
>
> Cheers,
> Max
>
> On Saturday, December 6, 2014 1:19:24 PM UTC+1, Tim Holy wrote:
> > This looks really cool. Nice work!
> >
> > As you know well, it looks like most or all of what's in your README is
> > covered with native Julia solutions (Images, VideoIO, and ImageView). But
> > clearly the fun is going to come via the "advanced applications," for
> > which
> > there is very little publicly available in julia.
> >
> > I'm really looking forward to playing with this!
> >
> > Best,
> > --Tim
> >
> > On Saturday, December 06, 2014 02:44:45 AM Max Suster wrote:
> > > Hi all,
> > >
> > > A few months ago I set out to learn Julia in an attempt to find an
> > > alternative to MATLAB for developing computer vision applications.
> > > Given the interest (1
> > > <
> >

> > https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-users/
> > P
> >
> > > jyfzxPt8Gk/SuwKtjTd9j4J> ,2
> > > <
> >
> > https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-users/
> > 8
> >
> > > 1V5zSNJY3Q/DRUT0dR2qhQJ> ,3
> > > <
> >
> > https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-user
> > s
> >
> > > /iUPqo8drYek/pUeHECk91AQJ> ,4
> > > <
> >
> > https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-user
> > s
> >

[Tim Holy]

I don't know if it will help at all in the context of OpenCV, but here's a
test script that demonstrates farming out work to multiple GPUs:
https://github.com/JuliaGPU/CUDArt.jl/blob/6409b57f7c80ed2459fd46c0e86ab8de681fd9bc/test/test.jl#L185-L222

--Tim

> >> Given the interest (1
> >> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-user
> >> s/PjyfzxPt8Gk/SuwKtjTd9j4J> ,2
> >> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-user

> >> s/81V5zSNJY3Q/DRUT0dR2qhQJ> ,3

> >> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-us
> >> ers/iUPqo8drYek/pUeHECk91AQJ> ,4
> >> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-us

> >> ers/6QunG66MfNs/C63pDfI-EMAJ> ) and wide application of OpenCV for fast

> >> real-time computer vision applications, I set myself to put together a
> >> simple interface for OpenCV in Julia. Coding in Julia and developing
> >> the interface between C++ and Julia has been a lot of fun!
> >>

> >> OpenCV.jl aims to provide an interface for OpenCV <http://opencv.org/>

> >> computer vision applications (C++) directly in Julia

> >> <http://julia.readthedocs.org/en/latest/manual/>. It relies primarily on
> >> Keno´s amazing Cxx.jl <https://github.com/Keno/Cxx.jl>, the Julia C++

> >> foreign function interface (FFI). You can find all the information on my
> >> package at https://github.com/maxruby/OpenCV.jl.
> >>
> >> You can download and run the package as follows:
> >>

> >> Pkg.clone("git://github.com/maxruby/OpenCV.jl.git")using OpenCV

> >>
> >>
> >> For MacOSX, OpenCV.jl comes with pre-compiled shared libraries, so it is
> >> extremely easy to run. For Windows and Linux, you will need to first
> >> compile the OpenCV libraries, but this is well documented and links to
> >> the
> >> instructions for doing so are included in the README.md file.
> >>
> >> The package currently supports most of OpenCV´s C++ API; however, at this
> >> point I have created custom wrappings for core, imgproc, videoio and
> >> highgui modules so that these are easy to use for anyone.
> >>
> >> The package also demonstrates/contains
> >>

> >> - preliminary interface with the Qt GUI framework (see imread() and
> >> imwrite() functions)
> >> - thin-wrappers for C++ objects such as std::vectors, std::strings
> >> - conversion from Julia arrays to C++ std::vector
> >> - conversion of Julia images (Images.jl) to Mat (OpenCV) - though

[Max Suster]

Thanks a lot for pointing out the script.  I will definitely have a close look at the CUDA implementation. 

Regarding the live GUI trackbar with Images, I did not know that this was implemented anywhere (i.e., did not see it in the documentation).

Personally, for my own needs, I also think there are advantages to having a single unified package that allows both the basic image array handling, GUI, and real-time tracking. . .   and in the spirit of Christmas, the more the merrier . . .

Cheers,

Max

[Tim Holy]

It's not in ImageView, it's just something I use privately. Given package load
times, there's some tension between "having everything" and "making things
easy to implement." So Images & ImageView largely focus on ingredients, and
need users to put them together to make the soup. But I suppose we could have
an ImageViewExtras package or something.

Anyway, I fully agree that having more options is great!

--Tim

[Max Suster]

 

http://developer.amd.com/community/blog/2014/10/15/opencv-3-0-transparent-api-opencl-acceleration/

The link above makes it sounds very promising that enabling OpenCL in OpenCV.jl will be rather feasible (assuming no surprises along the way...).

I was first looking frantically for the module ("ocl") in my OpenCV 3.0 folder because the (outdated) web API documentation 

said that there should be such a folder.  However, this is not true - it turns out that from v3.0, the “ocl” namespace and folder are gone, 

and  so is the “oclMat.”  To transfer data one has to use UMat.  

I will try to get this implemented soon.

Max

[Max Suster]

Simon, 

It turns out that using OpenCV.jl with OpenCL support is actually remarkably easy thanks to the transparent T-API of OpenCV 3.0.  All that is needed is to use cv::UMat in place of cv::Mat (and an extra header file in OpenCV.jl).  

A comparison of a simple RGB to gray conversion with CPU vs. GPU Mat suggest at least 10x improvement (even with a mediocre GTX-Force 330M 512MB, CC 1.2):

Declare the Mat and UMat (1000x1000 RGB) source and initialize target images
julia> srcMat = Mat(1000, 1000, CV_8UC3, cvScalar(0, 255, 0)); 
julia> srcUMat = UMat(1000, 1000, CV_8UC3, cvScalar(0, 255, 0));

julia> dstMat = Mat()
julia> dstUMat = UMat()

CPU

julia> @time(cvtColor(srcMat, dstMat, COLOR_BGR2GRAY))
elapsed time: 0.00164426 seconds (80 bytes allocated)
elapsed time: 0.001682741 seconds (80 bytes allocated)
elapsed time: 0.002086002 seconds (80 bytes allocated)
elapsed time: 0.001711169 seconds (80 bytes allocated)
elapsed time: 0.001686306 seconds (80 bytes allocated)
elapsed time: 0.001655958 seconds (80 bytes allocated)
elapsed time: 0.001707453 seconds (80 bytes allocated)

GPU
julia> @time(cvtColor(srcUMat, dstUMat, COLOR_BGR2GRAY))
elapsed time: 0.000149589 seconds (80 bytes allocated)
elapsed time: 0.000149705 seconds (80 bytes allocated)
elapsed time: 0.000149575 seconds (80 bytes allocated)
elapsed time: 0.000151404 seconds (80 bytes allocated)
elapsed time: 0.00017067 seconds (80 bytes allocated) 

elapsed time: 0.000181821 seconds (80 bytes allocated)

[Stefan Karpinski]

That's really cool. I'm looking forward to playing around with this.

[Simon Danisch]

That's really great! I will have to investigate, how to convert between OpenGL datatypes and UMat.
It would be incredible, if we can convert between julia, opencl, opengl and opencv datatypes without a big overhead.

I'm pretty sure, that no other language has this solved nicely! ;)

Am Samstag, 6. Dezember 2014 11:44:45 UTC+1 schrieb Max Suster:

[Max Suster]

http://stackoverflow.com/questions/9097756/converting-data-from-glreadpixels-to-opencvmat

Seem that the link above shows a solution to this. 

Max

[Simon Danisch]

That's the trivial solution with quite a few copies and transfers involved.

But if the Umat is used with OpenCL, the image data should reside on the GPU already, which means you don't need to download it from the gpu and upload it back again for displaying the data.

But that means, it must be possible to create a UMat from an OpenCL buffer, as it must be created first via OpenGL (silly restriction for the OpenCL OpenGL interoperability).

If that works, one can manipulate the Umat via OpenCL and the OpenGL buffer will be the same memory and can be displayed without any copies ;)

[Simon Danisch]

This would be a related (unanswered ) question: http://answers.opencv.org/question/38848/opencv-displaying-umat-efficiently/

As I don't see any related constructors for the Umat, the easiest starting point would be to ask in the forum.

This will probably get faster results, than diving into the code ourselves.

[Simon Danisch]

If you're interested here are some more links:

https://software.intel.com/en-us/articles/opencl-and-opengl-interoperability-tutorial
Valentine's and mine prototype for OpenGL OpenCL interoperability in Julia:

https://github.com/vchuravy/qjulia_gpu

Am Samstag, 6. Dezember 2014 11:44:45 UTC+1 schrieb Max Suster:

[Max Suster]

Thanks for the feedback.  I realize that the copying needs to be skipped if possible . . .

I have been playing a bit with the OpenCL UMat and it will need indeed some tweeking because UMat is not always advantageous.

While there 10x gain with cvtColor and other functions such as GasussianBlur are actually a little slower.

I will have closer look at this tonight.

Max

[Simon Danisch]

That's interesting, gaussian blur should definitely be faster on the gpu!

Maybe this thread helps?

http://answers.opencv.org/question/34127/opencl-in-opencv-300/

It seems like things are a little complicated, as it isn't really clear if the data is currently in VRAM or RAM...

[Max Suster]

I know this is a Julia coding forum, but if you have a chance, can you compare the two examples below?   

Also, what OS are you using?   Before testing a lot of Julia wrapped C++ API with OpenCV/OpenCL/OpenGL, 

it would be good to know what we can expect at best. . . 

CPU
int main(int argc, char** argv)
{
      Mat img, gray;
      img = imread("your RGB image", -1); 

      imshow("original", img);
      cvtColor(img, gray, COLOR_BGR2GRAY);                // 50 ms
      GaussianBlur(gray, gray,  Size(7, 7), 1.5);                  // 2 ms                                                    

      double t = (double)getTickCount();
      Canny(gray, gray, 0, 50);                                             // 6 ms 
      t = ((double)getTickCount() - t)/getTickFrequency(); 
      cout << "Times passed in seconds: " << t << endl;
      imshow("edges", gray);
      waitKey();
      return 0;
}


GPU
static UMat img, gray;

int main(int argc, char** argv) 
{     
    Mat src;
    src = imread("your RGB image", -1);
    src.copyTo(img); // copy from CPU to GPU buffer
    imshow("original", src);
    double t = (double)getTickCount();
    cvtColor(img, gray, COLOR_BGR2GRAY);                  // 3 ms
    t = ((double)getTickCount() - t)/getTickFrequency();
    cout << "Times passed in seconds: " << t << endl;
    GaussianBlur(gray, gray, Size(7, 7), 1.5);                   // 13 ms
    Canny(gray, gray, 0, 50);                                             // 10 ms
    imshow("edges", gray);
    waitKey();
    return 0;

} 

[Tim Holy]

I wonder if the bigger problem might be that your numbers for the grayscale
conversion (which were very promising) might be misleading. Are you sure the
calculation is "done" (and the results are available to the CPU) by the time
it finishes? If we assume a best-case scenario of 6GB/s of data transfer to the
GPU, then transferring 3MB to the GPU and 1MB back takes about 0.7ms. That's
many times longer than what you reported for that calculation. Or did you not
include transfer time in your results?

--Tim

> >>> Given the interest (1
> >>> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-use
> >>> rs/PjyfzxPt8Gk/SuwKtjTd9j4J> ,2
> >>> <https://groups.google.com/forum/#!searchin/julia-users/OpenCV/julia-use

> >>> rs/81V5zSNJY3Q/DRUT0dR2qhQJ> ,3

> >>> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-u
> >>> sers/iUPqo8drYek/pUeHECk91AQJ> ,4
> >>> <https://groups.google.com/forum/%23!searchin/julia-users/OpenCV/julia-u

> >>> sers/6QunG66MfNs/C63pDfI-EMAJ> ) and wide application of OpenCV for fast

> >>> real-time computer vision applications, I set myself to put together a
> >>> simple interface for OpenCV in Julia. Coding in Julia and developing
> >>> the interface between C++ and Julia has been a lot of fun!
> >>>

> >>> OpenCV.jl aims to provide an interface for OpenCV <http://opencv.org/>

> >>> computer vision applications (C++) directly in Julia

> >>> <http://julia.readthedocs.org/en/latest/manual/>. It relies primarily
> >>> on Keno´s amazing Cxx.jl <https://github.com/Keno/Cxx.jl>, the Julia

> >>> C++ foreign function interface (FFI). You can find all the information
> >>> on
> >>> my package at https://github.com/maxruby/OpenCV.jl.
> >>>
> >>> You can download and run the package as follows:
> >>>

> >>> Pkg.clone("git://github.com/maxruby/OpenCV.jl.git")using OpenCV

> >>>
> >>>
> >>> For MacOSX, OpenCV.jl comes with pre-compiled shared libraries, so it is
> >>> extremely easy to run. For Windows and Linux, you will need to first
> >>> compile the OpenCV libraries, but this is well documented and links to
> >>> the
> >>> instructions for doing so are included in the README.md file.
> >>>
> >>> The package currently supports most of OpenCV´s C++ API; however, at
> >>> this point I have created custom wrappings for core, imgproc, videoio
> >>> and highgui modules so that these are easy to use for anyone.
> >>>
> >>> The package also demonstrates/contains
> >>>

> >>> - preliminary interface with the Qt GUI framework (see imread() and
> >>> imwrite() functions)
> >>> - thin-wrappers for C++ objects such as std::vectors, std::strings
> >>> - conversion from Julia arrays to C++ std::vector
> >>> - conversion of Julia images (Images.jl) to Mat (OpenCV) - though

[Max Suster]

Its an exact comparison side-by-side of the same code, and actually already tested by others in the OpenCV forum.

The Mat/UMat image is available for display with imshow -- the step "imshow("edges", gray);" in both cases -- which is how the test was set up.

The main point is the time it takes to complete the entire process and the fact that cvtColor with OpenCL can generate an image for viewing much more quickly.

I never intended to be misleading, and I am sorry that you interpret it this way. 

Max

[Simon Danisch]

Under the premise, that everything can be done on the GPU up to a certain point, transfer times shouldn't be included in the benchmark, as it wouldn't be necessary to have the data available to the CPU.

For all simple image transformation, OpenCL should be strictly faster. ( obviously not if you have a Intel Xeon E5-2699 and a weak graphic card)

Could it be that OpenCL automatically accelerates even normal Mats with the GPU? 

This might be the reason why the initial cvtColor is so slow on the CPU, as it includes transferring the data to the CPU.

Other reason could be, that OpenCV does transfers and copies even with UMat.

I'm not sure what's going on here, but I might try to build OpenCV with OpenCL tomorrow and see for myself.

Have you tried to explicitly turn OpenCL on and off with OPENCV_OPENCL_RUNTIME?

[Tim Holy]

Now that you posted the code (along with timing markers), it's much clearer.
Your timings are just for the computation, and don't include transfer time.

My "misleading" statement was not intended as an accusation :-), merely trying
to help explain the gaussian blur result: if you included transfer time in one
but not the other, that might explain it. But from your posted code, it's
clear that wasn't the problem.

--Tim

> > > 2014-12-08 17:39 GMT+01:00 Max Suster <mxss...@gmail.com <javascript:>>:

[Max Suster]

Thanks Tim (I did not mean to be defensive). 

I totally agree that it is best to get to the bottom of the issue so that we can get it work!

The point I was trying to make more generally is that there does seem to be quite a bit of variability in the gains from 

OpenCL-acceleration at least with OpenCV. It would be nice to figure out how and which bits can truly be accelerated most efficiently 

and which may not be worth spending a lot more energy on. . .

Max

[Simon Danisch]

Strictly faster is probably a little bit exaggerated, but simple image manipulations fit the GPU very well. This should be valid for all algorithms which can be massively parallelized  and don't use scatter writes or reads.

So if you have decent video card, the cpu should have a hard time to match the performance.

I'm not sure about the transfers, as OpenCV might actually do transfers even in the UMat case... It's not that obvious how they manage their memory.

[Simon Danisch]

To give it some more underpinning, I searched for a benchmark.

I only found one for cuda so far from 2012: http://www.timzaman.com/?p=2256

Results in a table:

https://docs.google.com/spreadsheet/pub?key=0AkfBSyx6TpqMdDFnclY5UjlyNmVsSnhGV0hscnJQcVE&output=html

Resulting graph:

I think the scale is speedup from using the gpu, so the cpu is always one.

[Tim Holy]

Certainly it's possible that the quality of the kernels varies. The blur case
is presumably harder to write a good kernel for and more subject to the chosen
schedule, because it has to access memory non-locally. The grayscale
conversion is pointwise and so is fairly trivial. But I would have expected
the blur kernel to be highly optimized; I agree it's a puzzling result.

But that table that Simon linked suggests it should be faster, so I wonder if
there's something fishy going on.

--Tim

[Max Suster]

Thanks!  That is very useful.   

Why is cvColor so slow in the Intel tests?

I notice they used a 4000x4000, CV_8UC4  (Uint8, 4 channels) for the GaussianBlur test!

[Simon Danisch]

That's a good point! Are you using CV_8UC3?

Video cards like everything that has 32 bits, so if you use RGB (3*8=24) depending on how OpenCV optimizes things, it could mean quite a slow down for the UMat case.

[Max Suster]

Yes, I am using CV_8UC3 for my tests (and dim  512x512 or 1000x1000).

I think the most likely explanation for my results is that while OpenCV comes with OpenCL enabled in the makefile

-D WITH_OPENCL=ON

 the flags for including OpenCL AMD FFT and AMD BLAS libraries are OFF by default:

-D WITH_OPENCLAMDFFT=OFF 
-D WITH_OPENCLAMDBLAS=OFF 

I suppose that the consequence is likely that the gains with OpenCL are inconsistent at best on OSX.

Unfortunately, these libraries are not available for OSX right off the shelf, and it seems that I need to build them first. 

http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-math-libraries/

They are readily available for Windows and Linux.   I found  a post in which someone managed to get them working with a hack. 

I need to this investigate further.  . .