DirectBuffer / OpenCV / JavaCV / JavaFX

[Robert Ladstätter]

hi,

Recently JavaFX got a new API in it's PixelBuffer Implementation which allows to reuse native memory for displaying images / videos.

I tried to get it to run with JavaCV, but I learned that JavaFX wants an array with 4 channels, rgb and alpha, whereas my quick tries with OpenCVGrabber I only got RGB values (not sure if the ordering matches, at least no alpha value of that I'm sure).

Is there a way to get an array with 4 byte per pixel easily by using some clever parameters? Of course I want to circumvent pixel array copying that is the reason I'm asking.

I'm sure this is a common question, but obviously my google fu is too weak.

any tipps? 

thank you 

robert

[Samuel Audet]

There are many ways to do the conversion. We can do it manually as
implemented by Johan Vos in JavaFXFrameConverter:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/JavaFXFrameConverter.java

But for performance I recommend to use simply cvtColor() with
CV_BGR2RGBA. I don't think OpenCV supports anything else, no.

[Samuel Audet]

Hi,

Your results look good, valid, and correct, yes. Thank you for spending
time on this!

JavaFX is obviously faster than Swing, that's for sure. Another reason
JavaFXFrameConverter is slower is because it doesn't use cvtColor() to
avoid a dependency on OpenCV, since some people use JavaCV only with
FFmpeg, etc. Using vectorized native code to do the conversion typically
results in faster execution. You should also be able to get even more
speed by reusing the buffers you're currently recreating for each image.

When you have a URL for your blog post, please feel free to add it here:
https://github.com/bytedeco/javacv/wiki/Other-Resources

BTW, it's possible to avoid "downloading the Internet" pretty easily:
https://github.com/bytedeco/javacpp-presets/wiki/Reducing-the-Number-of-Dependencies

And don't worry about broadcasting incorrect information on the mailing
list. Anything wrong gets corrected in threads, so no harm is done :)

Samuel

On 5/11/20 3:21 AM, Robert Ladstätter wrote:
> Hello Mr. Audet,
>
> Finally I had time to get back to this, thank you very much for pointing
> me to the right direction concerning JavaFX and JavaCV.
>
> I’ve created a small comparison about different approaches to show a
> webcam image stream with JavaCV / OpenCV:
>
> - swing
> - ‘classic’ JavaFX
> - JavaFX with DirectBuffer
>
> my comparisons show the last one is the fastest one, I would be
> interested if you think this could be valid, I would also like to
> create a small blog post about it. Naturally, I would like to avoid to
> shout out wrong information to the internet ;-(
>
> It would be great if you could review the project on github. It is a
> trivial setup and should be callable like this:
>
> https://github.com/rladstaetter/javacv-webcam
>
> mvn package
>
> java -jar target/javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar
> java -jar target/javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar classic
> java -jar target/javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar swing
> java -Dsun.java2d.opengl=True -Dsun.java2d.xrender=True -jar target/javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar swing
>
> (I saw somewhere that one should call the app with those parameters to
> get full performance, I didn’t notice any difference … )
>
> It would be cool if my findings are correct.
>
> In any case: thank you for your work on JavaCV, it is a mind blowing
> library!
>
> best,
>
> robert
>
>
>
>
>
> here is a draft of my blog post:
>
> Finally I found some time to write about JavaFX and JavaCV.
>
> JavaCV is a very cool library <https://github.com/bytedeco/javacv> which
> empowers Java users to access native libraries like OpenCV or FFmpeg and
> many others.
>
> Among other things, it prepackages OpenCV with maven, as such you can
> jumpstart in OpenCV development in mere minutes. Even if it downloads
> the internet without special tuning, it is as far as I know the simplest
> and fastest way to try out computer vision algorithms on the Java platform.
>
> As a bonus, it enables also writing applications for Android, not to
> mention the usual desktop platforms like Windows / Mac and Linux.
>
> In the past I've created some applications using OpenCV and it's
> official API, which is also available via JavaCV as far as I know. But
> JavaCV additionally provides API's which give you an added value as it
> makes using OpenCV APIs more idiomatic for a Java user, which is always
> a good thing for adoption.
>
> The motivation for devoting some time to JavaCV was to see if I could
> take advantage of the new PixelBuffer API which can use since JavaFX13
> native arrays directly.
>
> It seems clear that this should boost performance considerably and
> improves framerates for applications like my SudokuFX project. I've
> created a proof of concept
> <https://github.com/rladstaetter/javacv-webcam>  which compares
> different ways of using a webcam image stream based on the OpenCV stack.
>
> In this project there are three implementations given:
>
> * a swing based, officially endorsed way to show a webcam / image
> * the classic JavaFX variant
> * the DirectBuffer approach
>
>
> I've tried to measure how fast each approach is, and on my machine it
> turns out that the last one using JavaFX with DirectBuffer is the
> fastest way to show a webcam image stream. I hope it wasn't just wishful
> thinking, but by comparing all three approaches, the JavaFX DirectBuffer
> approach feels snappier than the other two, my naive performance
> measurements also point in this direction.
>
> On my machine, I measured how long it took to grab and display an image,
> here are the results, which should be taken with a grain of salt, try it
> out on your machine to see if you get similiar results.
>
> Here is the output for running the three different versions, with the
> swing approach being the slowest one (?).
>
> home:target lad$ java -jar
> javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar
> Mean: 2238831
> Mean: 2176064
> Mean: 2160973
> Mean: 2158766
> Mean: 2148553
>
> home:target lad$ java -jar
> javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar classic
> Mean: 9508516
> Mean: 9461551
> Mean: 9423204
> Mean: 9419029
> Mean: 9415144
>
> home:target lad$ java -jar
> javacv-webcam-2020.2-SNAPSHOT-jar-with-dependencies.jar swing
> Mean: 35786482
> Mean: 35410293
> Mean: 35220229
> Mean: 35092391
> Mean: 35030641
>
> If those measurements do the right thing and I'm not mistaken, the
> DirectBuffer approach outperforms the other two by far. I'm not sure
> there are tricks to improve performance even more, but it is likely ...
> check out the repo :)
>
> I would like to thank Samuel Audet who does a great job maintaining
> JavaCV and who also helped me get my webcam image to fit to JavaFX's
> PixelBuffer. <https://groups.google.com/forum/#!topic/javacv/uTpd9FGRvyE>

[Robert Ladstätter]

Thanks again!

I've published my findings on github here:

https://github.com/rladstaetter/javacv-webcam

and wrote a little bit about here:

https://ladstatt.blogspot.com/2020/05/webcam-image-stream-with-javacv.html

Like you've mentioned in your reply, reusing the byte buffer makes another major impact in performance (of course!).

I also linked the above github repo on your wiki.

robert

[Samuel Audet]

Looks great! Thanks a lot for the contribution

Samuel