BoofCV on Raspberry with Pi Cam

Thomas Zwicker

unread,

Apr 24, 2020, 11:04:32 AM4/24/20

to BoofCV

Hi Peter & community

I am trying to get BoofCV going on Raspberry, with RasPi Cam (onboard; non-USB) and using the minimum set of libraries.

Final goal is to create a simple robot with basic vision, programmed in Java.

Questions so far:

- is there a tutorial to setup up the Raspberry Pi and Java project with BoofCV somewhere?

- specifically, which are the Maven dependencies to start with an absolute minimum, including access (still or video) to the Pi Cam?

- and furthermore, what needs to be installed on the Raspberry Pi aside from the JARs (given e.g. "raspistill" is working fine)?

So far I managed to

- setup Eclipse & Maven, including the recommended newer version of BridJ as from http://webcam-capture.sarxos.pl/

- automatically deploy from Mac to Raspberry and start to run the class with the BoofCV "WebCam Capture Example" https://boofcv.org/index.php?title=Example_Webcam_Capture

- it exits on init with the messages

"...OpenIMAJGrabber.so : libjpeg.so.8: cannot open shared object file: No such file or directory"

"...Caused by: java.lang.RuntimeException: Library 'OpenIMAJGrabber' was not loaded successfully"

- it uses the dependencies boofcv-core; boofcv-swing;boofcv-WebcamCapture (artifacts IDs, versions 0.35) to build without error

Furthermore

- I tried to run the code on Mac but "Webcam.getDefault();" returns NULL; would be great to use the same code on Mac straight from Eclipse (MacOS Catalina) and Raspi

Thanks for your support in advance!

Thomas

Thomas Zwicker

unread,

Apr 25, 2020, 4:41:34 PM4/25/20

to BoofCV

Here is the full log:

# BridJ: dlopen error when loading /tmp/BridJExtractedLibraries8123947184077079536/OpenIMAJGrabber.so : libjpeg.so.8: cannot open shared object file: No such file or directory

java.lang.RuntimeException: Library 'OpenIMAJGrabber' was not loaded successfully from file '/tmp/BridJExtractedLibraries8123947184077079536/OpenIMAJGrabber.so'

at org.bridj.BridJ.getNativeLibrary(BridJ.java:1072)

at org.bridj.BridJ.getNativeLibrary(BridJ.java:1049)

at org.bridj.BridJ.getNativeLibrary(BridJ.java:602)

at org.bridj.cpp.CPPRuntime.newCPPInstance(CPPRuntime.java:771)

at org.bridj.cpp.CPPRuntime$CPPTypeInfo.initialize(CPPRuntime.java:1022)

at org.bridj.cpp.CPPRuntime$CPPTypeInfo.initialize(CPPRuntime.java:904)

at org.bridj.CRuntime$CTypeInfo.initialize(CRuntime.java:271)

at org.bridj.BridJ.initialize(BridJ.java:1128)

at org.bridj.NativeObject.<init>(NativeObject.java:50)

at org.bridj.StructObject.<init>(StructObject.java:46)

at org.bridj.cpp.CPPObject.<init>(CPPObject.java:55)

at com.github.sarxos.webcam.ds.buildin.natives.OpenIMAJGrabber.<init>(OpenIMAJGrabber.java:64)

at com.github.sarxos.webcam.ds.buildin.WebcamDefaultDriver$WebcamNewGrabberTask.handle(WebcamDefaultDriver.java:55)

at com.github.sarxos.webcam.WebcamProcessor$AtomicProcessor.run(WebcamProcessor.java:81)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)

Exception in thread "main" com.github.sarxos.webcam.WebcamException: java.util.concurrent.ExecutionException: com.github.sarxos.webcam.WebcamException: Cannot execute task

at com.github.sarxos.webcam.WebcamDiscoveryService.getWebcams(WebcamDiscoveryService.java:124)

at com.github.sarxos.webcam.Webcam.getWebcams(Webcam.java:893)

at com.github.sarxos.webcam.Webcam.getDefault(Webcam.java:956)

at com.github.sarxos.webcam.Webcam.getDefault(Webcam.java:933)

at com.github.sarxos.webcam.Webcam.getDefault(Webcam.java:911)

at boofcv.io.webcamcapture.UtilWebcamCapture.openDefault(UtilWebcamCapture.java:38)

at simplyPiVision.simpleTestRun.main(simpleTestRun.java:31)

Caused by: java.util.concurrent.ExecutionException: com.github.sarxos.webcam.WebcamException: Cannot execute task

at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)

at com.github.sarxos.webcam.WebcamDiscoveryService.getWebcams(WebcamDiscoveryService.java:116)

... 6 more

Caused by: com.github.sarxos.webcam.WebcamException: Cannot execute task

at com.github.sarxos.webcam.WebcamProcessor$AtomicProcessor.process(WebcamProcessor.java:72)

at com.github.sarxos.webcam.WebcamProcessor.process(WebcamProcessor.java:140)

at com.github.sarxos.webcam.WebcamTask.process(WebcamTask.java:46)

at com.github.sarxos.webcam.ds.buildin.WebcamDefaultDriver$WebcamNewGrabberTask.newGrabber(WebcamDefaultDriver.java:45)

at com.github.sarxos.webcam.ds.buildin.WebcamDefaultDriver.getDevices(WebcamDefaultDriver.java:117)

at com.github.sarxos.webcam.WebcamDiscoveryService$WebcamsDiscovery.call(WebcamDiscoveryService.java:36)

at com.github.sarxos.webcam.WebcamDiscoveryService$WebcamsDiscovery.call(WebcamDiscoveryService.java:26)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.lang.RuntimeException: Failed to allocate new instance of type class com.github.sarxos.webcam.ds.buildin.natives.OpenIMAJGrabber

at org.bridj.cpp.CPPRuntime.newCPPInstance(CPPRuntime.java:812)

at org.bridj.cpp.CPPRuntime$CPPTypeInfo.initialize(CPPRuntime.java:1022)

at org.bridj.cpp.CPPRuntime$CPPTypeInfo.initialize(CPPRuntime.java:904)

at org.bridj.CRuntime$CTypeInfo.initialize(CRuntime.java:271)

at org.bridj.BridJ.initialize(BridJ.java:1128)

at org.bridj.NativeObject.<init>(NativeObject.java:50)

at org.bridj.StructObject.<init>(StructObject.java:46)

at org.bridj.cpp.CPPObject.<init>(CPPObject.java:55)

at com.github.sarxos.webcam.ds.buildin.natives.OpenIMAJGrabber.<init>(OpenIMAJGrabber.java:64)

at com.github.sarxos.webcam.ds.buildin.WebcamDefaultDriver$WebcamNewGrabberTask.handle(WebcamDefaultDriver.java:55)

at com.github.sarxos.webcam.WebcamProcessor$AtomicProcessor.run(WebcamProcessor.java:81)

... 3 more

Caused by: java.lang.RuntimeException: Library 'OpenIMAJGrabber' was not loaded successfully from file '/tmp/BridJExtractedLibraries8123947184077079536/OpenIMAJGrabber.so'

at org.bridj.BridJ.getNativeLibrary(BridJ.java:1072)

at org.bridj.BridJ.getNativeLibrary(BridJ.java:1049)

at org.bridj.BridJ.getNativeLibrary(BridJ.java:602)

at org.bridj.cpp.CPPRuntime.newCPPInstance(CPPRuntime.java:771)

... 13 more

Peter

unread,

Apr 25, 2020, 7:28:40 PM4/25/20

to BoofCV

Which OS on Raspberry PI are you using and which board? It's been about a year but I remember the Raspberry PI camera not working right out of the box. Plus there is an issue where the default JVM they install isn't optimized for ARM. I'll take my Raspberry PI out of storage and attempt to document this.

Thomas Zwicker

unread,

Apr 26, 2020, 3:33:31 AM4/26/20

to BoofCV

The environment is

Raspberry Pi 4 B board, Pi Cam

processors : 4
model name : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 108.00
Features : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm crc32
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 3

Hardware : BCM2835
Revision : c03111
Serial : 1000000071e4fc6f
Model : Raspberry Pi 4 Model B Rev 1.1
Pi cam 0x48435045mmal

Linux 4.19.97-v7l+ armv7l

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian

openjdk version "11.0.6" 2020-01-14

OpenJDK Runtime Environment (build 11.0.6+10-post-Raspbian-1deb10u1)
OpenJDK Server VM (build 11.0.6+10-post-Raspbian-1deb10u1, mixed mode)

/dev/video0 is available (as required by https://github.com/sarxos/webcam-capture/wiki/How-To-Configure-Raspberry-Pi)

Peter A

unread,

Apr 26, 2020, 8:25:11 PM4/26/20

to boofcv

https://boofcv.org/index.php?title=Raspberry_PI

try those instructions. Suggestions for improving them are welcomed! I tested it on my Raspberry PI 3B+ with whatever version of Raspbian was on it last year. I'm using Webcam Capture to grab the webcam and it's a lot slower than I remember it. However last time I used a library that I wrote and isn't trivial to use.

--
You received this message because you are subscribed to the Google Groups "BoofCV" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boofcv+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/04c1cf30-d40a-4da6-8306-476df513d494%40googlegroups.com.

--

"Now, now my good man, this is no time for making enemies." — Voltaire (1694-1778), on his deathbed in response to a priest asking that he renounce Satan.

Thomas Zwicker

unread,

Apr 27, 2020, 8:49:36 AM4/27/20

to BoofCV

Hi Peter

Ok, works now. Key knowledge I gained after going through your infos: the driver for Raspberry is highly specific. After adding the Raspberry-specific driver to the POM, my own test project also worked. See POM additions below.

Now one question remains: is this a performant driver? How will the whole setup perform in terms of FPS analyzed (processing images from the web cam to control a robot)? If you have hints, very appreciated (optimized JRE installed).

Additions to the POM:

<id>Sonatype OSS Snapshot Repository</id>

<url>http://oss.sonatype.org/content/repositories/snapshots</url>

</repository>

</repositories>

...

<groupId>com.github.sarxos</groupId>

<artifactId>webcam-capture-driver-raspberrypi</artifactId>

<version>0.3.13-SNAPSHOT</version>

</dependency>

Also ensure that the Maven Build plugin is set to "useUniqueVersions=false", as the Raspberry driver is referred to as SNAPSHOT

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-jar-plugin</artifactId>

<useUniqueVersions>false</useUniqueVersions>

...

Peter A

unread,

Apr 27, 2020, 11:18:39 AM4/27/20

to boofcv

I updated the webpage to include links to two alternative java libraries. The example project has also been updated to use a faster driver. I get 10 FPS on my system now. before it was much slower. Could you run it on your raspberry pi and past the output? Just a few lines is all that will be needed. Just want to see how fast it's running.

My suspicion is that we can do better than 10 FPS for image capture just will take a bit of work to figure out how to do that. For computer vision algorithms how fast it will run really depends on which algorithm you're running. I would shoot to process images that are fairly small. 640x480 or 320x240, depending on what you're trying to do and the minimum FPS. While I was working on this I also played around with doing visual odometry on a RPI. I was able to tune it (by sacrificing some stability) to process a stereo pair of images in about 50 to 100 ms.

--

You received this message because you are subscribed to the Google Groups "BoofCV" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boofcv+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/bd32932d-400b-4840-952d-38afd3300c62%40googlegroups.com.

Thomas Zwicker

unread,

Apr 28, 2020, 5:19:04 AM4/28/20

to BoofCV

Switching the driver basically halves the image processing time, RaspividDriver being the faster one (from webcam-capture-driver-raspberrypi 0.3.13-SNAPSHOT), as you pointed out. However, first run on Raspi 4 was 10 FPS roughly, too.

Given the "desktop performance difference" as reported at https://magpi.raspberrypi.org/articles/raspberry-pi-4-vs-raspberry-pi-3b-plus I don't see why Raspi 4 does not reach a better performance.

It seems the RAW image driver could be much faster given some people build high speed cams with it https://blog.robertelder.org/recording-660-fps-on-raspberry-pi-camera/ Tried to install libjepg8-dev and using a default driver from Webcam Capture for the test app (removed the static-method setting RaspiXXX Drivers), which resulted in accessing "Webcam mmal swrvice 16.1". However, no performance improvement.

Would be interesting how we could speed up frame rates - and also how fast the Python processing times for this task would be.

Thomas Zwicker

unread,

Apr 28, 2020, 8:29:51 AM4/28/20

to BoofCV

Added another significant performance improvement by using OpenImaJ

Change JAVA main:

import com.github.sarxos.webcam.ds.openimaj.*;

...

static {

Webcam.setDriver(new OpenImajDriver());//50ms 640x480

}

Change Maven POM to add repository & dependency for BridJ (note the version 0.7-20140918-2 also should help Mac users running it locally)

<id>bridjhack</id>

<url>http://maven.ecs.soton.ac.uk/content/repositories/thirdparty/</url>

</repository>

<groupId>com.nativelibs4java</groupId>

<artifactId>bridj</artifactId>

</dependency>

Add the Driver

<groupId>com.github.sarxos</groupId>

<artifactId>webcam-capture-driver-openimaj</artifactId>

<!-- version>0.3.13-SNAPSHOT</version / works too, use Sonatype SNAPSHOT repository >

</dependency>

Thomas Zwicker

unread,

Apr 28, 2020, 9:35:25 AM4/28/20

to BoofCV

OpenImaJ with BoofCV got around 30 FPS on Raspi 4 with 640x480 resolution and a 160 degree Pi Cam, while also displaying the result in the Java window. Cool!

Peter

unread,

Apr 28, 2020, 11:48:23 AM4/28/20

to BoofCV

I just get a "Segmentation fault" on my device. First it complained libjpeg wasn't installed. So I installed it. Which version did you install?

ldconfig -p | grep jpeg

libmjpegutils-2.1.so.0 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libmjpegutils-2.1.so.0

libjpeg.so.62 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libjpeg.so.62

libjpeg.so.8 (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libjpeg.so.8

libjpeg.so (libc6,hard-float) => /usr/lib/arm-linux-gnueabihf/libjpeg.so

Thomas Zwicker

unread,

Apr 30, 2020, 11:10:53 AM4/30/20

to BoofCV

"apt-get install libjpeg8-dev" did the trick.

Btw: on 640x480 the Raspi actually could do more than 30fps, if the camera is configured accordingly. Thus I raised an issue with Webcam Capture to change the OpenImaJ driver code to allow changing the camera FPS. Furthermore I compared image grabbing time between OpenImaJ and the Raspvid drivers for 1080p, where OpenImaJ is twice as fast. Therefore I am highlz interested that OpenImaJ works well :) Side benefit: it runs on earlier Mac OS out-of-the-box, no need to change a line of code between Raspberry and Mac OS (however, they need to update as the permissioning has changed for Mac OS Catalina)

Peter A

unread,

May 5, 2020, 1:28:50 PM5/5/20

to boofcv

My Raspberry PI 4 just arrived last night. Haven't had too much time to play with it yet. Does seem to be a big improvement over the PI 3. They also might have fixed the issue of installing the wrong java in Raspbian. I didn't notice any speed difference compared to the custom one I downloaded.

--

You received this message because you are subscribed to the Google Groups "BoofCV" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boofcv+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/a58e71b8-682d-4e2e-8357-a5d352ea31b2%40googlegroups.com.

Thomas Zwicker

unread,

May 11, 2020, 4:06:36 PM5/11/20

to BoofCV

Cool. I tried FPS setting. Turns out with some tweaking of Webcam Capture OpenImaJ Driver it's possible to change FPS - went from 25 FPS default to 50 FPS in a low-light environment, turning the frame very dark. It worked :-) Now tested face detection with BoofCV and works nice, even in low light with a wide angle (160 deg) camera.

Peter A

unread,

May 11, 2020, 5:02:31 PM5/11/20

to boofcv

Can you post a link to the instructions you used? There's a face detector in BoofCV? It's been on the to do list for a while.

In other Raspberry PI news, BoofCV can now read MP4 videos even if the FFMPEG driver doesn't have arm 32 support. Got JCodec to play nice with BoofCV so now there's an integrated Java MP4 codec.

On Mon, May 11, 2020 at 1:06 PM Thomas Zwicker <tomzw...@gmail.com> wrote:

Cool. I tried FPS setting. Turns out with some tweaking of Webcam Capture OpenImaJ Driver it's possible to change FPS - went from 25 FPS default to 50 FPS in a low-light environment, turning the frame very dark. It worked :-) Now tested face detection with BoofCV and works nice, even in low light with a wide angle (160 deg) camera.

--

You received this message because you are subscribed to the Google Groups "BoofCV" group.
To unsubscribe from this group and stop receiving emails from it, send an email to boofcv+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/2b70118c-b610-4daa-9a7b-f54a91f89c7f%40googlegroups.com.

Thomas Zwicker

unread,

May 12, 2020, 3:06:47 AM5/12/20

to BoofCV

The FaceDetector is actually OpenImaJ, not BoofCV :-) As I use the OpenImaJ Driver this is an easy step to take.

I used below Git to start

https://github.com/Fabryprog/cv-faces-example/blob/master/src/main/java/org/fabryprog/cv/faces/example/App.java

It bases on

org.openimaj.image.processing.face.detection.DetectedFace;

org.openimaj.image.processing.face.detection.HaarCascadeDetector;

On Monday, May 11, 2020 at 11:02:31 PM UTC+2, Peter wrote:

Can you post a link to the instructions you used? There's a face detector in BoofCV? It's been on the to do list for a while.

In other Raspberry PI news, BoofCV can now read MP4 videos even if the FFMPEG driver doesn't have arm 32 support. Got JCodec to play nice with BoofCV so now there's an integrated Java MP4 codec.

On Mon, May 11, 2020 at 1:06 PM Thomas Zwicker <tomzw...@gmail.com> wrote:

Cool. I tried FPS setting. Turns out with some tweaking of Webcam Capture OpenImaJ Driver it's possible to change FPS - went from 25 FPS default to 50 FPS in a low-light environment, turning the frame very dark. It worked :-) Now tested face detection with BoofCV and works nice, even in low light with a wide angle (160 deg) camera.

--
You received this message because you are subscribed to the Google Groups "BoofCV" group.

To unsubscribe from this group and stop receiving emails from it, send an email to boo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/2b70118c-b610-4daa-9a7b-f54a91f89c7f%40googlegroups.com.

Peter A

unread,

May 14, 2020, 12:41:03 AM5/14/20

to boofcv

How do you change the FPS on the driver? I didn't know that Webcam Capture let you do that.

To unsubscribe from this group and stop receiving emails from it, send an email to boofcv+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/boofcv/275d154e-a6b9-4f1a-a8aa-eebd2091f633%40googlegroups.com.

Reply all

Reply to author

Forward

BoofCV on Raspberry with Pi Cam - Setting up

Thomas Zwicker

Thomas Zwicker

Peter

Thomas Zwicker

Peter A

Thomas Zwicker

Peter A

Thomas Zwicker

Thomas Zwicker

Thomas Zwicker

Peter

Thomas Zwicker

Peter A

Thomas Zwicker

Peter A

Thomas Zwicker

Peter A