MIPI Cameras on Raspberry Pi 5?

[vespaman]

I am doing some work with some camera sensors connected to Pi5, and thought I should try if OpenPnP would detect them camera sensors (if it would work, it would be a nice solution for my side project "label-o-matic", which is a openpnp setup oriented for putting on labels on a small plastic housing).

Now, it did not work with OpenPnpCaptureCamera, same problem that has been reported by Pi4 users in the past.

But maybe the gstreamer camera can work?

Is there someone that can give some light to typical settings for the "pipeline launch string"? 

Or perhaps, someone has a working setup with Pi and the MIPI camera(s)?

  - Micael

[Ian Arkver]

I don't have an RPi5 to test with, but looking here:
https://forums.raspberrypi.com/viewtopic.php?t=360578
it seems something like:
     libcamerasrc ! video/x-raw, format=RGBx, width=1920, height=1080, framerate=30/1

might work.

As always with gstreamer, experiment with gst-launch-1.0 and display to your monitor, then use the front part that works.
Do all the usual RPi camera tests first, then get it going with GStreamer outside OpenPnP, then move it over.

Regards,
Ian

[vespaman]

Hi Ian,

So that line you are suggesting isn't complete, gives 'no element "video" ' if I run it from command line, but maybe gstreamer camera appends the pipe sink? Anyway, if I put it into openpnp gstreamer camera, i get absolutely no response what so ever. And this is true whatever I put into that Pipeline launch string.

I suspect there are prerequisites in OpenPnP about the format and codec, or do you mean that x-raw should be enough? Not mjpeg or jpeg?

It was a few years since I last used gst-launch, and libcamera also puts its limitations into play.

In the 'Pipeline launch string', should I include 'gst-launch-1.0' or only the arguments?

Not sure why, but I get absolutely no error output in the openpnp start console either.

I have also learned that there might be a path through opencv camera as well, but that probably has similar problems.

 - Micael

[Ian Arkver]

Hi Micael,

The string given to GstreamerCamera is parsed in the same way as gst-launch does. It doesn't need the "gst-launch-1.0" bit itself. Also GstreamerCamera does append an appsink at the end, so your pipeline snippet should end with something that provides video/x-raw of the correct format.

I'd definitely recommend getting it going outside OpenPnP first, using gst-launch and a sink like autovideosink. You can enable a lot more debug with gst-launch to see what's happening if you need to. Read up on GST_DEBUG here perhaps:
https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html

But yeah, GstreamerCamera isn't very easy to use and does assume some familiarity with Gstreamer and ideally a working pipeline already in hand.

Good luck!
Ian

[vespaman]

Thanks Ian,

Just knowing that it should work is a good start. :-) I'll do some more tests, and hopefully get the hang of what is the missing link!

 -  Micael

[vespaman]

OK, so finally I had some time to get this working (by creating a C program to get more error information).

One thing which might be useful to know for future adopters; if the Gstreamer pipeline is changed, OpenPnP needs to be restarted, for it to take effect.

Then, for a raspberry pi 5, and two imx290 sensors, both on MIPI;

libcamerasrc camera-name = "/base/axi/pcie@120000/rp1/i2c@88000/imx290@1a" ! video/x-raw,colorimetry=bt709,format=NV12,width=1920,height=1080,framerate=120/1 ! videoconvert

libcamerasrc camera-name = "/base/axi/pcie@120000/rp1/i2c@80000/imx290@1a" ! video/x-raw,colorimetry=bt709,format=NV12,width=1920,height=1080,framerate=60/1 ! videoconvert

In my case, I have slowed down the down cam to 60FPS, to lessen the stress on the rpi - the second row above. Not sure if it makes any difference in load though, have not yet setup my OpenPnP machine on this Pi yet.

Also - in standard Pi kernel, imx290 is limited to 60FPS, so start with lower FPS if you are unsure.

However, there seems to be a bug with GstreamerCamera - if you will try to check "Suspend during tasks?" (which seems useful on Raspberry), you will immediately receive a SIGSEGV as a "thank you".

 - Micael

[Ian Arkver]

Hi Micael,

Well done on getting an RPi camera going. Not sure what's up with the segv on the Preview Suspend. Looking at the source this prevents captureTransformed and notifyCapture from being called. The GStreamer pipeline won't be paused, so the appsink will still be receiving images. Hopefully it will be dropping them if they're not being taken on the Java side, but perhaps not.

To be honest, it's a very long time since I looked at this code, and I don't have an RPi to test with, so can't really debug it. I'll have a think, but can't make promises.

Regards,
Ian

[vespaman]

Hi Ian,

No worries. 

If you give it a think, it might help to know that the error is happening when pressing 'Apply'.

I'm guessing (but I have not tried), that this is general to GStreamer cameras, rather than Pi.

** (gst1-java-core:3492): CRITICAL **: 10:18:08.575: void GstLibcameraSrcState::requestCompleted(libcamera::Request*): assertion 'wrap->request_.get() == request' failed
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ffe7299bc0c, pid=3492, tid=3578
#
# JRE version: OpenJDK Runtime Environment (17.0.13+11) (build 17.0.13+11-Debian-2deb12u1)
# Java VM: OpenJDK 64-Bit Server VM (17.0.13+11-Debian-2deb12u1, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-aarch64)
# Problematic frame:
# C  [libgstreamer-1.0.so+0x8bc0c]  gst_mini_object_unref+0x1c
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/micke/openpnp/hs_err_pid3492.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.debian.org/openjdk-17
#
Aborted

 -  Micael

[Ian Arkver]

Afraid I can't reproduce the segv. Fired the machine up again and noticed I had "Suspend during tasks" set for both of them. Tried toggling it off and on again (with apply) - all OK. I did notice that the pipeline seems to be restarted when making the change, so perhaps it's something to do with libcamerasrc. Just guessing without debug. You could try running OpenPnP with the GST_DEBUG env variable set to something suitable, but debugging gstreamer can be a bit of a dark art.

Or you could try setting suspend-preview-in-tasks="true" in the machine.xml and opening OpenPnP with it already set. It might be the pipeline restart rather than the actual setting.

Test was with a fresh Java 23 compile, but I haven't rebased my branch since the PR to merge GStreamerCamera so the rest of OpenPnP is quite old.

Good luck!
Ian