Dual streaming cameras from raspberry pi

[Paul Jones]

Hi all,

 

Thought I might share some progress of something I’ve been working on – network streaming cameras using a rpi. I’ve had all sorts of drams trying to get this working and nearly gave up, but I think it might almost be usable now.

I tried about every streaming camera project I could find, but the results were terrible – either latency in the seconds (5 seconds was the worst) or the crappy ffmpeg integration in opencv causing issues (rtsp streaming of h264 video is completely broken). Eventually I accidently stumbled across a solution that works.

 

First thing is to modify openpnp so the OpenCvCamera can use a connection string rather than just a usb device number.

See https://github.com/PeeJay/openpnp/tree/rtsp for my code. (I’ll send it to Jason once it’s good enough)

 

The version of OpenCv that is bundled with Openpnp doesn’t have support for ffmpeg, but on windows the binaries are compiled with support but missing the dll files required. Putting the file opencv_ffmpeg2411_64.dll (or 32 bit version) in the main project directory seems to work somehow. On linux my recompiled versions didn’t work yet, but using the Ubuntu 16.04 opencv-java package did work (once the project was reconfigured accordingly). You can see my efforts to modify the openpnp opencv java bindings package at https://github.com/PeeJay/opencv-java/tree/update

 

To stream video from the rpi the following commands are used:

raspivid -t 0 -n -w 1600 -h 1200 -hf -ih -fps 10 -cd MJPEG -o - | nc -k -l 8080

avconv -f video4linux2 -input_format mjpeg -video_size 960x720 -i /dev/video0 -v copy -an -f mjpeg  - | nc -k -l 8081

 

The first one streams the rpi camera, the second one streams a usb camera that supports mjpeg output. The rpi is too slow to re-encode or transmit raw frames.

I’m using mjpeg because it seems to be the least fragile. H264 usually works with the rpi camera, but doubles the latency.

Both cameras were only using 30 Mbit bandwidth. The usb camera was using 30% cpu on one core of a pi2, and the pi camera was using 5%.

 

To receive the streams in openpnp just create two OpenCvCameras, with connection strings http://192.168.0.220:8080/stream/video.mjpeg and http://192.168.0.220:8081/stream/video.mjpeg

The connection string was the part that I discovered by accident – I had it left over from a previous streamer, and it works where the more usual address of tcp://x.x.x.x:8080 didn’t – I think it gives ffmpeg enough guidance to figure out how to interpret the stream. Not sure.

 

And the latency is good – between 200-400ms, as seen in the screenshot. For some reason I had to set the framerate to 50 fps in openpnp or there is about 2 seconds of latency – again I think something weird is happening in ffmpeg.

 

Now I need to mount the cameras – the pi camera is going on the bottom and the usb camera on top.

That’s it for now.

 

Paul.

[Jason von Nieda]

Very cool Paul! I suspect this will be an interesting option for people since the Raspberry Pi camera is quite good.

For OpenCV, if you can get everything sorted out with including ffmpeg I am happy to include it in the future builds.

Jason

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/SY3PR01MB1386EAE6DC323FB32ED12CB19E020%40SY3PR01MB1386.ausprd01.prod.outlook.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel]

I have a curiosity : would be possible to run OpenPNP on RPI and use PI camera as uplooking and for downlooking to use a usb camera ?

Would be fantastic to get rid of that huge computer around of pnp machine...

Daniel

[matt]

Hi Daniel

I'd suspect the Pi is well underpowered. I tried to run OpenPNP on a
500mhz K6-2 and it didn't work. As nice as Java is... its kinda bloaty.
I've a feeling OpenCV will struggle as well (i know there is a port).

Mind only one way to try!

Matt

> --
> You received this message because you are subscribed to the Google
> Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to openpnp+u...@googlegroups.com.
> To post to this group, send email to ope...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openpnp/a7bc3612-cfd8-48c8-a77d-899373f4ff9c%40googlegroups.com
> [1].
> For more options, visit https://groups.google.com/d/optout [2].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/openpnp/a7bc3612-cfd8-48c8-a77d-899373f4ff9c%40googlegroups.com?utm_medium=email&utm_source=footer
> [2] https://groups.google.com/d/optout

[Daniel Dumitru]

​Indeed may look small a RPI but last version claims 1.2GHz 64-bit quad-core ARMv8 CPU.

IT would be very nice to have a compact machine with a pi inside to boot fast...

[Paul Jones]

I think you could, but even on a pi3 it would be terminally slow. I briefly tried opening eclipse on a pi2 and after a minute waiting I gave up. Using an old laptop with external screen + keyboard would make much more sense imho. Perhaps I’ll give it another go, I’m curious myself to see how it goes.

Also, the current opencv bindings that openpnp use doesn’t have support for arm processors, so openpnp would need a few mods to use the one provided by raspian.

 

Paul.

 

--

You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To post to this group, send email to ope...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/a7bc3612-cfd8-48c8-a77d-899373f4ff9c%40googlegroups.com.

[Cri S]

Yes, it's works on Rpi, just remove the nu.pattern call and change pom.xml for opencv. 

Opencv need to be installed on rpi, and it's take hours (12). You need doing it only once (hopeful).

On Pc and Rpi, you need to have installed the same opencv version. That's all, beside you need retarging java8 back, but that's fast.

[Daniel Dumitru]

Cris, could you please detail how to build a RPI image with working OpenPNP ?

Thank you

Daniel

[Cri S]

I use opencv 2.4.8  and as said,

remove the nu.pattern line, comment it out, i use ed for that.

The file are, in case you make it manually:

code/vision/FluentCv.java

code/vision/LensCalibration.java

code/vision/pipeline/CvPipeline.java

code/experiments/FiducialTest.java

code/util/OpenCvUtils.java

code/machine/reference/ReferenceCamera.java

code/machine/reference/vision/OpenCvVisionProvider.java

code/machine/reference/camera/OpenCvCamera.java

Install opencv 2.4.8 on pc locally, and change pom.xml for using the local repostory:

                <dependency>

                        <groupId>local</groupId>

                        <artifactId>opencv</artifactId>

                        <version>2.4.8</version>

                </dependency>

Use retrolambda do packport the code and use Sun java v7 on rpi. 

The resulting file runs on rpi.

I have ready to run debian install files for rpi with grbl driver and Pi4JP , but i suspect you want tinyG ?

I have removed Batik, gson, jssc, jvfw, jython, lti-civil, onvif, avalon, jcodec and probably other lib,

and removed the eagle direct import and gerber direct import. 

[Erik Grönvall]

Maybe you could use the Udoo board? http://www.udoo.org/