How to introduce Flying Vision using global shutter camera

[Siddharth Gautam]

Hi everyone,

I'm currently working on the OpenPnP platform and I've been exploring the camera code, especially the OpenCvCamera class. I want to integrate Flying Vision using a global shutter camera, but I’m a beginner and I’m not sure where to start.

I understand the basics of capturing frames with OpenCV and how VideoCapture works, but I'm unclear about:

1. How to modify or extend the current camera code (OpenCvCamera.java) to support global shutter cameras suitable for Flying Vision.

2. What specific hardware or camera models are commonly used for Flying Vision and how they differ (e.g., global vs. rolling shutter).

3. How to ensure real-time synchronization between high-speed motion and frame capture.

4. Whether any additional configuration or triggering mechanism is needed to properly use global shutter cameras in motion.

Could someone please guide me on:

• How to approach this integration step-by-step?

• If any sample implementations or documentation exist within OpenPnP or elsewhere?

Thanks a lot for your help! Any advice or resource links would be appreciated.

[Jan]

Hi Siddharth!
Fly-By-Vision would be a nice feature, however, it requires some work
and probably reworking of the base structure of OpenPnP. So it may be
split into multiple sub tasks. First off, please search this list for
fly-by-vision. It has been discussed a few times in the past.
Here is what I remember:
- Fly-by-vision requires a camera with a hardware trigger in order to
take a picture exactly when commanded. The trigger has to be generated
by the motion controller to synchronize it with the head location.
- It seems, that ELPs ELP-USBGS1200P02 is a good candidate to try. Its a
UVC camera with global shutter color sensor and hardware trigger option.
So its already supported by OpenPnP on all platforms. (The *P02* is
essential, because it denotes the version with hardware trigger. The
more common P01 has no hardware trigger option.)
- Once one starts thinking about fly-by-vision, one will step over the
current bottom vision implementation, which is synchronized with the job
processors state machine. This would need to be disentangled to be
prepared for asynchronous operation. IIRC the main problem might be that
the image taking is part of the vision pipeline and requires access
using a machine task, that can not - yet - run in parallel with other
machine tasks.
- Probably all hardware triggered cameras will likely suffer from a
delay between trigger and shutter opening. This needs to be factored in
when planning the motion patch and when to trigger the camera. So some
kind of "flight route" would be needed to reliably trigger and cross the
camera.
- There might be modifications required on the controller side as well
to support precise triggering with motion. RRF (Duet) might support that
already.
IMHO asynchronous bottom vision shall be the first task to do and is
likely also very interesting for parallel bottom vision and a
performance boost even without fly-by-vision.
All the motion and hardware triggered vision would then be step two - IMHO.

Jan

On 31.05.2025 23:40, Siddharth Gautam wrote:
> Hi everyone,
>
> I'm currently working on the OpenPnP platform and I've been exploring
> the camera code, especially the OpenCvCamera class. I want to integrate

> *Flying Vision* using a *global shutter camera*, but I’m a *beginner*

> and I’m not sure where to start.
>
> I understand the basics of capturing frames with OpenCV and how
> VideoCapture works, but I'm unclear about:
>

> 1.
>
> *How to modify or extend the current camera code*

> (OpenCvCamera.java) to support global shutter cameras suitable for
> Flying Vision.
>

> 2.
>
> *What specific hardware or camera models* are commonly used for

> Flying Vision and how they differ (e.g., global vs. rolling shutter).
>

> 3.
>
> How to ensure *real-time synchronization* between high-speed motion
> and frame capture.
>
> 4.
>
> Whether any *additional configuration or triggering mechanism* is

> needed to properly use global shutter cameras in motion.
>
> Could someone please guide me on:
>

> *

>
> How to approach this integration step-by-step?
>

> *

>
> If any sample implementations or documentation exist within OpenPnP
> or elsewhere?
>
> Thanks a lot for your help! Any advice or resource links would be
> appreciated.
>

> --
> You received this message because you are subscribed to the Google
> Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openpnp+u...@googlegroups.com
> <mailto:openpnp+u...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/
> openpnp/61b84f1b-e5f5-4c0b-b0c9-f0c85712542bn%40googlegroups.com
> <https://groups.google.com/d/msgid/openpnp/61b84f1b-e5f5-4c0b-b0c9-
> f0c85712542bn%40googlegroups.com?utm_medium=email&utm_source=footer>.