Dual camera synchronisation

[hd1...@bristol.ac.uk]

Hi all,

I am looking to capture stereo video for 3D limb tracking through a reach. I will also (using separate hardware) be collecting neural event data. Naturally I wish to keep everything synchronised as tightly as is possible for ease of analysis etc.

My [current] issue is that the two video streams seem to desync. over time and I can't seem to work through it. I have attached the workflow in it's most current state and below are a few more details about the set up.

Cameras tested; 2x Microsoft lifecam HD 3000, and 2x logitech C930e , all @ 30fps (but will be looking to push this up to 60-120fps)

Computers used; intel i3 core and i7 cores running Windows 7, 4GB RAM for the i3, 8GB for the i7

Arduino uno via USB.

Briefly, I wish to capture the two camera feeds, overlay an 'elapsed time' timestamp onto the bottom corner of the feeds, and sink them to .avi. 

Additionally, perform some online object tracking on them and save the XY positions of a marker to .csv files.

I have a SubscribeWhen node at the head of the elapsed time sequence acting as the trigger for the capture- currently it is linked to a spacebar KeyDown ease of debugging, but eventually that will be triggered by a pulse from the neural acquisition hardware when I start that recording. Additionally, the Arduino DigitalInput will carry a conditional signal from the behavioural hardware which I will write to the .csv files.

I have used the Sample nodes after the CameraCapture nodes to try and keep the synchronisation tight- such that a frame is only 'taken' from the cameras when an 'elapsed time' value is received, which is itself tied to the framerate of the first camera. 

In my head this should lock the two capture streams together.

However, when I use the visualiser for the PythonTransform that overlays the elapsed time onto the video feeds, and 'hit go' the overlaid timestamps drift apart over time, generally noticeable within 30-60s. 

If during acquisition I use some sort of clapper-board, it seems as though the video streams are being displayed in real-time, and both without delay- so it seems as though writing the time to the first stream is quicker than to the second stream? 

The drift in the overlaid timestamps occurs with whatever hardware I use, frame sizes, and I have also tried using single SubscribeSubject nodes for each of the 'Elapsed time' subscriptions.

I've been puzzling this for about a week now, and think i've reached the end of my Bonsai know how! Any help or advice will be greatly appreciated

All the best, 

Henry Darch

[Gonçalo Lopes]

Hi Henry and welcome to the forums!

If your future goal is to record these behaviour videos simultaneous with neural data, then the preferred way of synchronizing all the different data streams is by using some kind of hardware sync pulse.

This signal can be adapted to the needs of each recording system. For example, in the case of the Microsoft lifecam, no analog or digital input port is available directly on the camera, so the easiest way would be to use an LED as an optical indicator of the sync pulse, somewhere where it would be visible by the two cameras. In this way, you can have an accurate recording of times of interest across the multiple systems in a way that is independent of operating system jitter or processing delays.

In case this approach is not feasible, or for fast online correlation of independent data streams, you can use CombineLatest directly to correlate the latest frames from both streams as fast as possible. A simple example of creating a single image synchronized in this way would be:

​

I have attached a slightly more involved example which also timestamps and records the video streams of each camera to different files.

Synchronizing streams in this way guarantees that the most recent possible data from each stream is used, so in a sense it gives you the guarantee of minimum delay. Because the correlation is done in software however, there is no absolute guarantee of the precision of the synchronization. In general, it will be good enough at the timescale of behaviour analysis and human inspection, but not for synchronizing with neural events.

It looks like in your workflow you were trying to use a shared event variable to sample data from each video source. This looks like it will be error-prone. in general simply using CombineLatest directly and letting the cameras record as fast as they can will give you the best possible guarantees you can get in software.

I have also noticed that your original workflow uses CombineLatest and Zip nodes in a perhaps slightly redundant way which may account for the weird discrepancies you were reporting, but if we can design a more simple synchronization workflow then perhaps these complexities can be avoided.

Feel free to let me know more about the particular constraints of your application and I can suggest further modifications. It looks like a very interesting setup. Hope these suggestions help.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/28c7e586-054b-4e50-a1f9-b9d6308cde4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charles Latchoumane]

Hi,

  I have a related issue. I decided to use screen capture recording to keep the video of my entire setup. Combine latest would resolve synchronization of the two cameras (I am guessing that as long as my higher speed camera produces a new frame, the slower one will provide the current frame, until it is changed. What I noticed with the screen capture was that the final video speed was slown down and I couldn't fix it by changing the frame interval parameter. (in the future if I can integrate my time graphs and the visual output from the cropped/normlized image nodes I would just save the original video after concat all of them together).

Thanks for your input! (see my workflow attached).

[Alessandro Di Filippo]

Hello Goncalo, I was looking at your example, and I'missing why you use the CombioneLatest and Concat after you already wrote separately the video files.

Actually, it is still not completely clear to me how CombineLatest is helping with the synchronization issue.

Thanks as always for your help!

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

[Gonçalo Lopes]

Hi Alessandro,

CombineLatest returns the sequence of all the pairs of images that are closest together in time (using the system clock). This is a kind of real-time synchronization, in the sense that if you save the combined video you already know which frame from one camera matches with the frame of the other camera, without the need to match timestamps, etc, of course to the precision of operating system jitter (assumed ~10ms).

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/3b13b9e8-192b-46b0-a32f-f641d5b0d1be%40googlegroups.com.

[Alessandro Di Filippo]

Thank you Goncalo, it is a bit clearer now.

Anyway, I tried to replicate your example: it is exactly the same, with CombineLatest and Concat, but with SpinnakerCapture instead of CameraCapture.

This way it is not working: is it because Spinnaker creates a different sequence?

[Gonçalo Lopes]

Hi Alessandro,

Possibly this is because SpinnakerCapture returns not just a sequence of images, but a sequence of images + metadata. You have to unpack the images first with right-click > Output > Image, and then do the CombineLatest on those sequences.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/774ded65-c5af-4213-b23c-7e1a6e1b637c%40googlegroups.com.

[Alessandro Di Filippo]

That was also my suspicion, thus I tried by unpacking the Source.Image.

Still, the Concat node gives me the "No method overload..." error.

[Gonçalo Lopes]

Can you attach the workflow you are using? Make sure to use Concat (Dsp) and not Concat (Reactive). The latter is for time (reactive sequences), the first one is for space (data matrices).

​

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/31549133-bf06-4176-91cb-300232e5f3f5%40googlegroups.com.

[Alessandro Di Filippo]

That was exactly the problem! Thank you, now everything works.

[Sarah P]

Hello!

I downloaded the attached file but am encountering error messages. I think the error messages may be due to the file having been written in a previous version of bonsai. I'd really like to be able to use this module. Could you please help me troubleshoot the error messages?

Thank you!

Best regards,

Sarah

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai...@googlegroups.com.

[Gonçalo Lopes]

Hi Sarah,

Yes, this is because of some modifications to later versions. The fix is easy enough: delete the Source prefix on the selectors in CsvWriter (or delete them and pick the timestamp again).

I am attaching converted workflow.

Hope this helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/88e90daa-1cbc-4e18-aa1c-e50f7cd2b0bb%40googlegroups.com.

[Sarah P]

It works now! This is great! Thank you!

I have a question though. I don't see a timestamp on the video frames. Is it still supposed to do this?

I have another question: If I drive two cameras to record at the same fps (i.e., 20 fps) using SampleInterval, is there any benefit to using CombineLatest and Concat also, to synchronize the frames? Is this redundant? Is one approach better (i.e., more efficient, between synchronization, etc) than the other?

Thank you!

Best regards,

Sarah

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai...@googlegroups.com.

[Gonçalo Lopes]

Hi Sarah,

In the workflow I sent previously, the timestamps are recorded in separate CSV files for each camera (they are not included in the video frame itself, so as to allow easy automated analysis post-hoc).

Be aware that for that specific example, the timestamps were being done before the image was correlated (i.e. before CombineLatest > Concat). If you are relying exclusively on CombineLatest for synchronization though, it might be more appropriate to use Timestamp > SampleInterval after CombineLatest, so that you are timestamping/sampling the single synchronized stream. Something like this (also attached):

As I mentioned in the original post, relying exclusively on CombineLatest has some limitations in terms of jitter between cameras, but for low frame-rates (e.g. 20-30 fps) it should actually be very reasonable. I would still recommending benchmarking the data first to ensure quality, e.g. by recording a video of a digital clock with both cameras using this system and comparing timestamp intervals versus the numbers on the clock face.

Hope this helps.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/34db9c1e-12f3-47df-95b3-378392985755%40googlegroups.com.

[Sarah P]

This is great! It works now and this code will make it so much easier to record and synchronize multiple videos! Thank you!

Best regards,

Sarah

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/34db9c1e-12f3-47df-95b3-378392985755%40googlegroups.com.