Shifting framerate/video duration for VideoWriter

[UserNorw]

Hi,

Im a relatively new to bonsai and Im trying to set up a recorded video that is synced to a keypress. The keypress will be used to sync the video to a separate electrophysiology system, so it is important to keep the stored video at a stable duration. Timestamping,  timer onto the video output and csv file writing is going smoothly, but I am having some issues with the stored avi file for each session. The duration of the video keeps alternating between different durations. It slows to half speed/duration between video recordings.  When I change the framerate for the VideoWriter from 30 fps to 60 fps this will fix the issue for a couple of sessions, but then it jumps back again unexpectedly. It always jumps to half/double of the real time of the session.

I am using different Logitech webcams to record the video and I have tried both VideoCapture and CameraCapture with no change. I have also tried around 10 different codecs to check if it was a compression problem and I always keep the fps of the VideoWriter constant when this happens. 

Ive looked through the forums, but I couldnt find anyone with this particular issue, but as I said, I am new, and there could be a very basic solution to this problem that I havent found, since no one else are having these problems.

Do you think this solution will help:  https://groups.google.com/forum/#!topic/bonsai-users/xwqMzIbyeC4 ? 

Or could it be an issue with the driver for the logitech webcams/ video card ? 

I would be very grateful for assistance with this! I am otherwise very happy with the system and the user friendly interface of bonsai.

Best,

I

[Gonçalo Lopes]

Welcome to the forums!

I don't remember ever having exactly this issue, but I agree that it can be a driver issue. Specifically, does your logitech camera have an auto-exposure mode? If the camera is adjusting the exposure automatically, this can change the framerate widely, which will then cause the video duration issue.

Another way to try and understand what is happening is to look at the timestamps coming out of the video stream. If you measure the interval between timestamps, is it constant? Or do you see it fluctuating?

If the issue turns out to be the auto-exposure, you can shut it off by using the VideoCapture node and accessing the property pages of the camera. It's a little button in the property window (see below).

I have also simplified the workflow somewhat (see attached) but I'm sure the issue is not related to the workflow.

Hope this helps.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/253b1c42-cdca-4e0b-ac85-06bf314bfd90%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[UserNorw]

Thanks for the quick reply!

 

So, apparently we've had this issue with several cameras, but the camera option is unfortunately not available through the properties tab or possible to modify. Ive contacted logitech with no reply yet as to whether i can change this property. However, I cant see any delay when using logitechs own software to record the video. Second, I have another Logitech camera, where this option can be unchecked, and the problem remains.

 

I've checked the timestamps, (i.e. the node timestamping every time of any coordinate) and there is indeed a fluctuating interval. I quess this indicates that the camera is auto-exposing and adjusting the framerate in the bonsai workflow.

Im dependent on a particular camera to fit my experimental setup, the logitech camera has a fisheye zeiss lens with correction for bending the image, so i'd really like to use this camera. 

Could there be a workaround in bonsai you think?

Storing each frame separately before the rest of the workflow is implemented for instance?

Im storing the real-time timestamp onto the video and this is always correct. Another point worth mentioning is that when I set the framerate of the videowriter to 0, this removes the delay during the workflow (I can see jumps in the video during recording when the camera is moved, however no file is stored when this is set to 0) , this could perhaps suggest that the problem is related to the framerate conversion/or the videowriter part. 

Is there any way of storing each frame as a jpeg and combining it to an avi later? like this: https://groups.google.com/forum/#!topic/bonsai-users/xwqMzIbyeC4 ? 

Or any simpler workaround?

Ida

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

[Gonçalo Lopes]

Quick question: can you try a simple workflow where you just timestamp each image of the video and save the timestamps to a file (i.e. only save the timestamp values, and not the video)?

If you do this and you still see jumps in the video, this indicates that there is an intrinsic delay in video acquisition (i.e. driver) that does not depend on further Bonsai processing. If the camera is indeed doing auto-exposure this will delay acquisition at the hardware level and there is little you can do downstream, I'm afraid...

Out of curiosity, what is the exact Logitech model that you are using? 

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/1da84137-a628-4f87-ac21-918ee4c7d59a%40googlegroups.com.

[UserNorw]

So, If I understand you correctly, my orignal workflow saved these via CameraCapture - Source.RegionofInterest-Timestamp. And yes, there are discrepancies in the stored intervals of timeframes.

So thats crappy. I really hope i dont have to alter my setup and get new cameras, so I wanted to ask you if you think an alternate solution will work:

The purpose for storing these videos is for post-experimental tracking and having videos with correct duration and length would be a good verification of the synced tracking. 

However, I could try to track it real-time and store the tracked coordinates in a timestamped file, that I could later double check with the stored video (that has the real-time printet onto the stream).

This way, delays/discrepancies between the frames would not be as important since they all should be appropriately timestamped. 

Im trying to track two different infra-red diodes, and i have applied a physical IR filter (my previos post), so the track is pretty easy. However, i am trying to store the two binary regions as two xy coordinates with real-time timestamps, but im having problems setting it up since Im still a novice to bonsai. Would you use BinaryRegionAnalysis-SortBinaryRegions for this? And where do you go from there?

The camera im using is a Logitech 930e webcam. Still no reply from them. 

Super grateful for any help you could give me! A little stressed timewise with the setup now. 

Ida

[goncal...@gmail.com]

Another quick question: When you opened the property pages of the camera in Bonsai, were you able to disable low light compensation? I’m asking because it can be easy to miss.

 

Basically use VideoCaptureDevice, hit the button I mentioned previously (or Ctrl+Enter with the node selected).

 

On the screen that shows up, go to the camera control tab. The low light compensation should be a small checkbox at the bottom of the config window.

 

The reason I’m insisting is I found this YouTube tutorial that seems to corroborate that the 930e does automatic exposure but it looks like it can be disabled:

https://m.youtube.com/watch?v=5rEBXJKMi4k

 

Syncing with timestamps will work, depending on  the desired precision of your syncing… In general it will not be possible to sync precisely at millisecond scale, you would have to live with an error around tens of milliseconds.

 

The timestamps themselves are microsecond accurate but there is a random jitter (OS dependent) between the frame being physically acquired and arriving decoded at the software.

 

Hope this helps.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/c2d93cb8-3950-442c-ba00-b290e50eff8a%40googlegroups.com.

[UserNorw]

Thanks for that! 

Yes so now I've unchecked the low light and expose tabs, and it looks like the interval between the frames has improved in accuracy. its now at max 20 but mostly around 1-5 ms. This will really improve my setup regardless of the final solution.

However, the doubling of duration is still pretty random. if videowriter is set to 30fps e.g. for a video with a length of 32 sec, the stored video is 52 seconds with the appropropriate number of frames for 52 secs (1567, measured in matlab). However, the output of the timestamps from Source.RegionofInterest is half of that (784). Is maybe the timestamped Source.RegionofInterest a poor solution to timestamping the frames?

If I set the video to 60 fps this aligns the timer and the timestamps for some time, before it jumps back. 

So, Im thinking, to be safe, id like to try the solution i mentioned: tracking live and saving the xy coordinates with the real-time timestamp (as well as the frame timestamp).  Since I know the real-time timestamp is accurate, if I couple it with the coordinates at that particular time I think I will be able to sync the koordinates with the keypress to the electrophys setup. Then I can just use the video for backup checking of the coordinates.

I can live with a jitter of 10ms, but only per timestamped frame/koordinate. 

Did you see the question about the two-diode setup in my last comment? 

[Gonçalo Lopes]

Ah, wait, actually I just opened your original workflow again and there is a problem there explains the doubling of the frames. Basically you're using CombineLatest to merge the output from multiple branches.

This is a problem if you want to save the output of that, since CombineLatest will fire once every time one of its input changes. This combinator is usually only used when the inputs are not synchronized (e.g. images from independent cameras).

If you know that the inputs match exactly one-to-one, then the correct operator to use is Zip. In general, whenever you have a pattern of branching and then merging (without changing or filtering the stream), then you always want to use Zip.

This should explain the extra frames. Can you try out the attached workflow and see if that fixes it? In general, your timestamp file should have exactly as many timestamps as you have frames in the video (+/- 1 at the end since when you stop the workflow you can occasionally skip writing of the last stamp or frame).

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/fd39b30a-fe72-48aa-879f-380f7acb1ea0%40googlegroups.com.

[Chi-Yu Lee]

Dear Goncaloc, UserNorw and Bonsai team,

I am using a setup with a ELP webcam <ELP-USBFHD04H-RL36>  for a workflow to monitor mice movement at 30 FPS and deliver visual stimulus to the mice.  

I used to experience dropping frames when I keep the auto-exposure on (dropping around 1/6 of frames). Since I turned off the auto-exposure as you mentioned in this discussion, the problem has been lessen much better. Nevertheless, there is a fluctuation in the time interval every 10 frames (as the attached picture). I have tried SampleInterval module, but it seems that the minimum interval I can set is 1 second. I have viewed other posts but still do not know how to fix/lessen this problem. 

Would it be possible to use a National Instrument DAQ to send command within a fixed time interval to trigger the camera?

BTW, I have tried the same setup with workflow <TempIA2.bonsai>  and frames also doubled in the video. For some reasons, I can not run other workflows you uploaded like TemplA3.bonsai, test3, TemplA2.zip. Do you have any idea what problem might be going on? I really want to do what UserNorw did to record the timepoint when a frame arrives.

Please feel free to let me know if you have any concern or comments

All the best and Merry Christmas!

Chi-Yu

[Gonçalo Lopes]

Hi Chi-Yu,

I have also experienced erratic timing issues with ELP cameras myself (which is why I don't like them so much anymore). Still, if you are acquiring at 30 FPS, a sporadic jitter of +/- 16 ms should not be enough to cause frame drops (although to be fair this may depend on hardware). Regardless, given the sporadic nature it should not be enough to change the number of frames by half or double. How are you counting the number of frames in the video? Did you check with Bonsai FileCapture, or is it by counting the number of rows in the text file?

Can you attach the workflows that you have used to the forum question? I had some difficulty tracing which exact workflows you are using.

--

You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/cbfb3a97-1366-4f57-8d7b-5324965c80fa%40googlegroups.com.

[Chi-Yu Lee]

Hi Goncaloc

I do not have the problem doubling frame anymore but only the erratic timing issues

I have read another post about cameras and you mentioned about PSEye. Have you found any tissues with their cameras (the one with removable IR filter)?

This is the workflow I am using, TemplA2.bonsai, the one uploaded by UserNorw. I count the number of frames by outputting the time of each frame as a number of rows in a text file and counting their number. I have used VideoCaptureDevice to turn off the autuexposure already but I have not used FileCapture yet. What can I do with FileCature? 

Best Regards

Chi-Yu

[Gonçalo Lopes]

Using FileCapture you can go through the video frame by frame using the editor interface (right-click > Show Default Editor). This way you can try to diagnose problematic frames.

As I mentioned before, if the problem is just erratic timing of the CPU timestamp, that does not necessarily mean that you are losing frames of the video, or that the video itself is irregularly timed. One way to confirm for sure is to point your camera at a digital stopwatch with at least two decimal places precision, and then see whether time advances periodically from frame to frame. If the video frames follow the clock in a reasonable way, then you can just use frame number as your time measure, and ignore the CPU timestamps.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/6aa7a9dd-fe39-4f77-aa32-61b856dc29e9%40googlegroups.com.