Point grey camera, video duration reduced

[Pau Nebot]

Hi everyone!

We're trying to implement a bonsai workflow to automatize our analysis, but at the moment we were just using bonsai to record our tests. We noticed some weeks ago a shift in video duration and since then we've been trying to solve it.

These are the details:

Camera details:

The camera used was a FLEA3 1.3MP monochromatic camera (actually Flea3 FL3-U3-13S2M).

The USB3.0 cable, camera drivers, USB port drivers, firmware and SDK suite (software) have been replaced or updated recently as part of the troubleshooting measures: 

• Brand new 2m cable (USB3.0)
• Camera details
• Flea3 FL3-U3-13S2M (15062947) Sony IMX035 (1/3" 1280x1024 CMOS)
• Firmware version: 2.7.3.0
• USB Camera Driver (PGRUsbCam.sys) - 2.7.3.84
• pgrusbcam.sys ver 2.7.3.84 is also installed.
  
• The latest FlyCapture suite with the corresponding drivers inside (2.10.3.266).

All this in a computer with the following specs:

• Windows 7 (64 bit)
  
• Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
  
• 8Gb RAM
  

Problem details:

When I try to record using the simple workflow I've attached (FlyCapture -> Source.Image -> VideoWriter), the output .avi duration is shorter than the reality. I've tested this and I get around 40s less video duration in a 10min recording. Camera is adjusted to 30fps. Exposure, shutter and gain parameters fixed. 1280x960px image size.

We thought the camera and/or configuration were the cause of the problem but after updating everything we found that using other recording software gives videos with the right duration, where bonsai doesn't.

I'd like to know how to solve this issue, so our lab can continue the work towards automatizaton.

Please ask for whatever information and/or configuration that isn't listed above that may help solving the problem.

Thanks!, regards,

Pau Nebot

[Bruno Cruz]

Hi,

I was having a similar problem because i was trying to force Bonsai to close through the taskkill command. The whole video was there if i opened it in bonsai and saved it again (filecapture into videowriter) but if i opened it in VLC was very short and even matlab could not index all the frames. The fix was actually stoping the script properly (using the stop sign) and i never had the same issue.

Also, this post seems to describe an alternative solution:

https://groups.google.com/forum/#!searchin/bonsai-users/video$20duration%7Csort:relevance/bonsai-users/wd4wAhgWlQA/CXvYt85SBQAJ

hope it helps,

Bruno

[Pau Nebot]

Hey! Thanks for the quick answer.

I'm afraid your fixes can't be applied to our problem. For example, we always stop the recording using bonsai's stop icon, so we're not forcing the recording to stop. I should also add that it always takes around 2s to actually stop the recording from the moment we click the icon to the actual ending of the record. We used to smash the button in a hope that bonsai would stop recording quickly, but that doesn't help at all to prevent those ~2s GUI freeze.

On the other hand, I don't think this is a problem related with the duration header or the "end" of the file, as I have checked all our tests and the whole video is there, from start to end. A careful observation, however, (or recording a chronometer, like in my case), reveals that the time runs slightly faster in the video. That really small delay adds up, and after a few minutes the video is seconds shorter than expected.

Thanks anyway,

Pau

[Gonçalo Lopes]

Hi Pau and welcome to the forums!

My best guess is that your recording workflow is somehow dropping frames. Since you are using a PointGrey, you can actually confirm this by recording the embedded hardware frame counter of every frame to a text file and then measure whether the numbers are consecutive (i.e. is the difference between frames larger than one?). If this is happening, then frames are being dropped.

I'm attaching a workflow to record this information, for convenience. Make sure you enable the hardware frame counter in the FlyCapture options before you run it, or else all the recorded numbers will be zero (if you cannot find this setting, see more information in this previous post).

Also, I would recommend that you update the Bonsai.PointGrey package to the latest version (2.3.0). It came out a couple of hours ago and has some minor performance tweaks which may help later.

If it turns out you are dropping frames, there may be a number of reasons for this, including the possibility that the video codec used by default in Bonsai is too processor intensive for the machine you are using, so we may need to use a different codec.

In any case, it is better to think about a solution once we have fully diagnosed what the issue is.

Hope this helps and let me know the results.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/f521a8a6-a375-4df3-912f-f992da3b65e9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Pau Nebot]

Hi Gonçalo and thanks for your help!

I managed to test your idea this afternoon and here are the results:

Successfully updated Bonsai.PointGrey and used your workflow. (I noticed that the "slow stopping" or "freeze when trying to stop the recording workflow" problems are gone). I recorded two videos (6min and 8 min). The first 6 min video skipped 68 frames (I'm recording at 30fps). The second one (8 minutes skipped 769).

With these results, and knowing that a recording with the manufacturer's software is flawless, what can I do to fix this?

I'd like to know (as a start), how can I modify the codec used. For example to use H264. Where should I install it?

I hope we can find a solution, as we are very interested in using bonsai for our analysis.

Thanks!,

Pau Nebot

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

[goncal...@gmail.com]

Hi Pau,

 

Thanks for the feedback. Before we explore the solution of changing codec, I have two quick questions:

 

1. Are you using Bonsai 64-bit to run these experiments? If not, can you give it a try and see if anything changes?
2. Do you have a dedicated GPU in your setup, or is it an integrated Intel?

 

I will reply later with an alternative workflow for encoding.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/ad4e02b2-aefd-41fb-903a-8daebc4816c8%40googlegroups.com.

[Pau Nebot]

Hello!

1- Yes, I'm using Bonsai 64-bit.

2- I don't. I'm using an integrated Intel. (I understand a dedicated GPU as having a graphics card installed (which I don't have), please correct me if I'm wrong).

Pau

[Gonçalo Lopes]

Sorry for the delay. Another quick question:

Can you try removing the VideoWriter from the workflow I sent you previously and see if you are still dropping frames? This is so we can get an accurate baseline of frame drops from the acquisition process itself.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/373a39ef-b7c3-4302-89a4-8f195e6d1f34%40googlegroups.com.

[Gonçalo Lopes]

​Hi Pau,

Attached you can find an alternative workflow for us to explore different codec possibilities.

In order to make it really flexible, we will be using the command line version of FFmpeg (the same video encoding library that VLC uses) in order to run our experiments. I recommend you download the latest FFmpeg 64-bit static release from the Zeranoe website. The zip file should come with a "ffmpeg.exe" file, which you should place in the same folder as the attached workflow.

Basically, the way this new workflow works is by piping the raw video data directly to an independent FFmpeg process which we set running in a Python script launched inside the workflow:

​
​

The first nodes of CameraCapture, Grayscale and Resize you can replace with your own FlyCapture source (I put them in so I could simulate the encoding with my webcam).

The PythonTransform bit below launches FFmpeg with appropriate parameters that define the input video format and the target codec parameters. Here's what it looks like:

import os
import clr
clr.AddReference("Bonsai.System")
from Bonsai.IO import PathHelper, PathSuffix

filename = 'video.mp4'

def process(value):
  os.system(r'ffmpeg -y -f rawvideo -vcodec rawvideo -s 1280x960 -r 30 -pix_fmt gray -i \\.\pipe\video -c:v mpeg4 -vb 20M {0}'.format(filename))

Basically the only thing the script does is to call ffmpeg and point it to the pipe where the data is coming from. Here's a basic explanation of the parameters I've used (as an example):

-y: this is just to answer "yes" to any possible question ffmpeg might have (like overriding files)

-f rawvideo -vcodec rawvideo -s 1280x960 -r 30 -pix_fmt gray -i \\.\pipe\videotest

This part is where you specify the format of the input source. Details:

-f rawvideo: this indicates the format of the video is raw (uncompressed) video

-vcodec rawvideo: same

-s 1280x960: size of video frames

-r 30: frame rate in FPS

-pix_fmt gray:  the format of each image pixel (in our case a pixel is grayscale 8-bit), but other values could be used for color (

-i \\.\pipe\videotest: this is the "named pipe" where the input is being streamed from (this is created by Bonsai using the ImageWriter node)

-vb 20M -vcodec mpeg4 out.avi

This part specifies the format for the output encoded video. Details:

-vb 20M: encoding proceeds at variable bitrate (meaning dynamically adjusted bitrate as a function of image complexity, more complex images will take longer to encode, simpler images will be faster)

-vcodec mpeg4: the codec used to compress images for output (mpeg4 seems to be one of the fastest for CPU-only encoding)

video.mp4:  the name of the file (you can change it to .avi as well, as you prefer).

I would be curious to know how the videos and the frame drops come out using this method.

Another thing to try: you may be opening a Bonsai video window in order to monitor the stream as it's being saved. You might want to try running the encoding with and without the window open, just to see if there is any impact of the visualizer on system performance.

There are a couple more things I still want to try but in any case this FFmpeg benchmark will be a good place to start.

[Pau Nebot]

Hi! Thanks for everything!

Lets see...

I removed the VideoWriter and I did 2 tests:

Test 1: 2 min recording, 0 dropped frames.

Test 2: 8 min recording, 538 dropped frames.

After that I loaded your last workflow but I modified the source. I did 2 more tests:

Test 1: 8 min recording. Right duration. 415Mb file.

Test 2: 10min 20s recording. Visualizer window opened. Around 10s less duration that the expected. 523Mb.

(I'm not sure if the 8min conditions should be trusted, the time shift seems quite variable)

I attach a capture of the bonsai terminal just in case you find it informative.

What do you think is the cause of the problem? Thanks!

Pau

[Gonçalo Lopes]

It looks like the CPU is struggling to keep up with the encoding for some reason. It's strange, as we are now able to routinely grab and save 1600x1200@60Hz with practically zero frame drops.

In any case, I want to try two more things to optimize the process.

Attached you can find a preview version of the 2.3.1 patch release for the Bonsai visualizers, which slightly optimizes the default image visualizer (one full image copy is eliminated). I am wondering if this helps at all to keep the durations correct for the case where the visualizer window is open.

In order to install it, you can follow this tutorial: https://bitbucket.org/horizongir/bonsai/wiki/InstallCustomPackageTutorial

Don't forget to select "Include Prerelease" in the Package Manager in order to see the package, since this is a preview/experimental version.

By the way, in the first test results you reported, was the visualizer window also open or closed?

The second approach I want to try is a different strategy for retrieving frames from the camera, using managed event callbacks. I'll send you a package for this tomorrow, but let me know if the visualizer optimization helps and the info from the first tests.

Thanks for trying this out, it will be good to understand how we can squeeze the best video encoding performance. We can still try other codecs in any case.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/b5b824ea-e8c6-451a-bfc3-0effef1bc3a5%40googlegroups.com.

[Pau Nebot]

Hi Gonçalo, good morning (from GMT +1 :) )

Here are the results:

I tested to record with the 2.3.1 patch and the visualizer opened. The expected 10min 10s video was 23s shorter.

After that I tested the same workflow (the last one you gave me, "ffmpegencoder.bonsai"), but this time keeping the visualizer closed. Expected 8min 15s video was 19s shorter. (Do these results follow a proportion? In the last message I wrote, I attached a screenshot of the bonsai terminal with a bunch of "fps= 29", should it be 30?)

Unless explicitly stated, all tests have been done with the visualizer closed.

Thanks!

By the way, I must say that the camera is located in a room I can't access whenever I want and sometimes it may take a while for me to do these tests.

Regards,

Pau

[Gonçalo Lopes]

Hi Pau,

Sorry that this one is proving a tough nut to crack. I'd like to ask a couple of questions about the computer hardware and software:

1. How many cores does the computer have? You mention it is an Intel i5, but is it dual-core, quad-core?

2. You mentioned other softwares were able to encode with a correct duration, can I ask their names? Is it just FlyCapture saving raw video? Or something else accessing the point grey camera? Also, do they consistently report correct durations every single time, or do they vary like Bonsai?

I am asking this since I was looking back through the previous test results you reported on removing the VideoWriter entirely and now they are striking me as really odd...

Test #1 reports zero frame drops in 2 minutes, but then test #2 reports 538 in 8 minutes. 538 frames@30Hz is basically 17s, which is roughly on the same scale of the dropped duration you have been reporting, and if this was with the visualizer closed, then there was really no processing at all applied to the frames.

The fact that you can drop so many frames even with no processing is starting to make me think that the frame drops are not coming from the video encoding process at all, and indeed changing the video encoding procedure seemed to help very little. This looks like some driver interaction issue.

I will send you an alternative version of the PointGrey package to more definitively test these hypotheses. Thanks for your patience running these tests, hopefully we'll get at the root of the issue soon.

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/c1e5069f-a057-497a-873d-b5ed21ab971c%40googlegroups.com.

[Pau Nebot]

Thanks for the quick answer!

My computer has a Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz. It has 4 cores.

I'm getting the right duration recording with FlyCap2 2.10.3.266, from PointGrey. It is also saving video in H264 format, 30fps. No duration shift has been reported yet. The diagnostics part of the software shows the warning: "link recovery count (camera)" eventually (not frequent at all, sometimes it even works flawless). I'm talking with PointGrey's support too to get more insight about why this appears.

To sum up, we're using FlyCap2 software as a temporary measure until we can trust bonsai's video duration. The output we're getting keeps correct video durations. It even counts frames and shows them in real time while recording, so we can check if any of them has been lost.

I agree with your ideas from tests 1 and 2. Losing frames in a workflow with just a PointGrey input node and a csv writer shows that the problem's nature must be different.

Thank you for your time and attention. I'll try to do longer tests and improve the accuracy of the results.

El viernes, 17 de marzo de 2017, 10:38:01 (UTC+1), goncaloclopes escribió:

Hi Pau,

[Eirinn Mackay]

Hi Pau, Goncalo,
Just a thought - the dropped frames don't seem to be proportional to the video length. If you can get Bonsai to report the number of dropped frames in real-time, we might see a "spike" which corresponds to some other activity, such as the computer's antivirus scanner, or conflict from another peripheral on the USB bus. It'd be good to watch the CPU meter during this test.
Cheers,
Eirinn