Slow frame rate ouptut from Bonsai

[Reese Findley]

Hi,

PROJECT
I am a researcher at University of Oregon studying freely moving olfactory localization in mice. I am using a fly capture camera to track nose position. I have a program in Bonsai that does the following things:

GOALS
1) Produces a video window for the researcher to monitor the subject
2) Saves a raw video of the session
3) Tracks nose position and outputs the coordinates to my main control program in python

METHOD
The way I track nose position is by isolating a colored triangle on the head of the mouse, identifying the long major axis of the shape, identifying the two extremes of that axis, and then comparing each of those to the center of mass of the mouse. The furthest point from the center of mass is the nose position and is output to Python via OSC.

ISSUE
I have my fly capture camera running at 100 fps and I have confirmed that my python loop can run and save much faster than 100 Hz. However, my output from Bonsai is never more than 40 fps. I had to set my video writer to 30 fps in Bonsai in order for the program to even work without crashing and the coordinates being sent to python do not arrive at more than 40 fps. I have attached my Bonsai program -- any ideas on how I can increase the frame rate and, consequently, the frequency of coordinate output? I would like it to run at 100 fps minimum - my camera is capable of 150 fps.

Thanks for your time!

Best, Reese

Teresa Findley
Smear Lab, University of Oregon

[Gonçalo Lopes]

Hi Reese and welcome to the forums! I will try and help you debug the situation. In general the approach will be to benchmark each of the individual processing components until we understand which part exactly is causing the slowdown.

I am attaching a modified version of your workflow where I simply grouped a couple of common nodes at the beginning (all the branches seem to start with the exact same crop) and replaced the final Zip operation on the AnalogInput node with a CombineLatest. This last modification is potentially relevant, since the Arduino input may be slower or simply out of phase with the camera input and the synchronization enforced by Zip may be causing memory and/or performance issues.

You should give it a try, but still I don't believe this will necessarily address your problem.

I'm curious about your report that changing the VideoWriter frame rate property prevented the workflow from crashing. This sounds strange because that value only controls the video playback speed and is not used to influence the encoding process at all, which always happens at the frame rate of the input video stream. Can you confirm that this still happens with your final workflow?

Also, you mentioned that you can run your python loop faster than 100Hz. Can I ask how these measurements were performed? Were they run on Bonsai with offline video streams or outside in another python environment? If outside, how was the image processing handled?

Finally, to help debugging, it would help to use the PointGrey's embedded hardware frame counter in order to have an unambiguous way of measuring capture performance. You can record this by going to "Advanced Camera Settings" tab in the PointGrey camera configuration dialog and making sure that the "Frame counter" checkbox is checked under the "Embedded Image Information" section. I have included a node in the modified workflow that will extract this information and record it onto a CSV file called "frame_counter.csv". You will be able to know precisely how many frames are being dropped and when by looking at the difference between consecutive frame counter values in each line of the CSV file. This is the preferred way of debugging performance in PointGrey video acquisition in general.

In any case, assuming no other issue, I can see three potential bottlenecks in your image processing pipeline:

1) Video recording: encoding large resolution color video can be an extremely CPU intensive process. I would say this is probably the most costly step in your whole workflow. Usually if you have multiple fast CPUs on your machine, it should still be able to do it with your resolution. However, one easy debug step would be to remove the VideoWriter and record only the hardware frame counter so we can have an idea of how much this is impacting the rest of your processing.

If this turns out to be a problem, you can choose either to reduce your resolution, encode in grayscale, or change your video encoding strategy, possibly changing the codec to a faster encoding with FFmpeg as described in this post. Let me know if this turns out to be the issue and I can help you transition to whatever method fits best.

2) How are you converting the colored video from the PointGrey? Usually these cameras acquire their raw color data using a Bayer mask and then use a debayering algorithm to convert back to color. Interestingly, we have discovered that the debayering algorithm running on the PointGrey driver is somehow slower than taking the raw bayer image and running the conversion directly in Bonsai. If you upgrade your FlyCapture package to the latest version (2.2.0) you will be able to set the color processing method of the FlyCapture node to "NoColorProcessing". This will retrieve the raw bayer array and you can then convert it manually back to color in Bonsai using the ConvertColor node with one of the Bayer*G2Bgr options.

3) I realize you need to run two processing algorithms in parallel on each full frame, one in color space, and the other in grayscale space. It would be good to know the relative costs of each of these branches to your overall processing. The easiest way to do this is to systematically delete each of these branches and run the workflow with only one of them at a time and record the influence in the hardware frame counter.

It would be good to know the impact of each of these strategies in the results recorded in the "frame_counter.csv" file. If you could run some experiments around these lines and let me know the results that would be immensely helpful to try and understand what is going on.

Hope this helps!

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/bonsai-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/df8ff3a4-8001-4d11-a6b0-18beed954289%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Reese Findley]

This is awesome, thank you for all your suggestions! Here are the results of testing the following (two csv files per test are attached): 

1) Changing framerate in the video writer does not crash the system anymore -- it must have been something wrong with my old program while writing it. 

2) My python program works by using the coordinates I send from Bonsai. So I tested it's speed by just sending the computer mouse coordinates from Bonsai (eliminating image processing in Bonsai) to Python and counted the number of samples saved by my python program divided by the time I ran the program (timestamped in Python). I did it a while ago, but I believe the slowest it ran was about 800 Hz. 

3) Eliminating the Video Writer seemed to help, but there are still quite a few frames being skipped. Would it be useful to save in only one color (eliminating the other 3 data points per pixel)? I should still be able to re-calculate the nose point using a red or possibly green image. 

4) I took out color processing and did it directly through Bonsai and I believe it helped, but again, it seems like it's skipping quite a few frames still. 

5) Removing the grayscale branch or the hsv thresholding branch helped immensely, but also took away my ability to track the nose position. It also meant I had to delete the display window that shows how well the point is tracking the nose. 

6) I added a test and just deleted that display window. Again, seemed to help, but still resulted in quite a few frames missed. 

Attached are all the tests I have run so far and the script I ran them on. They are titled so that the process they are labeled with is the one I removed for that test. I started with removing color processing and then ran all other tests with color processing being done in Bonsai. Grayscale and hsv_thresholding are both missing the display window (because the window requires input from both). Since removing hsv_thresholding (and therefore most of my image processing) seemed to help the most, I will look into ways to simplify my python transforms a bit. 

Again, thank you so much for your quick/in depth response! Let me know if you think of anything else! 

Cheers, Reese

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

[Gonçalo Lopes]

Hi Reese,

From your results it definitely looks like the HsvThreshold branch is your biggest bottleneck. Two further ideas to consider:

1) What exact color is the marker that you are tracking? If it's pure red/green/blue you may be able to track it simply with a threshold on the isolated R, G or B channel. This would be much faster than applying a range threshold on a color image.

2) Did you check the option on the PointGrey driver installer to disable cpu idle states? FlyCapture comes with a utility to disable changes to the CPU power state introduced in newer chipsets that they claim provides better performance and image acquisition stability. We have observed enhanced performance in some situations when the idle states are disabled, so maybe it would help also a bit in your situation.

You can read a bit more about the utility (and instructions on how to use it) here: https://www.ptgrey.com/KB/10367#PowerIssues

To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/bonsai-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/c0882e8a-f331-41e3-a295-ec72a3560814%40googlegroups.com.