broken trajectories, missing 2d data, dropping frames

[Floris van Breugel]

Hi Andrew,

(summary of our conversation from the prior day.. hopefully it posts properly..)

With braid 0.12-alpha5 (and alpha4) I get a lot of broken trajectories, even when tracking a high contrast LED wand. The broken trajectories are not just 3D breaks, but ~25 missing frames of 2D data. This happens at regular intervals. 

This is on our big system, running ubuntu 20, 20x 1920x1200 basler ace cameras spread over two machines. 

In the braid webui I also see the frames_collected drop from the nominal framerate to much lower for a brief moment at a regular interval, but only when actively saving braid data. The issue seems to be related to the amount of data that is being saved. When running at 100 fps with no objects it happens at a ~5 sec interval, when running at 50 fps with no objects it happens at ~10 sec interval. Running at 100 fps with lots of objects it happens at ~2 sec interval. 

Seems likely to be related to how/when data buffers are being written.

My quick fix: since we are already using ROS and publishing braid data to ROS, I wrote a ROS node to save braid data to an h5 file. If anyone needs a quick fix they can use this node:

https://github.com/vanbreugel-lab/braid_tools/blob/main/nodes/braid_save_data_to_hdf5.py

This requires that the braid_ros_listener node is also running. 

Example of loading saved data is here: 

https://github.com/vanbreugel-lab/braid_tools/blob/main/analysis/example_load_braid_hdf5_to_pandas.ipynb

- Floris

[Andrew Straw]

Hi Floris,

Thanks for the write-up and testing.

Based on your data and analysis, I guess the problem is that when Braid receives the data from the network, it sends it via an internal channel — with a limited capacity — to the thread which actually does the saving to disk. This is almost the very first thing that happens when Braid receives data to “ensure" it does save that data, no matter what else happens. I guess when the data is actually being saved to disk that this channel is getting full. I now boosted the default capacity of the channel from 10 to 10000 and made a new alpha release 0.12.0-alpha.6 , available as usual at https://github.com/strawlab/strand-braid/releases/tag/0.12.0-alpha.6 . Furthermore, the size of this channel is now configurable with the new parameter `write_buffer_size_num_messages` in the `[mainbrain]` section of the Braid `.toml` configuration file.

Can you try this new alpha release and see if it solves the problem?

-Andrew

--
You received this message because you are subscribed to the Google Groups "multi-camera software from the Straw Lab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to multicams+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/multicams/d9cf9c93-fbcb-47f6-b235-8511be6bc022n%40googlegroups.com.

[Floris van Breugel]

Hi Andrew,

That seems to have done the trick, thanks!

I confirmed that I no longer see the dropping "frames_collected" in the webui, with lights out, lights on, and many objects. And tracking an LED is perfectly continuous!

- Floris

--

Floris van Breugel   |   http://www.florisvanbreugel.com
Assistant Professor of Mechanical Engineering 

& Graduate Program for Neuroscience

University of Nevada, Reno

Wildlife and Landscape Photography

Galleries: http://www.ArtInNaturePhotography.com/
Blog: http://www.ArtInNaturePhotography.com/wordpress/