Hardware requirement: CPU and GPU (CUDA)

[Antoine Cribellier]

Dear Braid-users,

I just updated strand_braid (to 0.11.0) and it's great that I can now run 5 cameras at full resolution (4Mpx). However I am encountering problems when I am trying to run more cameras (I am trying to see what is the maximum number of cameras I can use as well as the maximum nb of points I can track):

• I get a lot of frame drops when trying to track in 2D more than 50 points
.
• When I try to run more than 5 camera, Braid stops with the followings errors:

ERROR strand_cam: Channel full sending frame to process thread. Dropping frame data.  Error: SingleFrameError(Pylon Error 3791651346: Payload data has been discarded. Payload data can be discarded by the camera device if the available bandwidth is insufficient.)

 

From my understanding, in strand_braid all the image processing is done on the CPU: one core is used for the analysis of one image (so max one core per camera) to do the 2D tracking and another does the 3d reconstruction. Am I right? If so, to be able to track many different insects as the same time, I guess the best would be to use a CPU with at least the nb of camera +1 core and which each of these core being the fastest as possible. Remco was telling me that maybe the cache memory of the CPU was also an important parameter. Do you know if it is?

 

Finally, I noticed this line when starting strand_braid:

strand_cam: CUDA and nvidia-encode libraries not loaded: dynlink-cuda returned error `dynamic library `libcuda.so` could not be loaded: `libcuda.so: cannot open shared object file: No such file or directory``

Can the GPU be used for tracking or is it only to save videos? If it's not used for tracking, is there any particular reason why not? Maybe it's a legacy from flydra?

If it can be used for tracking, does that mean that we need to install CUDA?

 

Thanks,

 

Antoine Cribellier

[Andrew Straw]

Dear Antoine,

I'm glad that acquisition seems to work now with more cameras with release 0.11.0 - it's good to hear that the adjustments in that direction seem to be successful up to this point.

Do you see the errors go away if you disable image processing or if nothing is happening in the camera field-of-view? If so, this does indeed mean that you are reaching the limit of your CPU to keep up with the image processing. Yes a faster system (mainly CPU and memory performance as well as more cores and larger cache) will likely help, even in the absence of further software changes. I have not done much benchmarking, so I cannot tell you exactly which computer parameters are most relevant. I guess any computer game benchmarks that are CPU limited will be highly relevant. I guess that indeed having at least N+1 cores where N is the number of cameras is also likely to be useful. Perhaps even N+2 or 3 so the kernel and other programs have dedicated cores, as well. In theory, Braid should run quite parallel when more cameras are present, meaning that indeed adding more CPU cores (at least up to n+1 or maybe n+2) should help. Again, however, I have not benchmarked enough to be absolutely confident in this.

Given how little performance investigation into Braid I have done so far, it seems quite possible that some simple profiling will reveal a few hot spots that can be more-or-less easily improved. This would be a useful thing to do...

There would be ways to potentially make Braid run "wider." For example, we could imagine splitting images for image processing into sub-images processed in parallel. I would really want to have some good benchmarks and profiling in place to track performance before spending such efforts, though.

The newly reintroduced remote camera feature can also help by offloading image processing from one or more cameras to another computer (like the old flydra used to do). Very likely most of the CPU time is indeed spent in image processing rather than the 3D math. That said, once there are a lot of 3D objects, the 3D math may also start taking a lot of time.

Once we can track previously saved videos (issue: https://github.com/strawlab/strand-braid/issues/9 ) we should be able to work on optimizing all this offline, which will be a lot better for iterative and incremental development than revising a system which can only run "online". (That said, the separate parts - 2D image processing and 3D math - can already be optimized separately.)

Regarding GPUs, there is currently no image processing done on the GPU other than the ability to use NVENC to encode compressed videos without further taxing the CPU if you have an Nvidia card. If you have an Nvidia card but cannot save h264 encoded videos, then indeed you should install the latest nvidia drivers and perhaps CUDA. I would not like to make an Nvidia GPU a hard requirement (especially with pricing and availability as it is now) and, combined with a limited time budget available to work on this, I'm not anticipating offloading image processing to the GPU in the near term. That said, in the next months we do hope to experiment with object detection frameworks such as YOLO which will use the GPU for tracking. A primary consideration will of course be keeping latency low. If this direction is something you have time and interest to work on, I'm happy to discuss this further. In addition to potential benefits in terms of tracking in more complex visual environments, potentially this could also make a single CPU system scale to more cameras if the GPU can take much of the compute load away from the CPU.

Best,

Andrew

[Andrew Straw]

Hi Antoine,

I had a couple more thoughts. First, the "Live View" pane in Strand Cam is not done particularly efficiently and you can save some CPU cycles by closing the pane. Secondly, you could reduce the frame rate of your triggering to a level that works with all cameras. (The tracking latency won't be changed by this although the quality of the tracking may be affected as, obviously, there is less data available to estimate the animals' positions.)

Best,

Andrew

[rpmpi...@gmail.com]

Hi  Andrew,

This morning I did some testing with six cameras at full resolution (4MP). The CPU power needed for the image processing is now for us the limiting factor. Errors go indeed away when the tracking is disable or when the number of tracked points are reduced, updating the background image had also an effect on this. We will look into the option of the remote camera feature. However, I did a quick test by using a single camera and track 50 points at the same time but this also results in image processing delays when the number of points change quickly (light on/off). We have to investigate this a bit further by reducing the resolution or lowering the frame rate. Would it reduce the CPU load when the new points don't appear suddenly , but are tracked already for some time?  Shall we propose a plan to systematically test the tracker limitations?

Best, Remco

Op maandag 6 december 2021 om 09:12:22 UTC+1 schreef dr.andr...@gmail.com:

[Andrew Straw]

Hi Remco,

I'm glad that the fundamentals seem to be working. Regarding CPU usage, you can gain insight by running Braid without a calibration. In this case, the 3D tracking will not run. This can help distinguish the relative load of the image processing and 2D feature detection versus the 3D math.

If 2d points are not associated to an existing 3D object and there is a calibration, then indeed there is extra computational performed to check if a new 3D object can be "born". This particular computation has combinatorial scaling with the number of points detected and there is currently no shortcut to abort the computation if it is taking too long. So indeed you could be triggering a pathological case with your testing. I presume that making new objects appear incrementally rather than all at once should be much better behaved, and it would also be easy to add a shortcut that limits the amount of time (or number of combinations or some other criterion) that Braid will spend testing new potential 3D objects from combinations of 2D points.

Perhaps just adding such a limiter would solve your use case - what do you think? I think I could create a new parameter that controls this quite simply.

-Andrew

--
You received this message because you are subscribed to the Google Groups "multi-camera software from the Straw Lab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to multicams+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/multicams/72b8392f-40e7-49e8-9aaa-a5d9ce2fa036n%40googlegroups.com.

[Antoine Cribellier]

Hi Andrew,

Thank you very much for your detailed answers.

All the tests that we did with this PC were done without a calibration, but often we did them by turning the light off and on (so modifying the entire background) because our flies are not very active. That might have resulted in some image processing overload. So we will work on changing our test setup in order to do more consistent testing, maybe by just tracking moving points on a computer screen (which would allow us to do repeatable tests with various known amount of points).

We are also looking into buying a new PC with a Intel Core-i9, which should have the fastest cores available at the moment (x16).

I will keep you updated.

Best,

Antoine

[rpmpi...@gmail.com]

Hi,

To add to this idea to use a computer screen for consistent testing of a series of points. We should use a monitor for gaming because of the low update frequency of regular monitors.

Best, Remco

Op maandag 6 december 2021 om 16:18:46 UTC+1 schreef dr.andr...@gmail.com:

[Antoine Cribellier]

Hello,

I finally had the time to test systematically our system by simulating moving insects with a computer screen. I focused on when I started to see many frame drops and wrote down the associated number of moving object for various configurations (resolution, # of camera and framerate).  All of this while testing only 2d tracking (without a calibration). You can find all my results attached. 

My main findings are:

- The maximum number of points that we can track (without frame drops) at full framerate and full resolution is 10. It goes to 60 with a binning of 2.

- The number of cameras doesn't change the minimum number of points tracked that results in frame drops

- There is a non-linear relationship between the number of pixel per second and the minimum number of points tracked that  results in frame drops, lower resolutions allow the tracking of more points proportionally to larger resolutions. And this even if when I corrected for the size of the feature window (so making it smaller with lower resolutions).

Finally, for all cases, I checked what happened when the frame drops appeared. I found that in all cases, two CPU were maxed out at 100% usage. Tracking a bit less points was possible and resulted in only one CPU at 100% usage (with usually another being at 80-90%).

I am not entirely sure how to interpret these results, but maybe you will find them useful. We ordered a PC with an intel core i9, and I will do the same tests with it when we will receive it.

Best,

Antoine

 

[Andrew Straw]

Thanks Antoine that is very useful. What were the PC specs, especially which model CPU?

Sent from my mobile device.

On 13. Dec 2021, at 18:09, Antoine Cribellier <antoine.c...@wanadoo.fr> wrote:

Hello,

To view this discussion on the web visit https://groups.google.com/d/msgid/multicams/05068ab9-0f23-4b1f-adae-bebaf80d8477n%40googlegroups.com.

[Antoine Cribellier]

Good evening Andrew,

The PC is a HP Z840 Workstation, it has a Intel Xeon E5-2673v4 2.3GHz 20 Core. 

Best,

Antoine

[Andrew Straw]

Thanks for that update. I am also not sure how to interpret the results. There does seem to be some "per point" overhead but I would have expected that the performance is limited more by pixels/second within a given camera (and independent of number of cameras at least up to number of CPUs). Thus it seems potentially fruitful to work on reducing the per-point overhead or to parallelize the per-point processing (at least for computers with many cores such as yours).

-Andrew

[Antoine Cribellier]

Dear Andrew,

We received the new PC with a Intel Core i9-12900K, and I just run the same tests with it (still without calibration). Here are attached screenshots of the results.

We achieve tracking of around 2.5 time more 2d points than with the previous PC (Intel Xeon E5-2673). Note: I haven't tried to run braid specifically with the performance-core of the i9 yet (but I guess it should do that automatically).

I noticed that frame drops appeared "earlier" than with the previous PC though. By earlier, I mean that it happened when one core was at 100% usage and another around 35% (while it was around 80% for the second core before). I am not sure what this means.

Also, when I tried to track more than 200 pts, I often got crashes (and the "Well this is embarrassing" message), as if Braid could just not handle that many point. I attached a few logs.

I hope this is useful.

Best,

Antoine