tracking aborts with leaked folder object warning

[johanne...@gmail.com]

Hi,

at a later stage of tracking of a 4 hour, 30fps, 18animal video, I get a a warning about leaked folder object and the process aborts.

the last lines of output are below. Does this indicate insufficient memory?

I am saving pixels to RAM, segmentation images to DISK and I experienced this both in v3 and v4, although I am not sure now if the problem occured at the same stage for v3 and v4. The output below is for v4.

assigning fragment identifier: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 569077/569077 [00:25<00:00, 22062.62it/s]
[INFO    ] __main__.list_of_blobs                   number_of_individual_fragments, 40465
[INFO    ] __main__.list_of_blobs                   number_of_crossing_fragments: 11005
[INFO    ] __main__.list_of_blobs                   total number of fragments: 51470
creating list of fragments: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 569077/569077 [01:19<00:00, 7116.52it/s]
[INFO    ] __main__.list_of_fragments               getting coexisting individual fragments indices
[INFO    ] __main__.list_of_fragments               saving list of fragments at /home/jlarsch/Documents/zebrafish/20200218_01_dn_freeSwimming/session_test/preprocessing/fragments.npy
[INFO    ] __main__.list_of_blobs                   saving blobs list at /home/jlarsch/Documents/zebrafish/20200218_01_dn_freeSwimming/session_test/preprocessing/blobs_collection.npy
/home/jlarsch/miniconda3/envs/idtrackerai_dev/lib/python3.6/site-packages/joblib/externals/loky/backend/resource_tracker.py:320: UserWarning: resource_tracker: There appear to be 6 leaked semlock objects to clean up at shutdown
  (len(rtype_registry), rtype))
/home/jlarsch/miniconda3/envs/idtrackerai_dev/lib/python3.6/site-packages/joblib/externals/loky/backend/resource_tracker.py:320: UserWarning: resource_tracker: There appear to be 76 leaked folder objects to clean up at shutdown
  (len(rtype_registry), rtype))
Killed

[Paco Romero]

Hi Johannes, 

I've seen this error before. I have seen a similar one but in parts of the code that are parallelized. I am not sure why this error occurs at this point of the tracking pipeline. It could be that the computer has a lot of memory in use from previous processes and that at this point it has not been cleaned yet. 

I can recommend two things:

1.- Monitor the RAM memory while tracking to see if the problem is there.

2.- Try to lower the number of threads used during parallel computations using the advanced parameters:

NUMBER_OF_JOBS_FOR_SEGMENTATION

NUMBER_OF_JOBS_FOR_SETTING_ID_IMAGES

3.- If the size of your animals in pixels is above 500 pxs/animal, try to use the resolution reduction parameter to downsample the video. This will generate smaller lists of pixels and smaller segmentation images and identification images, using less RAM. If you stay in the range of ~150 pxs/animal, the accuracy of the tracking should still be high. 

I hope this helps, 

We will explore ways of monitoring the resources and fixing this issues in the next versions. 

It would be great if you could report this error in a Gitlab issue, enphasizing the specs of the computer, OS, idtracker.ai version, length of the video, number of animals, size of animals, the values of the advance parameters.

Thanks for reporting this problem

Francisco Romero-Ferrero

PhD student at Collective Behaviour Lab, Champalimaud Research
Av. de Brasilia, Doca de Pedrouços
1400-038 Lisboa, Portugal

--
You received this message because you are subscribed to the Google Groups "idtracker.ai users group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to idtrackerai_us...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/idtrackerai_users/07236fcf-d576-45d0-8db8-a0a20b0a6f4cn%40googlegroups.com.