OSError', OSError(24, 'Too many open files')

19 views
Skip to first unread message

Gil Raitses

unread,
Jun 5, 2024, 1:12:59 PMJun 5
to idtracker.ai users group
  1. I am getting this error persistantly in the 3rd Protocol. This time it was almost at the 90% pretraining mark. 

  2. 88.65% of the images have been used during pretraining   tracker.py:256
  3.          (if higher than 90.00% we stop pretraining)                            
  4.          New pretraining iteration, using the #16 global fragment tracker.py:241
  5.          Loading identification images from disk (24           pre_trainer.py:32
  6.          iterations). It took 0:00:00                                           
  7.          Creating training dataloader with 47211 images labeled     train.py:225
  8.          with 11 distinct classes                                               
  9.          Creating validation dataloader with 5251 images labeled    train.py:225
  10.          with 11 distinct classes                                               
  11.          Reinitializing only fully connected layers                 models.py:56
  12.          Entering the training loop...                              train.py:115
  13. 12:51:58 ERROR An error occurred, saving data before printing          run.py:74
  14.          traceback and exiting the program                                      
  15.          Saving Session object in /Users/gilraitses/Documents/Mirna    run.py:78
  16.          Lab/videos/session_Gr21a@Chrimson(3)_T_Re_Sq_219to436P_15_2_3          
  17.          #T_Bl_Sq_2to7P_15_1_3_202402131442_3x_1/session.json                   
  18.          ERROR Error while saving data: [Errno 24] Too many open      run.py:108
  19.          files: <'file.path'>
Here is the traceback log:

--- Logging error ---

Traceback (most recent call last):

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/run.py", line 43, in track_video

    self.list_of_fragments = tracker_API(

                             ^^^^^^^^^^^^

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/__init__.py", line 52, in tracker_API

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 51, in track

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 139, in accumulation_protocol

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 246, in pretrain

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/pre_trainer.py", line 78, in pretrain_global_fragment

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/network/train.py", line 119, in train_loop

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/network/train.py", line 163, in evaluate

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 434, in __iter__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 387, in _get_iterator

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1022, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/context.py", line 103, in Queue

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/queues.py", line 49, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/context.py", line 88, in BoundedSemaphore

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/synchronize.py", line 152, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/synchronize.py", line 57, in __init__

OSError: [Errno 24] Too many open files

--- Logging error ---

Traceback (most recent call last):

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/run.py", line 43, in track_video

    self.list_of_fragments = tracker_API(

                             ^^^^^^^^^^^^

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/__init__.py", line 52, in tracker_API

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 51, in track

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 139, in accumulation_protocol

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/tracker.py", line 246, in pretrain

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/tracker/pre_trainer.py", line 78, in pretrain_global_fragment

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/network/train.py", line 119, in train_loop

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/idtrackerai/base/network/train.py", line 163, in evaluate

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 434, in __iter__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 387, in _get_iterator

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1022, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/context.py", line 103, in Queue

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/queues.py", line 49, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/context.py", line 88, in BoundedSemaphore

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/synchronize.py", line 152, in __init__

  File "/opt/anaconda3/envs/idtrackerai/lib/python3.11/multiprocessing/synchronize.py", line 57, in __init__

OSError: [Errno 24] Too many open files



idtrackerai

unread,
Jun 5, 2024, 1:40:52 PMJun 5
to idtracker.ai users group
Hi,

Thanks for your feedback. Could you send the entire log file? I would like to check some information in there.
In any case, I know this error, it's related to this PyTorch issue. I added a patch on the last version 5.2.11 of idtrackerai to workaround it. The patch kind of worked, but I think it's still possible to find this bug on the last version of idtrackerai in really long Protocol 3 sessions. Please send me the log file and we'll see what can we do. Thanks.

Jordi
Reply all
Reply to author
Forward
0 new messages