Multiple processes
Jack Waters
I'm encountering problems when calling multiple processes in SIMA. Problems occur with motion correction, segmentation and extraction.
Windows, 8 CPUs, Anaconda, SIMA 1.3.0, Example data set from http://www.losonczylab.org/workflow_data.zip
Motion correction
Either of the following approaches is successful:
sima.motion.HiddenMarkov2D(max_displacement=[30, 30], n_processes=1)
or
sima.motion.PlaneTranslation2D(max_displacement=[30,30], method='correlation', n_processes=1)
but either command produces an EOF error when called across more than one CPU:
sima.motion.HiddenMarkov2D(max_displacement=[30, 30], n_processes=2)
or
sima.motion.PlaneTranslation2D(max_displacement=[30,30], method='correlation', n_processes=2)
Full code for motion correction section:
dataset_path = 'dataset.sima'
correction_approach = sima.motion.PlaneTranslation2D(max_displacement=[30,30], method='correlation', n_processes=1)
dataset = correction_approach.correct(sequences, dataset_path)
output_filenames = [[[channel.replace('.tif', '_corrected.tif') for channel in cycle]] for cycle in tiff_filenames]
dataset.export_frames(output_filenames, fill_gaps=True) # export corrected frames
Full error message:
EOFError Traceback (most recent call last)
<ipython-input-9-0ea84c33c635> in <module>()
18
19 if motion_correction_required:
---> 20 dataset = correction_approach.correct(sequences, dataset_path)
21 output_filenames = [[[channel.replace('.tif', '_corrected.tif') for channel in cycle]] for cycle in tiff_filenames]
22 print("Output filenames:\n", output_filenames) # print resulting filenames
C:\Anaconda\lib\site-packages\sima\motion\motion.pyc in correct(self, dataset, savedir, channel_names, info, correction_channels, trim_criterion)
114 else:
115 mc_sequences = sequences
--> 116 displacements = self.estimate(sima.ImagingDataset(mc_sequences, None))
117 disp_dim = displacements[0].shape[-1]
118 max_disp = np.max(list(it.chain.from_iterable(d.reshape(-1, disp_dim)
C:\Anaconda\lib\site-packages\sima\motion\motion.pyc in estimate(self, dataset)
57 displacements : list of ndarray of int
58 """
---> 59 shifts = self._estimate(dataset)
60 assert np.any(np.all(x is not np.ma.masked for x in shift)
61 for shift in it.chain.from_iterable(shifts))
C:\Anaconda\lib\site-packages\sima\motion\frame_align.pyc in _estimate(self, dataset)
68 return _frame_alignment_base(
69 dataset, params['max_displacement'], params['method'],
---> 70 params['n_processes'])[0]
71
72
C:\Anaconda\lib\site-packages\sima\motion\frame_align.pyc in _frame_alignment_base(dataset, max_displacement, method, n_processes)
99 global lock
100 if n_processes > 1:
--> 101 namespace = multiprocessing.Manager().Namespace()
102 else:
103 namespace = Struct()
C:\Anaconda\lib\multiprocessing\__init__.pyc in Manager()
97 from multiprocessing.managers import SyncManager
98 m = SyncManager()
---> 99 m.start()
100 return m
101
C:\Anaconda\lib\multiprocessing\managers.pyc in start(self, initializer, initargs)
526 # get address of server
527 writer.close()
--> 528 self._address = reader.recv()
529 reader.close()
530
EOFError:
Segmentation
Segmentation (STICA) with one process generates ROIs in ~20 minutes:
segmentation_approach = sima.segment.STICA(channel=0, components=75, verbose=True)
segmentation_approach.append(sima.segment.SparseROIsFromMasks())
dataset.segment(segmentation_approach, 'auto_ROIs')
Calling multiple CPUs generates no error, but segmentation runs indefinitely (interrupted after ~1 hour):
segmentation_approach = sima.segment.STICA(channel=0, components=75, verbose=True)
segmentation_approach.append(sima.segment.SparseROIsFromMasks(n_processes=6))
dataset.segment(segmentation_approach, 'auto_ROIs')
Extraction
Much like segmentation, extraction with one CPU is fast; with more than one CPU slower.
Extraction with one process completes in a few seconds minutes. Code:
dataset.extract()
With multiple CPUs, extraction runs indefinitely (terminated after ~10 minutes. Code:
dataset.extract(n_processes=2)