Error when trying to generate noise

[Bram Koster]

I got error below during the training step, apparently when the noise is added.

Running tensorflow 2.5.0 cuda 11.2, rtx 2080 ti

What can I check to pinpoint the problem?

0.1245
148/150 [============================>.] - ETA: 0s - loss: 0.1245 - mse: 0.0265 - mae: 0.1245
150/150 [==============================] - ETA: 0s - loss: 0.1243 - mse: 0.0264 - mae: 0.1243
150/150 [==============================] - 9s 60ms/step - loss: 0.1243 - mse: 0.0264 - mae: 0.1243 - val_loss: 0.1251 - val_mse: 0.0268 - val_mae: 0.1251
03-20 13:38:10, INFO     Done training!
03-20 13:38:10, INFO     Done Iteration10!
03-20 13:38:10, INFO     Start Iteration11!
03-20 13:38:10, INFO     Start predicting subtomograms!
03-20 13:38:18, INFO     Done predicting subtomograms!
hamming
generating large noise volume; mode: hamming
03-20 14:18:49, ERROR    Traceback (most recent call last):
  File "/home/ajkoster/software/scipion/software/em/isonet-0.2.1/IsoNet/bin/refine.py", line 105, in run
    make_noise_folder(args.noise_dir,args.noise_mode,args.cube_size,num_noise_volume,ncpus=args.preprocessing_ncpus)
  File "/home/ajkoster/software/scipion/software/em/isonet-0.2.1/IsoNet/util/noise_generator.py", line 119, in make_noise_folder
    NoiseMap.refresh(large_side, noise_filter, ncpus)
  File "/home/ajkoster/software/scipion/software/em/isonet-0.2.1/IsoNet/util/noise_generator.py", line 73, in refresh
    NoiseMap.noise_map = simulate_noise([size_big, filter, ncpus])
  File "/home/ajkoster/software/scipion/software/em/isonet-0.2.1/IsoNet/util/noise_generator.py", line 108, in simulate_noise
    res = p.map(part_iradon_hamming,sinograms)
  File "/data/kosterlab/data/ajkoster/miniconda3/envs/isonet-0.2.1/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/data/kosterlab/data/ajkoster/miniconda3/envs/isonet-0.2.1/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],

[YUNTAO LIU]

Hi Bram,

Whether this error happens only when you using hamming filter? How about continuing from iter 10 but using ‘none’ filter?

--
You received this message because you are subscribed to the Google Groups "IsoNet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isonet+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isonet/b2de89cc-e7fc-4da0-a694-348d54673df8n%40googlegroups.com.

--

Best Regards,

Yuntao Liu,  Postdoc.

California NanoSystem Institute

University of California Los Angeles