IsoNet on Ubuntu 22 crashing during Refine

[felix.l...@gmail.com]

Hi guys,

I try to set up IsoNet on our machine with Ubuntu22.04 and I followed the recommended installation of TF in conda (python 3.9 or 3.10 - didn't make a difference) with CUDA 11.8 and cuDNN 8.6. Tensorflow test commands are successful but whenever reaching the refine step the system gets stuck at epoch 1/10 and crashes with internal error. 

I am using a single RTX3080 card and I already tried lowering to cube_size 32 but this doesn't work. I suspect that the workload is not distributed correctly and leads to a memory overload or something similar. 

Does anybody know whether we need a different configuration of either TF or IsoNet to make it run on Ubuntu or is there something I am missing entirely? 

Thank you for your help and all the best,

Felix 

[YUNTAO LIU]

Hi Felix,

I previously had a desktop computer with 3080 running on a windows subsystem Ubuntu. I used an older version of tensorflow2.5 with the corresponding cuda and cudnn list here https://www.tensorflow.org/install/source#gpu . The specific configuration is in this link https://github.com/IsoNet-cryoET/IsoNet/issues/20#issuecomment-962257020

But as I remember, a single RTX3080 has 11GB VRAM, which can not fit the subtomos with batch_size >= 4 and cube_size >= 64. This will produce suboptimal results. 

--
You received this message because you are subscribed to the Google Groups "IsoNet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isonet+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isonet/cc1993cc-2e29-4621-8047-33f3d1d74839n%40googlegroups.com.

--

Best Regards,

Yuntao Liu,  Postdoc.

California NanoSystem Institute

University of California Los Angeles