dark predicted image

[Huaxin Yu]

Hi, 

I met a warning during prediction in the attached, 

the predicted tomogram is completely dark,

how can I overcome it? 

I tried different numbers of GPUs also the nn from different iterations. 

[YUNTAO LIU]

Hi Huaxin,

I wonder whether the loss during the network training is NaN.

--
You received this message because you are subscribed to the Google Groups "IsoNet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isonet+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isonet/9a610d0a-a1cd-49fc-86ed-bc31c457c6fcn%40googlegroups.com.

--

Best Regards,

Yuntao Liu,  Postdoc.

California NanoSystem Institute

University of California Los Angeles

[Huaxin Yu]

Hi Yuntao, 

Yes, I checked it, it is also NaN. 

The only thing I can remember is that I did not set defocus and left it as 0. 

Is this the reason caused the error? 

Thanks,

Huaxin

[YUNTAO LIU]

Hi Huaxin,

I currently don't have an exact solution. Are there any specific characteristics in your tomogram that might be contributing to the issue? For instance, are there extreme values or certain regions with constant values? You might also consider processing a subset or part of your tomograms to see if the refinement encounters any NaN problems.

To view this discussion on the web visit https://groups.google.com/d/msgid/isonet/cd2cb78d-143f-45d5-b5d6-79d6327c1e9an%40googlegroups.com.

[Huaxin Yu]

Hi Yuntao,

I crop the tomogram to a smaller size, 

/vast/palmer/apps/avx2/software/TensorFlow/2.5.0-fosscuda-2020b/lib/python3.8/site-packages/tensorflow/python/keras/utils/generic_utils.py:494: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.
  warnings.warn('Custom mask layers require a config and must override '

The error only occurs at the last epoch of iteration 1, does this indicate a mask problem? 

Huaxin

On Saturday, November 11, 2023 at 5:46:59 PM UTC-5 yun...@g.ucla.edu wrote:

[YUNTAO LIU]

Hi Huaxin,

The CustomMaskWarning should not be related to the NaN problem and can be ignored. 

I guess that the NaN is caused by one bad subtomogram and will not always happen at the end of iteration 1. I am not sure but it is most likely that the tomogram has some abnormal values or regions. 

To view this discussion on the web visit https://groups.google.com/d/msgid/isonet/7978cf7d-8529-4404-a7c8-13d2d311d738n%40googlegroups.com.

[YUNTAO LIU]

Hi Huaxin,

I recently learned from a user that a tomogram with some noise free regions will lead to NaN value.