convnet autoboxing Train error

[Jacob Croft]

Hello,

When I click "train" in the convnet autoboxing (tomography) program, the GUI windows all close and I get the following error. Do you know what could be causing it?

Best,

Jake

Error:

Qt: Session management error: None of the authentication protocols specified are supported
2025-07-23 11:18:34.603357: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Reading tomograms/BG505_SOSIP-24jan30a-TS69__bin4.hdf...
New network..
/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/initializers/initializers_v2.py:120: UserWarning: The initializer TruncatedNormal is unseeded and being called multiple times, which will return identical values  each time (even if the initializer is unseeded). Please update your code to provide a seed to the initializer, or avoid using the same initalizer instance more than once.
  warnings.warn(
Preparing training set...
False
Training...
Traceback (most recent call last):
  File "/opt/applications/eman/2.99.47/bin/e2spt_boxer_convnet.py", line 695, in train_nnet
    self.nnet.do_training(
  File "/opt/applications/eman/2.99.47/bin/e2spt_boxer_convnet.py", line 155, in do_training
    opt.apply_gradients(zip(grad, wts))
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients
    return super().apply_gradients(grads_and_vars, name=name)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients
    iteration = self._internal_apply_gradients(grads_and_vars)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
    return tf.__internal__.distribute.interim.maybe_merge_call(
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/distribute/merge_call_interim.py", line 51, in maybe_merge_call
    return fn(strategy, *args, **kwargs)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
    distribution.extended.update(
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2637, in update
    return self._update(var, fn, args, kwargs, group)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/distribute/distribute_lib.py", line 3710, in _update
    return self._update_non_slot(var, fn, (var,) + tuple(args), kwargs, group)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/distribute/distribute_lib.py", line 3716, in _update_non_slot
    result = fn(*args, **kwargs)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py", line 595, in wrapper
    return func(*args, **kwargs)
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
    return self._update_step_xla(grad, var, id(self._var_key(var)))
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/opt/applications/eman/2.99.47/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: libdevice not found at ./libdevice.10.bc [Op:__inference__update_step_xla_337]
bash-4.2$

[Steve Ludtke]

Looks like a library version conflict of some sort. First thing to check is to make sure you don’t have LD_LIBRARY_PATH set in your shell. 

I assume you’re using a fairly recent eman version and you have the conda environment activated?

Sent from my iPhone

On Jul 23, 2025, at 2:29 PM, Jacob Croft <jac...@uw.edu> wrote:

Hello,

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/0174cf1b-e137-45bd-9292-528cc89937b6n%40googlegroups.com.

[Jacob Croft]

I checked like this, could this be causing a problem:

echo $LD_LIBRARY_PATH
/opt/applications/openmpi/4.0.2/gnu/lib:/opt/applications/openmpi/4.0.2/gnu/lib/openmpi

And yes, I am using eman/2.99.67 . Not sure what you mean when you ask if the conda environment is activated, Is that something that occurs during installation? I don't actually manage the installation of software on our lab computers, I can check with the person who manages and see if that means something to them.

Best,

Jake

[Steve Ludtke]

Ok, on a computer rather than my phone now.

As long as the only libraries in those openmpi folders are openMPI libraries, LD_LIBRARY_PATH shouldn’t be causing the issue. 

 You say you are using EMAN2.99.67, but the error message references  2.99.47. 

The way your path is set up implies a non-standard installation strategy. You clearly have a problem with the tensorflow version installed to work with EMAN2. You may need to seek local help with whomever did the installation.

To view this discussion visit https://groups.google.com/d/msgid/eman2/47c59411-c2f6-4c3e-b2c0-56992d1b338en%40googlegroups.com.

[Jacob Croft]

Good catch about 2.99.47, I tried it on both .67 and .47 (it previously worked on .47 several months ago so I was trying it again but something seems to be messed up).

Thanks for the input, we will work on installing the correct tensorflow version.

Best,

Jake

To view this discussion visit https://groups.google.com/d/msgid/eman2/D56846BA-097F-4108-AD3E-0EA5B2973376%40gmail.com.

[Muyuan Chen]

I think there is some communication issue between tensorflow and CUDA and the GPU? Does nvidia-smi work from the terminal? Normally tensorflow should say something about GPU devices if it detects any. In theory it should also work on CPUs when GPU is not available, but I don't test it regularly and sometimes it crashes as dependencies upgrades. Does test_tensorflow.py in EMAN2 examples work? 

To view this discussion visit https://groups.google.com/d/msgid/eman2/CAOa2YHAQxJaLV7DE-e8FHyN1hyZpkgyKuDfqDMtm91VeuWQpDQ%40mail.gmail.com.