Error for GMM-based refinement for spt
5 views
Skip to first unread message
Maia Azubel
Jun 14, 2024, 1:13:59 PMJun 14
to EMAN2
Hi Muyuan,
I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
Thanks,
Maia
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
Your Python version is: 3.7.9
command
e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
error
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Load 39660 2D particles
Loading 39660 particles of box size 512. shrink to 124
39000/39660 R 39660/39660
Data read complete
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
(39660, 5, 3)
Starting alignment...
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 173, in main
p2=xf2pts_3d_mult((p2, xfvar))
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
result = self._call(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 606, in _call
results = self._stateful_fn(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2362, in __call__
graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 968, in wrapper
raise e.ag_error_metadata.to_exception(e)
tensorflow.python.autograph.impl.api.StagingError: in converted code:
/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py:53 xf2pts_3d_mult *
return tf.vectorized_map(xf2pts_3d, args)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:394 vectorized_map
return pfor(loop_fn, batch_size)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:189 pfor
return f()
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:183 f
return _pfor_impl(loop_fn, iters, parallel_iterations=parallel_iterations)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:256 _pfor_impl
outputs.append(converter.convert(loop_fn_output))
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1280 convert
output = self._convert_helper(y)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1453 _convert_helper
if flags.FLAGS.op_conversion_fallback_to_while_loop:
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/platform/flags.py:84 __getattr__
wrapped(_sys.argv)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/absl/flags/_flagvalues.py:655 __call__
name, value, suggestions=suggestions)
UnrecognizedFlagError: Unknown command line flag 'ptclsin'
#######################
batch 46/47...
46/47 batches, 25704 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_046.lst --ptclsout gmm_18/tmp_output_046.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
2024-06-14 05:24:21.637490: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Load 25704 2D particles
Loading 25704 particles of box size 512. shrink to 124
25000/25704 R 25704/25704
Data read complete
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
(25704, 5, 3)
Starting alignment...
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 173, in main
p2=xf2pts_3d_mult((p2, xfvar))
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
result = self._call(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 606, in _call
results = self._stateful_fn(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2362, in __call__
graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2703, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2593, in _create_graph_function
capture_by_value=self._capture_by_value),
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 978, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
return weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 968, in wrapper
raise e.ag_error_metadata.to_exception(e)
tensorflow.python.autograph.impl.api.StagingError: in converted code:
/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py:53 xf2pts_3d_mult *
return tf.vectorized_map(xf2pts_3d, args)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:394 vectorized_map
return pfor(loop_fn, batch_size)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:189 pfor
return f()
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:183 f
return _pfor_impl(loop_fn, iters, parallel_iterations=parallel_iterations)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/control_flow_ops.py:256 _pfor_impl
outputs.append(converter.convert(loop_fn_output))
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1280 convert
output = self._convert_helper(y)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/ops/parallel_for/pfor.py:1453 _convert_helper
if flags.FLAGS.op_conversion_fallback_to_while_loop:
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/platform/flags.py:84 __getattr__
wrapped(_sys.argv)
/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/absl/flags/_flagvalues.py:655 __call__
name, value, suggestions=suggestions)
UnrecognizedFlagError: Unknown command line flag 'ptclsin'
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 147, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 118, in main
ptclsall.extend(load_lst_params(tmp[1]))
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2831, in load_lst_params
lsx=LSXFile(fsp,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2471, in __init__
if ifexists: raise Exception("Error: lst file {} does not exist".format(path))
Exception: Error: lst file gmm_18/tmp_output_000.lst does not exist
Exception ignored in: <function LSXFile.__del__ at 0x7f95c129a710>
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2537, in __del__
self.close()
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2548, in close
if self.ptr!=None :
AttributeError: 'LSXFile' object has no attribute 'ptr'
e2spa_make3d.py --input gmm_18/aliptcls2d_01_odd.lst --output gmm_18/threed_01_odd.hdf --keep 0.9 --outsize 256 --sym c1 --parallel thread:68
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 445, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 63, in main
tmp=EMData(options.input,0,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2883, in db_emd_init
self.read_image(*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2930, in db_read_image
return LSXFile(fsp).read_into_image(self,*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2641, in read_into_image
n,fsp,jsondict=self.read(N)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2593, in read
if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
IndexError: Attempt to read record 0 from #LSX gmm_18/aliptcls2d_01_odd.lst with 0 records
e2proc3d.py gmm_18/threed_01_odd.hdf gmm_18/threed_raw_odd.hdf --compressbits 12
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 856, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 174, in main
infile, _ = parse_infile_arg(infile)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 1090, in parse_infile_arg
raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'gmm_18/threed_01_odd.hdf' is not an existing regular file!
e2refine_postprocess.py --even gmm_18/threed_01_even.hdf --tophat localwiener --threads 68 --restarget 5.50 --align --sym c1
Use iteration number 1.
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 576, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 116, in main
hdr=EMData(evenfile,0,1)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2883, in db_emd_init
self.read_image(*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2935, in db_read_image
fsp, idxs = parse_infile_arg(fsp)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 1090, in parse_infile_arg
raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'gmm_18/threed_01_even.hdf' is not an existing regular file!
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 204, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 178, in main
r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 190, in calc_resolution
fsc=np.loadtxt(fscfile)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_18/fsc_masked_01.txt not found.
Muyuan Chen
Jun 14, 2024, 1:18:33 PMJun 14
to em...@googlegroups.com
The most notable issue is this:
2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Are you running on a machine without CUDA-able GPU? Or it just fails at the later iterations? Do you only get the error at batch 45, or the earlier batches give the same error as well?
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com.
Maia Azubel
Jun 14, 2024, 1:55:26 PMJun 14
to em...@googlegroups.com
We do have CUDA.
NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4
Last night I run it on a machine that has
Yes, there were error for earlier batches and also looking to find that I noticed also this error
batch 1/47...
1/47 batches, 38974 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_001.lst --ptclsout gmm_18/tmp_output_001.lst --model gmm_18/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
I've just tried a different machine
NVIDIA-SMI 545.23.08 Driver Version: 545.23.08 CUDA Version: 12.3
a got the following error messages (long, sorry)
e2gmm_spt_refine.py spt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
Loading refinement info from spt_141/threed_06.hdf
old refinement path spt_141, iteration 6.
Writing new refinement in gmm_17...
Using existing metadata files from the old refinement...
e2proclst.py spt_141/particle_info_3d.lst --create gmm_17/particle_info_3d.lst
Processing 94358 images in spt_141/particle_info_3d.lst
e2proclst.py spt_141/particle_info_2d.lst --create gmm_17/particle_info_2d.lst
Processing 3685466 images in spt_141/particle_info_2d.lst
e2proc3d.py spt_141/threed_06.hdf gmm_17/threed_00.hdf
e2proc3d.py spt_141/threed_06_even.hdf gmm_17/threed_00_even.hdf
e2proclst.py spt_141/aliptcls2d_06.lst --create gmm_17/aliptcls2d_00_even.lst --getclass 0
Processing 1842735 images in spt_141/aliptcls2d_06.lst
e2proc3d.py spt_141/threed_06_odd.hdf gmm_17/threed_00_odd.hdf
e2proclst.py spt_141/aliptcls2d_06.lst --create gmm_17/aliptcls2d_00_odd.lst --getclass 1
Processing 1842731 images in spt_141/aliptcls2d_06.lst
######## iter 1 ##########
### Subtilt rotational refinement....
e2project3d.py gmm_17/threed_00_even.hdf --outfile gmm_17/projections_even.hdf --orientgen=eman:delta=4 --parallel=thread:12
project3d: Namespace(append=False, check=False, compressbits=10, cuda=False, help_to_html=False, nofilecheck=False, orientgen='eman:delta=4', outfile='gmm_17/projections_even.hdf', parallel='thread:12', positionalargs=['gmm_17/threed_00_even.hdf'], postprocess=None,
ppid=206790, prethreshold=False, projector='standard', smear=0, sym='c1', verbose=0)
Precache gmm_17/threed_00_even.hdf
12 total CPUs available
Task ids are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
Task 10 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11]
Task 9 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 8, 11]
Task 8 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 11]
Task 7 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 11]
Task 6 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 11]
Task 5 completed
These tasks are remaining: [0, 1, 2, 3, 4, 11]
Task 4 completed
These tasks are remaining: [0, 1, 2, 3, 11]
Task 3 completed
These tasks are remaining: [0, 1, 2, 11]
Task 2 completed
These tasks are remaining: [0, 1, 11]
Task 1 completed
These tasks are remaining: [0, 11]
Task 0 completed
These tasks are remaining: [11]
1 projection tasks left in main loop
Task 11 completed
These tasks are remaining: []
e2gmm_refine_new.py --ptclsin gmm_17/projections_even.hdf --model gmm_17/gmm_init_even.pdb --maxres 5.5 --modelout gmm_17/model_01_even.txt --niter 40 --trainmodel --learnrate 1e-6
Generating Gaussian model from pdb...
8000 Gaussian loaded from gmm_17/gmm_init_even.pdb
Recompute model from coordinates...
8000 gaussian in the model
Abs loss from loaded model : 0.00001
using box size 124, max resolution 5.5
FRC compares from 1 to 62 Fourier pixels
Loading 1297 particles of box size 256. shrink to 124
1000/1297 R 1297/1297
Data read complete
Train model from ptcl-xfrom pairs...
2024-06-14 10:39:40.815214: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 2000, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 1782, in main
train_decoder(gen_model, trainset, params, options, pts)
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 898, in train_decoder
imgs_cpx=pts2img(pout, xf)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
result = self._call(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 638, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1611, in _filtered_call
self.captured_inputs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1701, in _call_flat
cancellation_manager=cancellation_manager)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 545, in call
ctx=ctx)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[32,8000,3], b.shape=[32,3,3], m=8000, n=3, k=3, batch_size=32
[[{{node loop_body/MatMul/pfor/ArithmeticOptimizer/FoldTransposeIntoMatMul_MatMul}}]] [Op:__forward_pts2img_8637]
Function call stack:
pts2img
e2gmm_batch.py "e2gmm_spt_subtlt.py --ptclsin gmm_17/aliptcls2d_00_even.lst --ptclsout gmm_17/aliptcls2d_01_even.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst" --batch 40000 --niter
0 --subtilt
e2gmm_batch: cannot find --decoderout
e2gmm_batch: cannot find --encoderout
e2gmm_batch: cannot find --midout
e2gmm_batch: cannot find --retraindec
e2gmm_batch: cannot find --learnrate
Writing tmp files in gmm_17/
List input gmm_17/aliptcls2d_00_even.lst, with 1842735 particles
44 tomograms, 47179 3d particles, 1842735 2d particles
0 39550
1 38974
2 39292
3 39078
4 39201
5 39414
6 39303
7 39744
8 39958
9 39195
10 39027
11 39942
12 39060
13 39732
14 38878
15 39911
16 39452
17 39766
18 39747
19 39510
20 39873
21 39577
22 38911
23 39224
24 39924
25 39688
26 39708
27 39447
28 39700
29 39894
30 39904
31 39798
32 39655
33 39445
34 39055
35 39190
36 39040
37 39260
38 39545
39 39180
40 39202
41 39305
42 39275
43 39605
44 39761
45 39015
46 27820
#######################
batch 0/47...
0/47 batches, 39550 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 1/47...
1/47 batches, 38974 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_001.lst --ptclsout gmm_17/tmp_output_001.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 2/47...
2/47 batches, 39292 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_002.lst --ptclsout gmm_17/tmp_output_002.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 3/47...
3/47 batches, 39078 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_003.lst --ptclsout gmm_17/tmp_output_003.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 4/47...
4/47 batches, 39201 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_004.lst --ptclsout gmm_17/tmp_output_004.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 5/47...
5/47 batches, 39414 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_005.lst --ptclsout gmm_17/tmp_output_005.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 6/47...
6/47 batches, 39303 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_006.lst --ptclsout gmm_17/tmp_output_006.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 7/47...
7/47 batches, 39744 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_007.lst --ptclsout gmm_17/tmp_output_007.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 8/47...
8/47 batches, 39958 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_008.lst --ptclsout gmm_17/tmp_output_008.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 9/47...
9/47 batches, 39195 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_009.lst --ptclsout gmm_17/tmp_output_009.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 10/47...
10/47 batches, 39027 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_010.lst --ptclsout gmm_17/tmp_output_010.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 11/47...
11/47 batches, 39942 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_011.lst --ptclsout gmm_17/tmp_output_011.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 12/47...
12/47 batches, 39060 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_012.lst --ptclsout gmm_17/tmp_output_012.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 13/47...
13/47 batches, 39732 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_013.lst --ptclsout gmm_17/tmp_output_013.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 14/47...
14/47 batches, 38878 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_014.lst --ptclsout gmm_17/tmp_output_014.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 15/47...
15/47 batches, 39911 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_015.lst --ptclsout gmm_17/tmp_output_015.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 16/47...
16/47 batches, 39452 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_016.lst --ptclsout gmm_17/tmp_output_016.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 17/47...
17/47 batches, 39766 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_017.lst --ptclsout gmm_17/tmp_output_017.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 18/47...
18/47 batches, 39747 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_018.lst --ptclsout gmm_17/tmp_output_018.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 19/47...
19/47 batches, 39510 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_019.lst --ptclsout gmm_17/tmp_output_019.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 20/47...
20/47 batches, 39873 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_020.lst --ptclsout gmm_17/tmp_output_020.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 21/47...
21/47 batches, 39577 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_021.lst --ptclsout gmm_17/tmp_output_021.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 22/47...
22/47 batches, 38911 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_022.lst --ptclsout gmm_17/tmp_output_022.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 23/47...
23/47 batches, 39224 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_023.lst --ptclsout gmm_17/tmp_output_023.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 24/47...
24/47 batches, 39924 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_024.lst --ptclsout gmm_17/tmp_output_024.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 25/47...
25/47 batches, 39688 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_025.lst --ptclsout gmm_17/tmp_output_025.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 26/47...
26/47 batches, 39708 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_026.lst --ptclsout gmm_17/tmp_output_026.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 27/47...
27/47 batches, 39447 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_027.lst --ptclsout gmm_17/tmp_output_027.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 28/47...
28/47 batches, 39700 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_028.lst --ptclsout gmm_17/tmp_output_028.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 29/47...
29/47 batches, 39894 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_029.lst --ptclsout gmm_17/tmp_output_029.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 30/47...
30/47 batches, 39904 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_030.lst --ptclsout gmm_17/tmp_output_030.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 31/47...
31/47 batches, 39798 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_031.lst --ptclsout gmm_17/tmp_output_031.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 32/47...
32/47 batches, 39655 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_032.lst --ptclsout gmm_17/tmp_output_032.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 33/47...
33/47 batches, 39445 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_033.lst --ptclsout gmm_17/tmp_output_033.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 34/47...
34/47 batches, 39055 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_034.lst --ptclsout gmm_17/tmp_output_034.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 35/47...
35/47 batches, 39190 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_035.lst --ptclsout gmm_17/tmp_output_035.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 36/47...
36/47 batches, 39040 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_036.lst --ptclsout gmm_17/tmp_output_036.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 37/47...
37/47 batches, 39260 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_037.lst --ptclsout gmm_17/tmp_output_037.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 38/47...
38/47 batches, 39545 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_038.lst --ptclsout gmm_17/tmp_output_038.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 39/47...
39/47 batches, 39180 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_039.lst --ptclsout gmm_17/tmp_output_039.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 40/47...
40/47 batches, 39202 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_040.lst --ptclsout gmm_17/tmp_output_040.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 41/47...
41/47 batches, 39305 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_041.lst --ptclsout gmm_17/tmp_output_041.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 42/47...
42/47 batches, 39275 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_042.lst --ptclsout gmm_17/tmp_output_042.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 43/47...
43/47 batches, 39605 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_043.lst --ptclsout gmm_17/tmp_output_043.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 44/47...
44/47 batches, 39761 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_044.lst --ptclsout gmm_17/tmp_output_044.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 45/47...
45/47 batches, 39015 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_045.lst --ptclsout gmm_17/tmp_output_045.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
#######################
batch 46/47...
46/47 batches, 27820 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_046.lst --ptclsout gmm_17/tmp_output_046.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_even.txt not found.
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 147, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 118, in main
ptclsall.extend(load_lst_params(tmp[1]))
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2831, in load_lst_params
lsx=LSXFile(fsp,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2471, in __init__
if ifexists: raise Exception("Error: lst file {} does not exist".format(path))
Exception: Error: lst file gmm_17/tmp_output_000.lst does not exist
Exception ignored in: <function LSXFile.__del__ at 0x7fa8074e97a0>
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2537, in __del__
self.close()
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2548, in close
if self.ptr!=None :
AttributeError: 'LSXFile' object has no attribute 'ptr'
e2spa_make3d.py --input gmm_17/aliptcls2d_01_even.lst --output gmm_17/threed_01_even.hdf --keep 0.9 --outsize 256 --sym c1 --parallel thread:68
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 445, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 63, in main
tmp=EMData(options.input,0,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2883, in db_emd_init
self.read_image(*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2930, in db_read_image
return LSXFile(fsp).read_into_image(self,*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2641, in read_into_image
n,fsp,jsondict=self.read(N)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2593, in read
if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
IndexError: Attempt to read record 0 from #LSX gmm_17/aliptcls2d_01_even.lst with 0 records
e2proc3d.py gmm_17/threed_01_even.hdf gmm_17/threed_raw_even.hdf --compressbits 12
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 856, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 174, in main
infile, _ = parse_infile_arg(infile)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 1090, in parse_infile_arg
raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'gmm_17/threed_01_even.hdf' is not an existing regular file!
e2project3d.py gmm_17/threed_00_odd.hdf --outfile gmm_17/projections_odd.hdf --orientgen=eman:delta=4 --parallel=thread:12
project3d: Namespace(append=False, check=False, compressbits=10, cuda=False, help_to_html=False, nofilecheck=False, orientgen='eman:delta=4', outfile='gmm_17/projections_odd.hdf', parallel='thread:12', positionalargs=['gmm_17/threed_00_odd.hdf'], postprocess=None,
ppid=206790, prethreshold=False, projector='standard', smear=0, sym='c1', verbose=0)
Precache gmm_17/threed_00_odd.hdf
12 total CPUs available
Task ids are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
12 projection tasks left in main loop
Task 11 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
Task 9 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 8, 10]
Task 8 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 7, 10]
Task 7 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 6, 10]
Task 6 completed
These tasks are remaining: [0, 1, 2, 3, 4, 5, 10]
Task 5 completed
These tasks are remaining: [0, 1, 2, 3, 4, 10]
Task 4 completed
These tasks are remaining: [0, 1, 2, 3, 10]
Task 3 completed
These tasks are remaining: [0, 1, 2, 10]
Task 2 completed
These tasks are remaining: [0, 1, 10]
Task 1 completed
These tasks are remaining: [0, 10]
Task 0 completed
These tasks are remaining: [10]
1 projection tasks left in main loop
Task 10 completed
These tasks are remaining: []
e2gmm_refine_new.py --ptclsin gmm_17/projections_odd.hdf --model gmm_17/gmm_init_odd.pdb --maxres 5.5 --modelout gmm_17/model_01_odd.txt --niter 40 --trainmodel --learnrate 1e-6
Generating Gaussian model from pdb...
8000 Gaussian loaded from gmm_17/gmm_init_odd.pdb
Recompute model from coordinates...
8000 gaussian in the model
Abs loss from loaded model : 0.00001
using box size 124, max resolution 5.5
FRC compares from 1 to 62 Fourier pixels
Loading 1297 particles of box size 256. shrink to 124
1000/1297 R 1297/1297
Data read complete
Train model from ptcl-xfrom pairs...
2024-06-14 10:43:15.281691: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 2000, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 1782, in main
train_decoder(gen_model, trainset, params, options, pts)
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_refine_new.py", line 898, in train_decoder
imgs_cpx=pts2img(pout, xf)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 568, in __call__
result = self._call(*args, **kwds)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 638, in _call
return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds) # pylint: disable=protected-access
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1611, in _filtered_call
self.captured_inputs)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1701, in _call_flat
cancellation_manager=cancellation_manager)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 545, in call
ctx=ctx)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/tensorflow_core/python/eager/execute.py", line 67, in quick_execute
six.raise_from(core._status_to_exception(e.code, message), None)
File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.InternalError: Blas xGEMMBatched launch failed : a.shape=[32,8000,3], b.shape=[32,3,3], m=8000, n=3, k=3, batch_size=32
[[{{node loop_body/MatMul/pfor/ArithmeticOptimizer/FoldTransposeIntoMatMul_MatMul}}]] [Op:__forward_pts2img_8637]
Function call stack:
pts2img
e2gmm_batch.py "e2gmm_spt_subtlt.py --ptclsin gmm_17/aliptcls2d_00_odd.lst --ptclsout gmm_17/aliptcls2d_01_odd.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst" --batch 40000 --niter 0
--subtilt
e2gmm_batch: cannot find --decoderout
e2gmm_batch: cannot find --encoderout
e2gmm_batch: cannot find --midout
e2gmm_batch: cannot find --retraindec
e2gmm_batch: cannot find --learnrate
Writing tmp files in gmm_17/
List input gmm_17/aliptcls2d_00_odd.lst, with 1842731 particles
44 tomograms, 47179 3d particles, 1842731 2d particles
0 39512
1 38974
2 39267
3 39087
4 39232
5 39428
6 39304
7 39780
8 39936
9 39180
10 39027
11 39944
12 39096
13 39735
14 38892
15 39938
16 39452
17 39766
18 39733
19 39483
20 39984
21 39426
22 38911
23 39240
24 39949
25 39688
26 39739
27 39457
28 39672
29 39918
30 39868
31 39798
32 39680
33 39446
34 39026
35 39165
36 39040
37 39260
38 39570
39 39999
40 39840
41 39332
42 39926
43 39423
44 39244
45 39660
46 25704
#######################
batch 0/47...
0/47 batches, 39512 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 1/47...
1/47 batches, 38974 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_001.lst --ptclsout gmm_17/tmp_output_001.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 2/47...
2/47 batches, 39267 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_002.lst --ptclsout gmm_17/tmp_output_002.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 3/47...
3/47 batches, 39087 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_003.lst --ptclsout gmm_17/tmp_output_003.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 4/47...
4/47 batches, 39232 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_004.lst --ptclsout gmm_17/tmp_output_004.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 5/47...
5/47 batches, 39428 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_005.lst --ptclsout gmm_17/tmp_output_005.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 6/47...
6/47 batches, 39304 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_006.lst --ptclsout gmm_17/tmp_output_006.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 7/47...
7/47 batches, 39780 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_007.lst --ptclsout gmm_17/tmp_output_007.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 8/47...
8/47 batches, 39936 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_008.lst --ptclsout gmm_17/tmp_output_008.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 9/47...
9/47 batches, 39180 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_009.lst --ptclsout gmm_17/tmp_output_009.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 10/47...
10/47 batches, 39027 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_010.lst --ptclsout gmm_17/tmp_output_010.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 11/47...
11/47 batches, 39944 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_011.lst --ptclsout gmm_17/tmp_output_011.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 12/47...
12/47 batches, 39096 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_012.lst --ptclsout gmm_17/tmp_output_012.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 13/47...
13/47 batches, 39735 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_013.lst --ptclsout gmm_17/tmp_output_013.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 14/47...
14/47 batches, 38892 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_014.lst --ptclsout gmm_17/tmp_output_014.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 15/47...
15/47 batches, 39938 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_015.lst --ptclsout gmm_17/tmp_output_015.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 16/47...
16/47 batches, 39452 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_016.lst --ptclsout gmm_17/tmp_output_016.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 17/47...
17/47 batches, 39766 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_017.lst --ptclsout gmm_17/tmp_output_017.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 18/47...
18/47 batches, 39733 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_018.lst --ptclsout gmm_17/tmp_output_018.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 19/47...
19/47 batches, 39483 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_019.lst --ptclsout gmm_17/tmp_output_019.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 20/47...
20/47 batches, 39984 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_020.lst --ptclsout gmm_17/tmp_output_020.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 21/47...
21/47 batches, 39426 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_021.lst --ptclsout gmm_17/tmp_output_021.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 22/47...
22/47 batches, 38911 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_022.lst --ptclsout gmm_17/tmp_output_022.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 23/47...
23/47 batches, 39240 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_023.lst --ptclsout gmm_17/tmp_output_023.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 24/47...
24/47 batches, 39949 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_024.lst --ptclsout gmm_17/tmp_output_024.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 25/47...
25/47 batches, 39688 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_025.lst --ptclsout gmm_17/tmp_output_025.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 26/47...
26/47 batches, 39739 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_026.lst --ptclsout gmm_17/tmp_output_026.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 27/47...
27/47 batches, 39457 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_027.lst --ptclsout gmm_17/tmp_output_027.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 28/47...
28/47 batches, 39672 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_028.lst --ptclsout gmm_17/tmp_output_028.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 29/47...
29/47 batches, 39918 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_029.lst --ptclsout gmm_17/tmp_output_029.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 30/47...
30/47 batches, 39868 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_030.lst --ptclsout gmm_17/tmp_output_030.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 31/47...
31/47 batches, 39798 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_031.lst --ptclsout gmm_17/tmp_output_031.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 32/47...
32/47 batches, 39680 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_032.lst --ptclsout gmm_17/tmp_output_032.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 33/47...
33/47 batches, 39446 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_033.lst --ptclsout gmm_17/tmp_output_033.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 34/47...
34/47 batches, 39026 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_034.lst --ptclsout gmm_17/tmp_output_034.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 35/47...
35/47 batches, 39165 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_035.lst --ptclsout gmm_17/tmp_output_035.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 36/47...
36/47 batches, 39040 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_036.lst --ptclsout gmm_17/tmp_output_036.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 37/47...
37/47 batches, 39260 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_037.lst --ptclsout gmm_17/tmp_output_037.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 38/47...
38/47 batches, 39570 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_038.lst --ptclsout gmm_17/tmp_output_038.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 39/47...
39/47 batches, 39999 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_039.lst --ptclsout gmm_17/tmp_output_039.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 40/47...
40/47 batches, 39840 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_040.lst --ptclsout gmm_17/tmp_output_040.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 41/47...
41/47 batches, 39332 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_041.lst --ptclsout gmm_17/tmp_output_041.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 42/47...
42/47 batches, 39926 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_042.lst --ptclsout gmm_17/tmp_output_042.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 43/47...
43/47 batches, 39423 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_043.lst --ptclsout gmm_17/tmp_output_043.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 44/47...
44/47 batches, 39244 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_044.lst --ptclsout gmm_17/tmp_output_044.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_045.lst --ptclsout gmm_17/tmp_output_045.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
#######################
batch 46/47...
46/47 batches, 25704 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_046.lst --ptclsout gmm_17/tmp_output_046.lst --model gmm_17/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 222, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py", line 76, in main
pts=np.loadtxt(options.model).astype(floattype)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/model_01_odd.txt not found.
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 147, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_batch.py", line 118, in main
ptclsall.extend(load_lst_params(tmp[1]))
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2831, in load_lst_params
lsx=LSXFile(fsp,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2471, in __init__
if ifexists: raise Exception("Error: lst file {} does not exist".format(path))
Exception: Error: lst file gmm_17/tmp_output_000.lst does not exist
Exception ignored in: <function LSXFile.__del__ at 0x7f23668c17a0>
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2537, in __del__
self.close()
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2548, in close
if self.ptr!=None :
AttributeError: 'LSXFile' object has no attribute 'ptr'
e2spa_make3d.py --input gmm_17/aliptcls2d_01_odd.lst --output gmm_17/threed_01_odd.hdf --keep 0.9 --outsize 256 --sym c1 --parallel thread:68
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 445, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 63, in main
tmp=EMData(options.input,0,True)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2883, in db_emd_init
self.read_image(*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2930, in db_read_image
return LSXFile(fsp).read_into_image(self,*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2641, in read_into_image
n,fsp,jsondict=self.read(N)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2593, in read
if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
IndexError: Attempt to read record 0 from #LSX gmm_17/aliptcls2d_01_odd.lst with 0 records
e2proc3d.py gmm_17/threed_01_odd.hdf gmm_17/threed_raw_odd.hdf --compressbits 12
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 856, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2proc3d.py", line 174, in main
infile, _ = parse_infile_arg(infile)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 1090, in parse_infile_arg
raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'gmm_17/threed_01_odd.hdf' is not an existing regular file!
e2refine_postprocess.py --even gmm_17/threed_01_even.hdf --tophat localwiener --threads 68 --restarget 5.50 --align --sym c1
Use iteration number 1.
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 576, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 116, in main
hdr=EMData(evenfile,0,1)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2883, in db_emd_init
self.read_image(*parms)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 2935, in db_read_image
fsp, idxs = parse_infile_arg(fsp)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/EMAN2.py", line 1090, in parse_infile_arg
raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'gmm_17/threed_01_even.hdf' is not an existing regular file!
Traceback (most recent call last):
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 204, in <module>
main()
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 178, in main
r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")
File "/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_refine.py", line 190, in calc_resolution
fsc=np.loadtxt(fscfile)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/npyio.py", line 968, in loadtxt
fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 269, in open
return ds.open(path, mode, encoding=encoding, newline=newline)
File "/data/maia2/miniconda3/envs/eman2/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 623, in open
raise IOError("%s not found." % path)
OSError: gmm_17/fsc_masked_01.txt not found.
From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
Sent: Friday, June 14, 2024 10:18 AM
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Sent: Friday, June 14, 2024 10:18 AM
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
To view this discussion on the web visit
https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com.
Muyuan Chen
Jun 14, 2024, 2:10:10 PMJun 14
to em...@googlegroups.com
Can you still get the correct output from nvidia-smi from the terminal after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is not using CUDA to begin with, or it only crashed at some point. Presumably if you got through the e2gmm_guess_n step, at least some part of it is working?
There is also a “test_tensorflow.py” in examples folder in EMAN2 that you can run to check if EMAN is using CUDA properly.
I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2). But I don’t think that would be the issue.
Also check "conda list” to see if you have the cudatoolkit and the cuda version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions should work but I never tested.
If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old version of everything so it is possible that something break in new CUDA versions that I am not aware of.
Muyuan
> 2024-06-14 10:39:40.815214: E tensorflow/stream_executor/cuda/cuda_blas.cc:428 <http://cuda_blas.cc:428/>] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301 <http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
> 2024-06-14 10:43:15.281691: E tensorflow/stream_executor/cuda/cuda_blas.cc:428 <http://cuda_blas.cc:428/>] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301 <http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
> 2024-06-14 05:24:21.637490: E tensorflow/stream_executor/cuda/cuda_driver.cc:351 <http://cuda_driver.cc:351/>] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
> To post to this group, send email to em...@googlegroups.com <mailto:em...@googlegroups.com>
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> To view this discussion on the web visithttps://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>.
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
There is also a “test_tensorflow.py” in examples folder in EMAN2 that you can run to check if EMAN is using CUDA properly.
I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2). But I don’t think that would be the issue.
Also check "conda list” to see if you have the cudatoolkit and the cuda version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions should work but I never tested.
If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old version of everything so it is possible that something break in new CUDA versions that I am not aware of.
Muyuan
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301 <http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301 <http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
> From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
> Sent: Friday, June 14, 2024 10:18 AM
> To: em...@googlegroups.com <em...@googlegroups.com>
> Subject: Re: [EMAN2] Error for GMM-based refinement for spt
>
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351 <http://cuda_driver.cc:351/>] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
> Sent: Friday, June 14, 2024 10:18 AM
> To: em...@googlegroups.com <em...@googlegroups.com>
> Subject: Re: [EMAN2] Error for GMM-based refinement for spt
>
> The most notable issue is this:
>
> Are you running on a machine without CUDA-able GPU? Or it just fails at the later iterations? Do you only get the error at batch 45, or the earlier batches give the same error as well?
>
> On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu> wrote:
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351 <http://cuda_driver.cc:351/>] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
> Are you running on a machine without CUDA-able GPU? Or it just fails at the later iterations? Do you only get the error at batch 45, or the earlier batches give the same error as well?
>
> On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu> wrote:
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com <mailto:eman2+un...@googlegroups.com>.
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To view this discussion on the web visithttps://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
>
> --
> --
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com <mailto:em...@googlegroups.com>
> --
> --
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com <mailto:eman2+un...@googlegroups.com>.
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>.
> --
> --
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com <mailto:em...@googlegroups.com>
> --
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com <mailto:eman2+un...@googlegroups.com>.
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
Maia Azubel
Jun 14, 2024, 3:37:30 PMJun 14
to em...@googlegroups.com
I switched to another machine
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
Reconstruction finishend (12.3 s)
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
------------------------------------------------------------------------------------------
>conda list
...
cudatoolkit 10.1.243 h6bb024c_0
--------------------------------------------------------------------------------------
>phyton3 test_tensorflow.py
no errors (full report attached)
----------------------------------
However,
>e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:12--threads=12
....
Loading 39550 particles of box size 512. shrink to 124
39000/39550 R 39550/39550
batch 1/47...
________________________________
Sent: Friday, June 14, 2024 11:09 AM
________________________________
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com<https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>.
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
Reconstruction finishend (12.3 s)
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
------------------------------------------------------------------------------------------
>conda list
...
cudatoolkit 10.1.243 h6bb024c_0
--------------------------------------------------------------------------------------
>phyton3 test_tensorflow.py
no errors (full report attached)
----------------------------------
However,
>e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:12--threads=12
....
#######################
batch 0/47...
0/47 batches, 39550 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
batch 0/47...
0/47 batches, 39550 particles
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
Gaussian model shape: (8000, 5)
Load 39550 2D particles
Loading 39550 particles of box size 512. shrink to 124
39000/39550 R 39550/39550
Data read complete
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
(39550, 5, 3)
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
________________________________
Sent: Friday, June 14, 2024 11:09 AM
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Muyuan Chen
Jun 14, 2024, 3:44:26 PMJun 14
to em...@googlegroups.com
Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know
why it does not throw the normal OOM error. I did not add batch size as an
option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
make the change now, but before that, can you try
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
--clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
--batchsz 2
(or even batchsz=1) Just to confirm it is a memory issue?
On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu> wrote:
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
> *Reconstruction finishend (12.3 s)*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 11:09 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_blas.cc:428] failed to run cuBLAS routine:
> cuda_blas.cc:428] failed to run cuBLAS routine:
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
why it does not throw the normal OOM error. I did not add batch size as an
option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
make the change now, but before that, can you try
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
--clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
(or even batchsz=1) Just to confirm it is a memory issue?
On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu> wrote:
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 11:09 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
Maia Azubel
Jun 14, 2024, 4:07:05 PMJun 14
to em...@googlegroups.com
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
________________________________
From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
Sent: Friday, June 14, 2024 12:43 PM
From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know why it does not throw the normal OOM error. I did not add batch size as an option for e2gmm_spt_refine, so it would be a bit tricky to change. I can make the change now, but before that, can you try
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
(or even batchsz=1) Just to confirm it is a memory issue?
On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu<mailto:maz...@stanford.edu>> wrote:
I switched to another machine
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
I switched to another machine
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
Reconstruction finishend (12.3 s)
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 11:09 AM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Can you still get the correct output from nvidia-smi from the terminal after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is not using CUDA to begin with, or it only crashed at some point. Presumably if you got through the e2gmm_guess_n step, at least some part of it is working?
There is also a “test_tensorflow.py” in examples folder in EMAN2 that you can run to check if EMAN is using CUDA properly.
I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2). But I don’t think that would be the issue.
Also check "conda list” to see if you have the cudatoolkit and the cuda version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions should work but I never tested.
If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old version of everything so it is possible that something break in new CUDA versions that I am not aware of.
Muyuan
2024-06-14 10:39:40.815214: E tensorflow/stream_executor/cuda/cuda_blas.cc:428<http://cuda_blas.cc:428/>] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301<http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301<http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 10:18 AM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
The most notable issue is this:
Are you running on a machine without CUDA-able GPU? Or it just fails at the later iterations? Do you only get the error at batch 45, or the earlier batches give the same error as well?
On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu<mailto:maz...@stanford.edu>> wrote:
Hi Muyuan,
I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
Thanks,
Maia
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
Your Python version is: 3.7.9
command
e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
error
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351<http://cuda_driver.cc:351/>] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Hi Muyuan,
I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
Thanks,
Maia
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
Your Python version is: 3.7.9
command
e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
error
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visithttps://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com<https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com<https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com<https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Muyuan Chen
Jun 14, 2024, 4:54:12 PMJun 14
to em...@googlegroups.com
Strange. Can you run
e2proclst.py gmm_17/tmp_input_000.lst --create gmm_17/test.lst --range 1000
then
e2gmm_spt_subtlt.py --ptclsin gmm_17/test.lst --ptclsout
gmm_17/test1.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 12:43 PM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_blas.cc:428] failed to run cuBLAS routine:
> cuda_blas.cc:428] failed to run cuBLAS routine:
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
e2proclst.py gmm_17/tmp_input_000.lst --create gmm_17/test.lst --range 1000
then
e2gmm_spt_subtlt.py --ptclsin gmm_17/test.lst --ptclsout
gmm_17/test1.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256
--minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 12:43 PM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know
> why it does not throw the normal OOM error. I did not add batch size as an
> option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
> make the change now, but before that, can you try
>
> e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
> gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
> --batchsz 2
>
> (or even batchsz=1) Just to confirm it is a memory issue?
>
> Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know
> why it does not throw the normal OOM error. I did not add batch size as an
> option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
> make the change now, but before that, can you try
>
> e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
> gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
> --batchsz 2
>
> (or even batchsz=1) Just to confirm it is a memory issue?
>
> On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu> wrote:
>
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
>
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
> *Reconstruction finishend (12.3 s)*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
>
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
>
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> Can you still get the correct output from nvidia-smi from the terminal
> after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is
> not using CUDA to begin with, or it only crashed at some point. Presumably
> if you got through the e2gmm_guess_n step, at least some part of it is
> working?
> There is also a “test_tensorflow.py” in examples folder in EMAN2 that you
> can run to check if EMAN is using CUDA properly.
>
> I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2).
> But I don’t think that would be the issue.
>
> Also check "conda list” to see if you have the cudatoolkit and the cuda
> version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions
> should work but I never tested.
>
> If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or
> CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old
> version of everything so it is possible that something break in new CUDA
> versions that I am not aware of.
>
> Muyuan
>
> Can you still get the correct output from nvidia-smi from the terminal
> after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is
> not using CUDA to begin with, or it only crashed at some point. Presumably
> if you got through the e2gmm_guess_n step, at least some part of it is
> working?
> There is also a “test_tensorflow.py” in examples folder in EMAN2 that you
> can run to check if EMAN is using CUDA properly.
>
> I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2).
> But I don’t think that would be the issue.
>
> Also check "conda list” to see if you have the cudatoolkit and the cuda
> version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions
> should work but I never tested.
>
> If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or
> CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old
> version of everything so it is possible that something break in new CUDA
> versions that I am not aware of.
>
> Muyuan
>
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> CUDA-capable device is detected
>
> Are you running on a machine without CUDA-able GPU? Or it just fails at
> the later iterations? Do you only get the error at batch 45, or the earlier
> batches give the same error as well?
>
>
> Are you running on a machine without CUDA-able GPU? Or it just fails at
> the later iterations? Do you only get the error at batch 45, or the earlier
> batches give the same error as well?
>
> On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu> wrote:
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed
> that there were error messages from one batch to the next, but the program
> kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8
> 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts
> spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout
> gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed
> that there were error messages from one batch to the next, but the program
> kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8
> 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts
> spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout
> gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
Maia Azubel
Jun 14, 2024, 4:58:39 PMJun 14
to em...@googlegroups.com
>e2proclst.py gmm_17/tmp_input_000.lst --create gmm_17/test.lst --range 1000
Processing 1000 images in gmm_17/tmp_input_000.lst
>e2gmm_spt_subtlt.py --ptclsin gmm_17/test.lst --ptclsout gmm_17/test1.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
Gaussian model shape: (8000, 5)
Load 1000 2D particles
Loading 1000 particles of box size 512. shrink to 124
0/1000 R 1000/1000
Data read complete
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
(1000, 5, 3)
particle size 512, clip to 124, shrink to 124
compare 4 to 62 Fourier pixels
prepare nearest neighbor distance
________________________________
From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
Sent: Friday, June 14, 2024 1:53 PM
From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Strange. Can you run
e2proclst.py gmm_17/tmp_input_000.lst --create gmm_17/test.lst --range 1000
then
e2gmm_spt_subtlt.py --ptclsin gmm_17/test.lst --ptclsout gmm_17/test1.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 12:43 PM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know why it does not throw the normal OOM error. I did not add batch size as an option for e2gmm_spt_refine, so it would be a bit tricky to change. I can make the change now, but before that, can you try
e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst --batchsz 2
(or even batchsz=1) Just to confirm it is a memory issue?
On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu<mailto:maz...@stanford.edu>> wrote:
I switched to another machine
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
I switched to another machine
NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
-------------------------------
e2gmm_guess_n step -->ran with no errors
Reconstruction finishend (12.3 s)
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf
e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc spt_141/threed_06.hdf
final pdb model: spt_141/threed_06_seg.pdb
final GMM in text file: spt_141/threed_06_gmm.txt
final GMM in density map: spt_141/threed_06_gmm.hdf
map-model FSC: spt_141/threed_06_fsc.txt
________________________________
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 11:09 AM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 11:09 AM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Can you still get the correct output from nvidia-smi from the terminal after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is not using CUDA to begin with, or it only crashed at some point. Presumably if you got through the e2gmm_guess_n step, at least some part of it is working?
There is also a “test_tensorflow.py” in examples folder in EMAN2 that you can run to check if EMAN is using CUDA properly.
I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2). But I don’t think that would be the issue.
Also check "conda list” to see if you have the cudatoolkit and the cuda version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions should work but I never tested.
If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old version of everything so it is possible that something break in new CUDA versions that I am not aware of.
Muyuan
2024-06-14 10:39:40.815214: E tensorflow/stream_executor/cuda/cuda_blas.cc:428<http://cuda_blas.cc:428/>] failed to run cuBLAS routine: CUBLAS_STATUS_NOT_SUPPORTED
2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301<http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/cuda_blas.cc:2301<http://cuda_blas.cc:2301/>] Internal: failed BLAS call, see log for details
From: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>> on behalf of Muyuan Chen <g5v...@gmail.com<mailto:g5v...@gmail.com>>
Sent: Friday, June 14, 2024 10:18 AM
To: em...@googlegroups.com<mailto:em...@googlegroups.com> <em...@googlegroups.com<mailto:em...@googlegroups.com>>
Subject: Re: [EMAN2] Error for GMM-based refinement for spt
The most notable issue is this:
Are you running on a machine without CUDA-able GPU? Or it just fails at the later iterations? Do you only get the error at batch 45, or the earlier batches give the same error as well?
On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu<mailto:maz...@stanford.edu>> wrote:
Hi Muyuan,
I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
Thanks,
Maia
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
Your Python version is: 3.7.9
command
e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
error
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/cuda_driver.cc:351<http://cuda_driver.cc:351/>] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
Hi Muyuan,
I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed that there were error messages from one batch to the next, but the program kept going until finally failed. Details below. I'll appreciate any advice.
Thanks,
Maia
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8 4.18.0-540.el8.x86_64
Your Python version is: 3.7.9
command
e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
error
#######################
batch 45/47...
45/47 batches, 39660 particles
e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5 --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
Gaussian model shape: (8000, 5)
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visithttps://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email toeman2+u...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com<https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com<https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com<https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com<mailto:em...@googlegroups.com>
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To unsubscribe from this group, send email to eman2+un...@googlegroups.com<mailto:eman2%2Bunsu...@googlegroups.com>
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com<https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>.
--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com<mailto:eman2+un...@googlegroups.com>.
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2
---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com<https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Muyuan Chen
Jun 14, 2024, 5:04:02 PMJun 14
to em...@googlegroups.com
Well, at least it gets to the error part more quickly. I tried on my side
and cannot reproduce the issue, so debugging would be a bit tricky. One
possibility is edit the file
/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py
from your side, and comment out line 51, so it becomes:
# @tf.function()
Then run the same command again. I am not sure if it will fix the issue but
it might give you more sensible error message...
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 1:53 PM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 11:09 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_blas.cc:428] failed to run cuBLAS routine:
> cuda_blas.cc:428] failed to run cuBLAS routine:
> ------------------------------
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657790C8529371F5295CA4ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
and cannot reproduce the issue, so debugging would be a bit tricky. One
possibility is edit the file
/data/maia2/miniconda3/envs/eman2/bin/e2gmm_spt_subtlt.py
from your side, and comment out line 51, so it becomes:
# @tf.function()
Then run the same command again. I am not sure if it will fix the issue but
it might give you more sensible error message...
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 1:53 PM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know
> why it does not throw the normal OOM error. I did not add batch size as an
> option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
> make the change now, but before that, can you try
>
> e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
> gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
> --batchsz 2
>
> (or even batchsz=1) Just to confirm it is a memory issue?
>
> Oh. 2080Ti... 10GB RAM probably would be too small, although I dont know
> why it does not throw the normal OOM error. I did not add batch size as an
> option for e2gmm_spt_refine, so it would be a bit tricky to change. I can
> make the change now, but before that, can you try
>
> e2gmm_spt_subtlt.py --ptclsin gmm_17/tmp_input_000.lst --ptclsout
> gmm_17/tmp_output_000.lst --model gmm_17/model_01_even.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_17/particle_info_3d.lst
> --batchsz 2
>
> (or even batchsz=1) Just to confirm it is a memory issue?
>
> On Fri, Jun 14, 2024 at 12:37 PM Maia Azubel <maz...@stanford.edu> wrote:
>
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
>
> I switched to another machine
> NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4
>
> -------------------------------
> e2gmm_guess_n step -->ran with no errors
>
> *Reconstruction finishend (12.3 s)*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
>
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_gmm.hdf --process
> mask.soft:outer_radius=-16 --matchto spt_141/threed_06.hdf*
> *e2proc3d.py spt_141/threed_06_gmm.hdf spt_141/threed_06_fsc.txt --calcfsc
> spt_141/threed_06.hdf*
> *final pdb model: spt_141/threed_06_seg.pdb*
> *final GMM in text file: spt_141/threed_06_gmm.txt*
> *final GMM in density map: spt_141/threed_06_gmm.hdf*
> *map-model FSC: spt_141/threed_06_fsc.txt*
>
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 11:09 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> Can you still get the correct output from nvidia-smi from the terminal
> after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is
> not using CUDA to begin with, or it only crashed at some point. Presumably
> if you got through the e2gmm_guess_n step, at least some part of it is
> working?
> There is also a “test_tensorflow.py” in examples folder in EMAN2 that you
> can run to check if EMAN is using CUDA properly.
>
> I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2).
> But I don’t think that would be the issue.
>
> Also check "conda list” to see if you have the cudatoolkit and the cuda
> version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions
> should work but I never tested.
>
> If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or
> CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old
> version of everything so it is possible that something break in new CUDA
> versions that I am not aware of.
>
> Muyuan
>
> Can you still get the correct output from nvidia-smi from the terminal
> after getting the “CUDA_ERROR_NO_DEVICE” error? I am not sure if EMAN is
> not using CUDA to begin with, or it only crashed at some point. Presumably
> if you got through the e2gmm_guess_n step, at least some part of it is
> working?
> There is also a “test_tensorflow.py” in examples folder in EMAN2 that you
> can run to check if EMAN is using CUDA properly.
>
> I am using an older version myself (Nvidia driver 535.171.04, CUDA 12.2).
> But I don’t think that would be the issue.
>
> Also check "conda list” to see if you have the cudatoolkit and the cuda
> version of tensorflow. My cudatoolkit is 11.6.0. I think higher versions
> should work but I never tested.
>
> If Steve or anyone else have more knowledge of the CUDA_ERROR_NO_DEVICE or
> CUBLAS_STATUS_NOT_SUPPORTED error, please comment too… I generally use old
> version of everything so it is possible that something break in new CUDA
> versions that I am not aware of.
>
> Muyuan
>
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:39:40.815251: E tensorflow/stream_executor/cuda/
> CUBLAS_STATUS_NOT_SUPPORTED
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> cuda_blas.cc:2301] Internal: failed BLAS call, see log for details
> 2024-06-14 10:43:15.281731: E tensorflow/stream_executor/cuda/
> *From:* em...@googlegroups.com <em...@googlegroups.com> on behalf of
> Muyuan Chen <g5v...@gmail.com>
> *Sent:* Friday, June 14, 2024 10:18 AM
> *To:* em...@googlegroups.com <em...@googlegroups.com>
> *Subject:* Re: [EMAN2] Error for GMM-based refinement for spt
>
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
> The most notable issue is this:
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> CUDA-capable device is detected
>
> Are you running on a machine without CUDA-able GPU? Or it just fails at
> the later iterations? Do you only get the error at batch 45, or the earlier
> batches give the same error as well?
>
>
> Are you running on a machine without CUDA-able GPU? Or it just fails at
> the later iterations? Do you only get the error at batch 45, or the earlier
> batches give the same error as well?
>
> On Jun 14, 2024, at 10:13 AM, Maia Azubel <maz...@stanford.edu> wrote:
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed
> that there were error messages from one batch to the next, but the program
> kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8
> 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts
> spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout
> gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> cuda_driver.cc:351] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no
>
> Hi Muyuan,
> I ran e2gmm_spt_refine.py overnight. When I came this morning, I noticed
> that there were error messages from one batch to the next, but the program
> kept going until finally failed. Details below. I'll appreciate any advice.
> Thanks,
> Maia
>
>
> e2version.py
> EMAN 2.99.54 ( GITHUB: 2023-12-05 12:49 - commit: 4e38f2525 )
> Your EMAN2 is running on: Linux-4.18.0-540.el8.x86_64-x86_64-with-centos-8
> 4.18.0-540.el8.x86_64
> Your Python version is: 3.7.9
>
>
> command
> e2gmm_spt_refine.pyspt_141/threed_06.hdf --startres 5.5 --sym c1 --initpts
> spt_141/threed_06_seg.pdb --iters r,p,r3 --parallel=thread:68 --threads=68
>
> error
> #######################
> batch 45/47...
> 45/47 batches, 39660 particles
> e2gmm_spt_subtlt.py --ptclsin gmm_18/tmp_input_045.lst --ptclsout
> gmm_18/tmp_output_045.lst --model gmm_18/model_01_odd.txt --maxres 5.5
> --clip 256 --minres 200 --niter 20 --info3d gmm_18/particle_info_3d.lst
> Gaussian model shape: (8000, 5)
> 2024-06-14 05:12:07.896979: E tensorflow/stream_executor/cuda/
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66576FE8519A07F6A7989D9FDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toeman2+u...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com
> <https://groups.google.com/d/msgid/eman2/880E9D93-33D9-472E-A832-2F7B099D4E77%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657D31F23237B2741B7D84EDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com
> <https://groups.google.com/d/msgid/eman2/47058D58-CA06-45F4-BD0E-F1252A03782E%40gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB66577A1BDB486D3464E66D7ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFiZy0thRPjt%2BPPHBAhAGWJSRyerdaETFmJaPii1Oy-PQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB665757250B780B1FB90B049DDFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com
> <https://groups.google.com/d/msgid/eman2/CAO_0xqFJXf70wW4k7NiEUBBni%3DcVMk_rwRT_L9VbCPyo2hdJag%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to eman2+un...@googlegroups.com.
> --
>
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to
> eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups
> "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/eman2/BY5PR02MB6657790C8529371F5295CA4ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com
> <https://groups.google.com/d/msgid/eman2/BY5PR02MB6657790C8529371F5295CA4ADFC22%40BY5PR02MB6657.namprd02.prod.outlook.com?utm_medium=email&utm_source=footer>
> .
>
Reply all
Reply to author
Forward
0 new messages