e2_make3d crash after subtilt refinement step

[Mikey Grunst]

Dear EMAN2 community,

I am refining approximately 27,000 particles on a local workstation running linux. 

I previously refined ~6,000 particles on this workstation from the same dataset, and got a good result (~14A structure). I hoped after this test run to include all 27k particles. 

I am having trouble finishing the first subtilt refinement step. I get good results for the "p" iterations (aligning well), but on the "t" iterations the program keeps crashing on the e2_make3d part. I can't seem to find an error message, because the whole terminal crashes and closes. Is this a memory/performance issue?

The one error message I have ignored that keeps repeating is: 

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

One work-around I thought could be to make the new averages outside of the e2spt_refine_new.py pipeline, then restart from here. What would be the e2_make3d.py command to make the new averages from both the aliptcls3d_*.lst and the aliptcls2d_*lst? The e2spt_refine_new.py did make the new aliptcls2d_*.lst file, it just seems to crash making the new resulting average/reference.

Here is e2version output:

EMAN 2.99.55 ( GITHUB: 2024-05-15 13:18 - commit: 2aa2b781e )
Your EMAN2 is running on: Linux-6.5.0-35-generic-x86_64-with-glibc2.35 6.5.0-35-generic
Your Python version is: 3.11.9

Thank you for any advise you may have regarding this!

-Mikey

[Mikey Grunst]

Apologies, I forgot to include the last command I ran: 

/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spt_refine_new.py --ptcls=spt_03/aliptcls3d_01.lst --ref=spt_03/threed_01.hdf --startres=13.0 --goldcontinue --sym=c1 --iters=p,t,d,t,p,t,d,t --keep=0.9 --tophat=local --parallel=thread:20 --threads=20 --ppid=-2. The bold "t" is as far as it went.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/ac59f016-c3bf-4bb0-948b-e5b5593e3bcbn%40googlegroups.com.

[Muyuan Chen]

Can you provide the full command line output? I don’t think the warning you posted is the cause of the problem…

On Jun 14, 2024, at 8:02 PM, Mikey Grunst <mikey....@yale.edu> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CACisnzrtYUoJTQzj19LNVvrP_cDehmV%3DGFFft_xWvJN2To0%2Bkg%40mail.gmail.com.

[Mikey Grunst]

Thanks Muyuan! 

My entire terminal crashes,  so I am not able to provide the exact output displayed in the terminal. Is this information stored in an output file somewhere? Sorry if I missed that.

I can also run it again and try to capture the entire terminal output using other means.

-Mikey

[Muyuan Chen]

You can always just add “> xxx” at the end of the command to redirect the command line output to a file. 

In .eman2log.txt, you may find the last command it ran before the crash, but not the error message.

Muyuan

On Jun 14, 2024, at 9:19 PM, Mikey Grunst <mikey....@yale.edu> wrote:

Thanks Muyuan! 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/81f22aa0-11e8-4081-806a-d12bd5ff1e8cn%40googlegroups.com.

[Steve Ludtke]

If its an error you may need 2>xxx to collect the output

Sent from my iPhone

On Jun 14, 2024, at 9:36 PM, Muyuan Chen <g5v...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/4C499FAB-9FEB-4625-849F-17783A3F1608%40gmail.com.

[Mikey Grunst]

This is handy! Once I rerun and acquire the output I will post here.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/C456C96A-F242-4357-9C45-4B27AC62A12B%40gmail.com.

[Mikey Grunst]

Hi All,

I ran the refinement a few times with the 2>xxx and >xxx at the end of the command. I used cat to read the "xxx" files.

"2>xxx" repeated the same error several times as before (no other info):

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

">xxx" only wrote:

Writing in spt_06...
Gathering metadata...
 5688/27910

Once it did actually make it through one "t" iteration (and the structure improved!), but then it crashed at the subsequent "d" iteration. I can't directly use --goldcontinue from a "t" or "d" iteration ("t" requires a "p" iteration first, and "d" requires both a "p" and a "t" first).

I noticed when running "top" during the "d" iteration that kswapdo was activated instead of seeing the individual python threads running. I suspect it may be running out of RAM. We are running a AMD 16 core CPU with 128 Gb RAM (and 2x 4090 Nvidia GPUs).

I'll transfer the data and run it on our cluster as well. Happy to try any other suggestions you may have.

Thanks again!

-Mikey

[Mikey Grunst]

Dear EMAN2 Community,

Following up on this:

I had transferred the whole eman2 directory from our local machine to our cluster. It looks like it was running okay for the p and t iterations, but the d iteration fails. Thank you for any advice you may have!

The cluster version:

EMAN 2.99.47 ( GITHUB: 2023-03-04 13:31 - commit: 3f313008c3185410fe859663e763dffb9c0b6fcc )
Your EMAN2 is running on: Linux-4.18.0-477.36.1.el8_8.x86_64-x86_64-with-glibc2.28 4.18.0-477.36.1.el8_8.x86_64
Your Python version is: 3.9.16

Our local machine version:

EMAN 2.99.55 ( GITHUB: 2024-05-15 13:18 - commit: 2aa2b781e )

Your EMAN2 is running on: Linux-6.5.0-41-generic-x86_64-with-glibc2.35 6.5.0-41-generic

Your Python version is: 3.11.9

The command executed on cluster (continuing from a previous run):

/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spt_refine_new.py --ptcls=spt_06/aliptcls3d_01.lst --ref=spt_06/threed_01.hdf --startres=12.3 --goldcontinue --sym=c1 --iters=p,t,d,t,p,t,d --keep=0.9 --tophat=local --loadali2d=spt_06/aliptcls2d_01.lst --parallel=thread:50 --threads=50 --ppid=-2

The terminal output (terminal didnt crash here):

Writing in spt_08...
Gathering metadata...
 27910/27910
Preparing references...
e2proc3d.py spt_06/threed_01_even.hdf spt_08/threed_00_even.hdf
e2proc3d.py spt_06/threed_01_odd.hdf spt_08/threed_00_odd.hdf
e2proc3d.py spt_08/threed_00_even.hdf spt_08/threed_00.hdf --addfile spt_08/threed_00_odd.hdf --mult 0.5
e2proclst.py spt_06/aliptcls2d_01.lst --create spt_08/aliptcls2d_00.lst
Processing 1696761 images in spt_06/aliptcls2d_01.lst
######## iter 1 ##########
### Subtomogram alignment....
e2spt_align_subtlt.py spt_08/particle_info_3d.lst spt_08/threed_00.hdf --path spt_08 --iter 1 --maxres 12.30 --sym c1 --parallel thread:50  --fromscratch --plst spt_08/aliptcls2d_00.lst --goldcontinue
27910 jobs on 50 CPUs
e2spa_make3d.py --input spt_08/aliptcls2d_01.lst --output spt_08/threed_01_even.hdf --keep 0.9 --clsid even --outsize 256 --sym c1  --parallel thread:50
848380 particles total
  Excluding particles on 3D particle score. Now 90.0%  left
  Excluding particles on 2D score. Now 81.0%  left
687188 particle, score max 1.00, min 0.12
50 total CPUs available
50 jobs
Reconstruction finishend (728.2 s)
e2proc3d.py spt_08/threed_01_even.hdf spt_08/threed_raw_even.hdf --compressbits 12
e2spa_make3d.py --input spt_08/aliptcls2d_01.lst --output spt_08/threed_01_odd.hdf --keep 0.9 --clsid odd --outsize 256 --sym c1  --parallel thread:50
848381 particles total
  Excluding particles on 3D particle score. Now 90.0%  left
  Excluding particles on 2D score. Now 81.0%  left
687167 particle, score max 1.00, min 0.14
50 total CPUs available
50 jobs
Reconstruction finishend (726.6 s)
e2proc3d.py spt_08/threed_01_odd.hdf spt_08/threed_raw_odd.hdf --compressbits 12
e2refine_postprocess.py --even spt_08/threed_01_even.hdf   --tophat local --threads 50 --restarget 12.30 --sym c1
Use iteration number 1.
Fri Jun 28 08:28:12 2024: e2proc3d.py spt_08/threed_01_even.hdf spt_08/fsc_unmasked_01.txt --calcfsc=spt_08/threed_01_odd.hdf
Performing structure factor amplitude correction
Fri Jun 28 08:28:14 2024: e2proc3d.py spt_08/threed_01.hdf spt_08/tmp.hdf  --process=filter.wiener.byfsc:fscfile=spt_08/fsc_unmasked_01.txt:snrmult=2:maxfreq=0.08130081300813008 --process normalize.edgemean
Fri Jun 28 08:28:23 2024: e2proc3d.py spt_08/threed_01_even.hdf spt_08/tmp_even.hdf --multfile spt_08/mask_tight.hdf
Fri Jun 28 08:28:24 2024: e2proc3d.py spt_08/threed_01_odd.hdf spt_08/tmp_odd.hdf --multfile spt_08/mask_tight.hdf
Fri Jun 28 08:28:24 2024: e2proc3d.py spt_08/tmp_even.hdf spt_08/fsc_maskedtight_01.txt --calcfsc spt_08/tmp_odd.hdf
Fri Jun 28 08:28:25 2024: e2proc3d.py spt_08/threed_01_even.hdf spt_08/tmp_even.hdf --multfile spt_08/mask.hdf
Fri Jun 28 08:28:25 2024: e2proc3d.py spt_08/threed_01_odd.hdf spt_08/tmp_odd.hdf --multfile spt_08/mask.hdf
Fri Jun 28 08:28:26 2024: e2proc3d.py spt_08/tmp_even.hdf spt_08/fsc_masked_01.txt --calcfsc spt_08/tmp_odd.hdf
Fri Jun 28 08:28:27 2024: e2proc3d.py spt_08/threed_01_even.hdf spt_08/threed_even_unmasked.hdf  --process filter.lowpass.tophat:cutoff_abs=0.5
Fri Jun 28 08:28:27 2024: e2proc3d.py spt_08/threed_01_odd.hdf spt_08/threed_odd_unmasked.hdf  --process filter.lowpass.tophat:cutoff_abs=0.5
Fri Jun 28 08:28:28 2024: e2fsc_real_local.py spt_08/threed_even_unmasked.hdf spt_08/threed_odd_unmasked.hdf --output spt_08/fscvol_01.hdf --outfilt spt_08/threed_01.hdf --outfilte spt_08/threed_01_even.hdf --outfilto spt_08/threed_01_odd.hdf --mask spt_08/mask.hdf --threads 50 --localsizea 49 --compressbits 12 --tophat -v 1
Using 1.35 A/pix
256 x 256 x 256
Using box-size:  300
Preparing for local calculation
149  threads
Fri Jun 28 08:32:12 2024: e2proc3d.py spt_08/threed_01.hdf spt_08/threed_01.hdf --multfile spt_08/mask.hdf --process normalize.edgemean  
Fri Jun 28 08:32:13 2024: e2proc3d.py spt_08/threed_01_even.hdf spt_08/threed_01_even.hdf --multfile spt_08/mask.hdf --process normalize.edgemean  
Fri Jun 28 08:32:14 2024: e2proc3d.py spt_08/threed_01_odd.hdf spt_08/threed_01_odd.hdf  --multfile spt_08/mask.hdf --process normalize.edgemean  
Resolution (FSC<0.2) is ~12.3 A
Iteration 0 complete, 11:33:49
######## iter 2 ##########
### Subtilt translational refinement....
e2spt_subtlt_local.py --ref spt_08/threed_01.hdf --path spt_08 --iter 2 --maxres 9.85 --parallel thread:50 --aliptcls3d spt_08/aliptcls3d_01.lst --smooth 100 --smoothN 15 --refine_trans --goldcontinue
1696761 jobs on 50 CPUs
e2spa_make3d.py --input spt_08/aliptcls2d_02.lst --output spt_08/threed_02_even.hdf --keep 0.9 --clsid even --outsize 256 --sym c1  --parallel thread:50
848380 particles total
  Excluding particles on 3D particle score. Now 90.0%  left
  Excluding particles on 2D score. Now 81.0%  left
  Excluding particles on translation. Now 72.9%  left
618481 particle, score max 1.00, min 0.15
50 total CPUs available
50 jobs
Reconstruction finishend (1075.0 s)
e2proc3d.py spt_08/threed_02_even.hdf spt_08/threed_raw_even.hdf --compressbits 12
e2spa_make3d.py --input spt_08/aliptcls2d_02.lst --output spt_08/threed_02_odd.hdf --keep 0.9 --clsid odd --outsize 256 --sym c1  --parallel thread:50
848381 particles total
  Excluding particles on 3D particle score. Now 90.0%  left
  Excluding particles on 2D score. Now 81.0%  left
  Excluding particles on translation. Now 72.9%  left
618493 particle, score max 1.00, min 0.18
50 total CPUs available
50 jobs
Reconstruction finishend (1135.1 s)
e2proc3d.py spt_08/threed_02_odd.hdf spt_08/threed_raw_odd.hdf --compressbits 12
e2refine_postprocess.py --even spt_08/threed_02_even.hdf   --tophat local --threads 50 --restarget 9.85 --sym c1
Use iteration number 2.
Fri Jun 28 14:54:45 2024: e2proc3d.py spt_08/threed_02_even.hdf spt_08/fsc_unmasked_02.txt --calcfsc=spt_08/threed_02_odd.hdf
Performing structure factor amplitude correction
Fri Jun 28 14:54:47 2024: e2proc3d.py spt_08/threed_02.hdf spt_08/tmp.hdf  --process=filter.wiener.byfsc:fscfile=spt_08/fsc_unmasked_02.txt:snrmult=2:maxfreq=0.10152284263959391 --process normalize.edgemean
Fri Jun 28 14:54:56 2024: e2proc3d.py spt_08/threed_02_even.hdf spt_08/tmp_even.hdf --multfile spt_08/mask_tight.hdf
Fri Jun 28 14:54:57 2024: e2proc3d.py spt_08/threed_02_odd.hdf spt_08/tmp_odd.hdf --multfile spt_08/mask_tight.hdf
Fri Jun 28 14:54:57 2024: e2proc3d.py spt_08/tmp_even.hdf spt_08/fsc_maskedtight_02.txt --calcfsc spt_08/tmp_odd.hdf
Fri Jun 28 14:54:58 2024: e2proc3d.py spt_08/threed_02_even.hdf spt_08/tmp_even.hdf --multfile spt_08/mask.hdf
Fri Jun 28 14:54:59 2024: e2proc3d.py spt_08/threed_02_odd.hdf spt_08/tmp_odd.hdf --multfile spt_08/mask.hdf
Fri Jun 28 14:54:59 2024: e2proc3d.py spt_08/tmp_even.hdf spt_08/fsc_masked_02.txt --calcfsc spt_08/tmp_odd.hdf
Fri Jun 28 14:55:00 2024: e2proc3d.py spt_08/threed_02_even.hdf spt_08/threed_even_unmasked.hdf  --process filter.lowpass.tophat:cutoff_abs=0.5
Fri Jun 28 14:55:01 2024: e2proc3d.py spt_08/threed_02_odd.hdf spt_08/threed_odd_unmasked.hdf  --process filter.lowpass.tophat:cutoff_abs=0.5
Fri Jun 28 14:55:02 2024: e2fsc_real_local.py spt_08/threed_even_unmasked.hdf spt_08/threed_odd_unmasked.hdf --output spt_08/fscvol_02.hdf --outfilt spt_08/threed_02.hdf --outfilte spt_08/threed_02_even.hdf --outfilto spt_08/threed_02_odd.hdf --mask spt_08/mask.hdf --threads 50 --localsizea 39 --compressbits 12 --tophat -v 1
Using 1.35 A/pix
256 x 256 x 256
Using box-size:  288
Preparing for local calculation
143  threads
Fri Jun 28 14:58:23 2024: e2proc3d.py spt_08/threed_02.hdf spt_08/threed_02.hdf --multfile spt_08/mask.hdf --process normalize.edgemean  
Fri Jun 28 14:58:24 2024: e2proc3d.py spt_08/threed_02_even.hdf spt_08/threed_02_even.hdf --multfile spt_08/mask.hdf --process normalize.edgemean  
Fri Jun 28 14:58:25 2024: e2proc3d.py spt_08/threed_02_odd.hdf spt_08/threed_02_odd.hdf  --multfile spt_08/mask.hdf --process normalize.edgemean  
Resolution (FSC<0.2) is ~11.9 A
Iteration 1 complete, 6:26:11
######## iter 3 ##########
### Defocus refinement....
e2spt_subtlt_local.py --ref spt_08/threed_02.hdf --path spt_08 --iter 3 --maxres 9.51 --parallel thread:50 --refine_defocus --aliptcls3d spt_08/aliptcls3d_01.lst --aliptcls2d spt_08/aliptcls2d_02.lst  --smooth 100 --smoothN 15 --goldcontinue
1696761 jobs on 50 CPUs
Error running task :  31
e2spa_make3d.py --input spt_08/aliptcls2d_03.lst --output spt_08/threed_03_even.hdf --keep 0.9 --clsid even --outsize 256 --sym c1  --parallel thread:50
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spa_make3d.py", line 445, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2880, in db_emd_init
    self.read_image(*parms)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2927, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2638, in read_into_image
    n,fsp,jsondict=self.read(N)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2590, in read
    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
IndexError: Attempt to read record 0 from #LSX spt_08/aliptcls2d_03.lst with 0 records
e2proc3d.py spt_08/threed_03_even.hdf spt_08/threed_raw_even.hdf --compressbits 12
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2proc3d.py", line 856, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2proc3d.py", line 174, in main
    infile, _ = parse_infile_arg(infile)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 1087, in parse_infile_arg
    raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'spt_08/threed_03_even.hdf' is not an existing regular file!
e2spa_make3d.py --input spt_08/aliptcls2d_03.lst --output spt_08/threed_03_odd.hdf --keep 0.9 --clsid odd --outsize 256 --sym c1  --parallel thread:50
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spa_make3d.py", line 445, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2880, in db_emd_init
    self.read_image(*parms)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2927, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2638, in read_into_image
    n,fsp,jsondict=self.read(N)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2590, in read
    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
IndexError: Attempt to read record 0 from #LSX spt_08/aliptcls2d_03.lst with 0 records
e2proc3d.py spt_08/threed_03_odd.hdf spt_08/threed_raw_odd.hdf --compressbits 12
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2proc3d.py", line 856, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2proc3d.py", line 174, in main
    infile, _ = parse_infile_arg(infile)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 1087, in parse_infile_arg
    raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'spt_08/threed_03_odd.hdf' is not an existing regular file!
e2refine_postprocess.py --even spt_08/threed_03_even.hdf   --tophat local --threads 50 --restarget 9.51 --sym c1
Use iteration number 3.
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2refine_postprocess.py", line 563, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2refine_postprocess.py", line 116, in main
    hdr=EMData(evenfile,0,1)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2880, in db_emd_init
    self.read_image(*parms)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 2932, in db_read_image
    fsp, idxs = parse_infile_arg(fsp)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/EMAN2.py", line 1087, in parse_infile_arg
    raise Exception(f"'{fname}' is not an existing regular file!")
Exception: 'spt_08/threed_03_even.hdf' is not an existing regular file!
Traceback (most recent call last):
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spt_refine_new.py", line 416, in <module>
    main()
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spt_refine_new.py", line 319, in main
    r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spt_refine_new.py", line 399, in calc_resolution
    fsc=np.loadtxt(fscfile)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/numpy/lib/npyio.py", line 1065, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/numpy/lib/_datasource.py", line 194, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/lib/python3.9/site-packages/numpy/lib/_datasource.py", line 531, in open
    raise IOError("%s not found." % path)
OSError: spt_08/fsc_masked_03.txt not found.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/f56cf43f-c9ec-43fb-901d-9217b03c3fe6n%40googlegroups.com.

[Muyuan Chen]

I would recommend just skip the defocus refinement, since it does not help much anyway. If you really want to debug this, run 

e2spt_subtlt_local.py --ref spt_08/threed_02.hdf --path spt_08 --iter 3 --maxres 9.51 --parallel thread:50 --refine_defocus --aliptcls3d spt_08/aliptcls3d_01.lst --aliptcls2d spt_08/aliptcls2d_02.lst  --smooth 100 --smoothN 15 --goldcontinue --debug

will provide more information on the error. 

Muyuan

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CACisnzojYgLz57q7ZJXps4dhed%3D1JMvBCR%3D%3D1pFqP6%2BnDmK2yQ%40mail.gmail.com.

[Mikey Grunst]

Thanks Muyan! I will see what I can find.

So in general, if I want to run a defocus refinement or a subtilt refinement alone, I cannot use e2spt_refine_new.py? This program seems to only allow t iterations after p iterations, and only allows d iterations after t iterations. I would need to use the e2spt_subtlt_local.py command? This would of course be starting from a previous run(s) in which I did other p and t iterations.

For example I ran this command using reference data in which I completed p and t iterations:

/vast/palmer/apps/avx2/software/EMAN2/2.99.47-SPHIRE-1.4-SPARX/bin/e2spt_refine_new.py --ptcls=spt_10/aliptcls3d_01.lst --ref=spt_10/threed_01.hdf --startres=11.9 --goldcontinue --sym=c1 --iters=t,t,p --keep=0.9 --tophat=local --loadali2d=spt_10/aliptcls2d_00.lst --parallel=thread:60 --threads=60 --ppid=-2

and I get the error:

NOT Writing notes, ppid=-2
Writing in spt_11...

Gathering metadata...
 27910/27910
Preparing references...

e2proc3d.py spt_10/threed_01_even.hdf spt_11/threed_00_even.hdf
e2proc3d.py spt_10/threed_01_odd.hdf spt_11/threed_00_odd.hdf
e2proc3d.py spt_11/threed_00_even.hdf spt_11/threed_00.hdf --addfile spt_11/threed_00_odd.hdf --mult 0.5
e2proclst.py spt_10/aliptcls2d_00.lst --create spt_11/aliptcls2d_00.lst
Processing 1696761 images in spt_10/aliptcls2d_00.lst
######## iter 1 ##########
### Subtilt translational refinement....
Need 3D particle alignment before subtilt refinement. exit.

Thank you!

-Mikey

[Muyuan Chen]

I thought it would work if you add --loadali3d? 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/2b788431-c372-431b-9e7d-a9d0b0187732n%40googlegroups.com.

[Mikey Grunst]

Hi Muyan,

I am finally returning to this. Thank you for your suggestions. I am not sure if defocus refinement would make a big difference. I am processing data collected with VPP and -0.5 defocus. The intial defocus estimations seemed reasonable. 

I ran your suggested command (different run now):

Command: e2spt_subtlt_local.py --ref spt_21/threed_01.hdf --path spt_21 --iter 3 --maxres 9 --parallel thread:6 --refine_defocus --aliptcls3d spt_21/aliptcls3d_00.lst --aliptcls2d spt_21/aliptcls2d_01.lst  --smooth 100 --smoothN 15 --goldcontinue --debug

Output: /home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

24 jobs on 6 CPUs
0
loading metadata from spt_21/particle_info_2d.lst, spt_21/particle_info_3d.lst...
max res: 9.00, max box size 120
0 - 0.1450000000000003 : -0.10958458622663779 -> -0.12542641617163322
Transform({'tx':-0.85,'ty':-0.37,'alpha':0.000,'mirror':0,'scale':1.0000,'type':'2d'})
Transform({'az':-50.330,'alt':139.965,'phi':70.144,'tx':-0.58,'ty':8.24,'tz':2.80,'mirror':0,'scale':1.0000,'type':'eman'})
25
6 - -0.01999999999999985 : -0.1517898496885824 -> -0.15760333633507476
Transform({'tx':-4.08,'ty':1.09,'alpha':0.000,'mirror':0,'scale':1.0000,'type':'2d'})
Transform({'az':-43.539,'alt':128.799,'phi':75.051,'tx':-4.40,'ty':10.11,'tz':2.49,'mirror':0,'scale':1.0000,'type':'eman'})
50
12 - -0.004999999999999838 : -0.1859816575947368 -> -0.19149269113886036
Transform({'tx':-0.48,'ty':1.87,'alpha':0.000,'mirror':0,'scale':1.0000,'type':'2d'})
Transform({'az':-39.040,'alt':117.392,'phi':77.674,'tx':-0.82,'ty':11.10,'tz':2.12,'mirror':0,'scale':1.0000,'type':'eman'})
75
18 - -0.0799999999999999 : -0.20834129548356073 -> -0.21934622524374092
Transform({'tx':-1.48,'ty':1.06,'alpha':0.000,'mirror':0,'scale':1.0000,'type':'2d'})
Transform({'az':-35.375,'alt':105.990,'phi':79.069,'tx':-2.79,'ty':10.42,'tz':1.66,'mirror':0,'scale':1.0000,'type':'eman'})
100

It looks like this was working, but no output files.

I also ran:

Command:
e2spt_refine_new.py --ptcls=spt_21/aliptcls3d_00.lst --ref=spt_21/threed_01.hdf --startres=9 --goldcontinue --sym=c1 --iters=d --keep=0.90 --tophat=local --localrefine --loadali2d=spt_20/aliptcls2d_01.lst --maxres=0.0 --minres=0.0 --parallel=thread:6 --threads=6 --loadali3d --mask=spt_19/relionmask/relionmask.hdf --ppid=-2

Upon looking at the newly generated aliptcls2d_01.lst file, I see there are no particles. The run had progressed to 100% but seems to be failing at the e2make3d.py step (Error running task: 3). You can see I also tried running with only 6 threads in case that would help.

Output (sorry it's long)

e2spt_refine_new.py --ptcls=spt_21/aliptcls3d_00.lst --ref=spt_21/threed_01.hdf --startres=9 --goldcontinue --sym=c1 --iters=d --keep=0.90 --tophat=local --localrefine --loadali2d=spt_20/aliptcls2d_01.lst --maxres=0.0 --minres=0.0 --parallel=thread:6 --threads=6 --loadali3d --mask=spt_19/relionmask/relionmask.hdf --ppid=-2
Writing in spt_22...
Gathering metadata...
 14024/14024
Preparing references...
e2proc3d.py spt_21/threed_01_even.hdf spt_22/threed_00_even.hdf
e2proc3d.py spt_21/threed_01_odd.hdf spt_22/threed_00_odd.hdf
e2proc3d.py spt_22/threed_00_even.hdf spt_22/threed_00.hdf --addfile spt_22/threed_00_odd.hdf --mult 0.5
e2proclst.py spt_21/aliptcls3d_00.lst --create spt_22/aliptcls3d_00.lst
Processing 14024 images in spt_21/aliptcls3d_00.lst
e2proclst.py spt_20/aliptcls2d_01.lst --create spt_22/aliptcls2d_00.lst
Processing 854367 images in spt_20/aliptcls2d_01.lst
######## iter 1 ##########
### Defocus refinement....
e2spt_subtlt_local.py --ref spt_22/threed_00.hdf --path spt_22 --iter 1 --maxres 9.00 --parallel thread:6 --refine_defocus --aliptcls3d spt_22/aliptcls3d_00.lst --aliptcls2d spt_22/aliptcls2d_00.lst  --smooth 100 --smoothN 15 --goldcontinue
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

854367 jobs on 6 CPUs
Error running task :  3
e2spa_make3d.py --input spt_22/aliptcls2d_01.lst --output spt_22/threed_01_even.hdf --keep 0.90 --clsid even --outsize 128 --sym c1  --parallel thread:6
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spa_make3d.py", line 449, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2888, in db_emd_init
    self.read_image(*parms)
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2935, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2646, in read_into_image
    n,fsp,jsondict=self.read(N)
                   ^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2598, in read

    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: Attempt to read record 0 from #LSX spt_22/aliptcls2d_01.lst with 0 records
e2proc3d.py spt_22/threed_01_even.hdf spt_22/threed_raw_even.hdf --compressbits 12

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2proc3d.py", line 853, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2proc3d.py", line 297, in main
    else : nimg = EMUtil.get_image_count(infile)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 3248, in db_get_image_count
    return EMUtil.get_image_count_c(fsp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf6 in position 1: invalid start byte
e2spa_make3d.py --input spt_22/aliptcls2d_01.lst --output spt_22/threed_01_odd.hdf --keep 0.90 --clsid odd --outsize 128 --sym c1  --parallel thread:6
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spa_make3d.py", line 449, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2888, in db_emd_init
    self.read_image(*parms)
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2935, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2646, in read_into_image
    n,fsp,jsondict=self.read(N)
                   ^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2598, in read

    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: Attempt to read record 0 from #LSX spt_22/aliptcls2d_01.lst with 0 records
e2proc3d.py spt_22/threed_01_odd.hdf spt_22/threed_raw_odd.hdf --compressbits 12

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2proc3d.py", line 853, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2proc3d.py", line 297, in main
    else : nimg = EMUtil.get_image_count(infile)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 3248, in db_get_image_count
    return EMUtil.get_image_count_c(fsp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd7 in position 0: invalid continuation byte
e2refine_postprocess.py --even spt_22/threed_01_even.hdf   --tophat local --threads 6 --restarget 9.00 --align --sym c1  --mask spt_19/relionmask/relionmask.hdf
Use iteration number 1.

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2refine_postprocess.py", line 576, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2refine_postprocess.py", line 116, in main
    hdr=EMData(evenfile,0,1)
        ^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2888, in db_emd_init
    self.read_image(*parms)
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2940, in db_read_image
    fsp, idxs = parse_infile_arg(fsp)
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 1095, in parse_infile_arg

    raise Exception(f"'{fname}' is not an existing regular file!")

Exception: 'spt_22/threed_01_even.hdf' is not an existing regular file!

Traceback (most recent call last):

  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spt_refine_new.py", line 421, in <module>
    main()
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spt_refine_new.py", line 327, in main

    r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spt_refine_new.py", line 407, in calc_resolution
    fsc=np.loadtxt(fscfile)
        ^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/lib/npyio.py", line 1338, in loadtxt
    arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/lib/npyio.py", line 975, in _read

    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/lib/_datasource.py", line 193, in open

    return ds.open(path, mode, encoding=encoding, newline=newline)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/lib/_datasource.py", line 533, in open
    raise FileNotFoundError(f"{path} not found.")
FileNotFoundError: spt_22/fsc_masked_01.txt not found.

e2version.py output:

EMAN 2.99.55 ( GITHUB: 2024-05-15 13:18 - commit: 2aa2b781e )

Your EMAN2 is running on: Linux-6.8.0-52-generic-x86_64-with-glibc2.35 6.8.0-52-generic

Your Python version is: 3.11.9

Thank you for the help and suggestions!

-Mikey

[Muyuan Chen]

I think right now it is easier to just skip the defocus refinement step? The fact that —debug works but the full command doesn’t suggest went wrong when more particles are processed. Either there is one or a few particles in the middle of particle stack that has some strange artifacts that mess up with defocus estimation, or some memory issue when processing many particles. Both are hard to diagnose…

On Apr 4, 2025, at 8:38 AM, Mikey Grunst <mikey....@yale.edu> wrote:

Hi Muyan,

To view this discussion visit https://groups.google.com/d/msgid/eman2/15e21fd8-c888-4450-8aef-331ef32fbaa5n%40googlegroups.com.

[Mikey Grunst]

Thanks Muyuan!

I agree. I appreciate your help! I will proceed with confidence knowing that I have not made some obvious mistake.

All the best,

-Mikey

To view this discussion visit https://groups.google.com/d/msgid/eman2/CE7F7C37-33EA-45AA-BEFB-FED8700C6537%40gmail.com.