error running e2spt_refine_new.py

[Esther Bullitt]

Can you please help?  This failed after a full day of processing. There are very many helically symmetric particles in each original subtomogram of a filamentous virus nucleocapsid, and I made a mask of one asymmetric unit to try to improve the structure, and used a new list of particles that was created based on symmetry of the subtomogram:

 

cp spt_14/aliptcls3d_06.lst sets/spt14_06_sym.lst 

e2proclst.py sets/spt14_06_sym.lst --sym h96:1:30.51:1.82 

 

1) output from e2version.py

 

EMAN 2.99.72 ( GITHUB: 2026-03-06 14:19 - commit: NOT-INSTALLED-FROM-GIT-REPO )

Your EMAN2 is running on: Linux-4.18.0-553.89.1.el8_10.x86_64-x86_64-with-glibc2.28 4.18.0-553.89.1.el8_10.x86_64

Your Python version is: 3.12.13

2) exact command you ran to produce the failure, and if possible, the output from that command

 

/spshared/apps/miniconda3/envs/eman2/bin/e2spt_refine_new.py --ptcls=sets/spt14_06_sym.lst --ref=asym_unit_mask_lp8A_threed08a.hdf --startres=40.0 --goldstandard --sym=c1 --iters=p4,t2,p2,t,r,d --keep=0.95,0.8,0.8 --localrefine --maxres=8.0 --minres=35.0 --parallel=thread:16 --threads=32 --ppid=-2

 

Output is below (after question 5). 

 

3) if it relates to a single particle reconstruction, box size in pixels and microscope voltage --not spa
4) if running programs from the GUI, are you using e2workflow.py (deprecated) or e2projectmanager.py ?

e2projectmanager.py

5) please make sure to sign your message with a proper signature indicating your institution and research group (if academic), or some other basic information about who you are (to give context to your question).

 

Esther Bullitt, Boston University – subtomo averaging

 

(eman2) bullitt@atlas [LLOV_eman2] NOT Writing notes, ppid=-2

Writing in spt_16...

Gathering metadata...

Loading 3D particles - 133632 particles from 14 tomograms

 133632/133632

Preparing references...

e2proc3d.py asym_unit_mask_lp8A_threed08a.hdf spt_16/threed_00_even.hdf --process filter.lowpass.tophat:cutoff_freq=0.0250

e2proc3d.py asym_unit_mask_lp8A_threed08a.hdf spt_16/threed_00_odd.hdf --process filter.lowpass.tophat:cutoff_freq=0.0250

e2proc3d.py spt_16/threed_00_even.hdf spt_16/threed_00.hdf --addfile spt_16/threed_00_odd.hdf --mult 0.5

######## iter 1 ##########

### Subtomogram alignment....

e2spt_align_subtlt.py spt_16/particle_info_3d.lst spt_16/threed_00.hdf --path spt_16 --iter 1 --maxres 8.00 --sym c1 --parallel thread:16  --fromscratch --minres=35.0 --goldcontinue

133632 jobs on 16 CPUs

ls spt_16

0_spt_params.json     particle_info_3d.lst  threed_00.hdf

particle_info_2d.lst  threed_00_even.hdf    threed_00_odd.hdf

(eman2) bullitt@atlas [LLOV_eman2] ls spt_16

0_spt_params.json     particle_info_3d.lst  threed_00.hdf

particle_info_2d.lst  threed_00_even.hdf    threed_00_odd.hdf

 

(eman2) bullitt@atlas [LLOV_eman2] ls spt_16

0_spt_params.json     particle_info_3d.lst  threed_00.hdf

particle_info_2d.lst  threed_00_even.hdf    threed_00_odd.hdf

        

(eman2) bullitt@atlas [LLOV_eman2] Error running task :  8

e2spa_make3d.py --input spt_16/aliptcls2d_01.lst --output spt_16/threed_01_even.hdf --keep 0.95,0.8,0.8 --clsid even --outsize 128 --sym c1  --parallel thread:16

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 447, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 63, in main

    tmp=EMData(options.input,0,True)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2912, in db_emd_init

    self.read_image(*parms)

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2959, in db_read_image

    return LSXFile(fsp).read_into_image(self,*parms)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2670, in read_into_image

    n,fsp,jsondict=self.read(N)

                   ^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2616, in read

    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

IndexError: Attempt to read record 0 from #LSX spt_16/aliptcls2d_01.lst with 0 records

e2proc3d.py spt_16/threed_01_even.hdf spt_16/threed_raw_even.hdf --compressbits 12

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2proc3d.py", line 878, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2proc3d.py", line 305, in main

    else : nimg = EMUtil.get_image_count(infile)

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 3271, in db_get_image_count

    return EMUtil.get_image_count_c(fsp)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: FileAccessException at /home/conda/feedstock_root/build_artifacts/eman-dev_1772806501538/work/libEM/io/hdfio2.cpp:450: error with 'spt_16/threed_01_even.hdf': 'cannot access file 'spt_16/threed_01_even.hdf'' caught

 

e2spa_make3d.py --input spt_16/aliptcls2d_01.lst --output spt_16/threed_01_odd.hdf --keep 0.95,0.8,0.8 --clsid odd --outsize 128 --sym c1  --parallel thread:16

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 447, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spa_make3d.py", line 63, in main

    tmp=EMData(options.input,0,True)

        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2912, in db_emd_init

    self.read_image(*parms)

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2959, in db_read_image

    return LSXFile(fsp).read_into_image(self,*parms)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2670, in read_into_image

    n,fsp,jsondict=self.read(N)

                   ^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2616, in read

    if n>=self.n : raise IndexError("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

IndexError: Attempt to read record 0 from #LSX spt_16/aliptcls2d_01.lst with 0 records

e2proc3d.py spt_16/threed_01_odd.hdf spt_16/threed_raw_odd.hdf --compressbits 12

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2proc3d.py", line 878, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2proc3d.py", line 305, in main

    else : nimg = EMUtil.get_image_count(infile)

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 3271, in db_get_image_count

    return EMUtil.get_image_count_c(fsp)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

RuntimeError: FileAccessException at /home/conda/feedstock_root/build_artifacts/eman-dev_1772806501538/work/libEM/io/hdfio2.cpp:450: error with 'spt_16/threed_01_odd.hdf': 'cannot access file 'spt_16/threed_01_odd.hdf'' caught

 

e2refine_postprocess.py --even spt_16/threed_01_even.hdf   --threads 32 --restarget 8.00 --align --sym c1 

Use iteration number 1.

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 583, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2refine_postprocess.py", line 119, in main

    hdr=EMData(evenfile,0,1)

        ^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2912, in db_emd_init

    self.read_image(*parms)

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 2964, in db_read_image

    fsp, idxs = parse_infile_arg(fsp)

                ^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/EMAN2.py", line 1112, in parse_infile_arg

    raise Exception(f"'{fname}' is not an existing regular file!")

Exception: 'spt_16/threed_01_even.hdf' is not an existing regular file!

Traceback (most recent call last):

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spt_refine_new.py", line 524, in <module>

    main()

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spt_refine_new.py", line 354, in main

    r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/bin/e2spt_refine_new.py", line 510, in calc_resolution

    fsc=np.loadtxt(fscfile)

        ^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/numpy/lib/_npyio_impl.py", line 1384, in loadtxt

    arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter,

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/numpy/lib/_npyio_impl.py", line 1011, in _read

    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/numpy/lib/_datasource.py", line 192, in open

    return ds.open(path, mode, encoding=encoding, newline=newline)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/spshared/apps/miniconda3/envs/eman2/lib/python3.12/site-packages/numpy/lib/_datasource.py", line 529, in open

    raise FileNotFoundError(f"{path} not found.")

FileNotFoundError: spt_16/fsc_masked_01.txt not found.

 

[1]+  Killed                  e2projectmanager.py

 

(eman2) bullitt@atlas [LLOV_eman2] ls spt_16

0_spt_params.json  particle_info_2d.lst  threed_00_even.hdf  threed_00_odd.hdf

aliptcls2d_01.lst  particle_info_3d.lst  threed_00.hdf

thanks,

Esther -- my email is bul...@bu.edu (not @atlas)

[Steve Ludtke]

Ok, the error you're seeing:

IndexError: Attempt to read record 0 from #LSX spt_16/aliptcls2d_01.lst with 0 records

is downstream of the actual problem, which is the prior process for some reason failed to produce the .lst output file, meaning it probably crashed itself.   

 Muyuan may spot something going on, but here are my guesses:

My first guess would be that his is related to specifying --minres=35, which is pretty aggressive for any SPT project. Many SPT refinements (due to data issues generally) get "stuck" in refinement somewhere in the 25-40 Å resolution regime, so excluding any information lower than 35 Å is throwing away a large fraction of the SNR in the data, which may be needed to keep the data properly anchored at the correct location.

The helical symmetry you're using may be a problem as well.  It looks like you are making a .lst file with symmetry related copies of the particles. With a locally constrained symmetry like Cn, this works well. That is, if you have a particle in a particular orientation with C4 symmetry, there are 4 other identical orientations that particle might be in. However, with helical symmetry, the translational component of the symmetry means you are effectively shifting some of the helix out of the box. With helical symmetry, the helical repeat component of the symmetry needs to be handled by increasing the number of extracted particles along the length of the filament (overlapping boxes) rather than by imposing additional copies with translation on the original extracted particles. Do you see the difference? 

The only portion of the helical symmetry it makes sense to handle using the "e2proclst.py --sym" method is the Nstart component, eg - the Cn symmetry which may be present in the helix. Since you are specifying this as 1 in your command, I don't believe this process is useful for you. While I could see potentially having nsym large enough to cover 1 rotation (eg ~12), I'm not sure even this argument holds up very well, since the symmetry isn't being enforced downstream. That is, you are making an initial assertion that there is another valid orientation for the same set of data with a specific amount of rotation and a specific shift, but then you are allowing the refinement (c1 symmetry) to let it drift away from this value, so you really get no benefit that you wouldn't have gotten by generating more particles along the length of the filament (shift-rotation equivalence), while keeping all of the data in the box.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/a53bc2e7-19c9-4195-a7c3-466bc077bb39n%40googlegroups.com.

[Bullitt, Esther]

Hi Steve,

thank you.

I think you are suggesting that I switch to Approach 1 from https://blake.bcm.edu/emanwiki/doku.php?id=eman2:tomosymmetry

If this is the case, how do I deal with the - - postxf parameters in the command e2spt_extract.py ?

The helical symmetry is :  11.9 u/t, 72.5 A pitch, so 6.14 A rise per subunit; rotation angle 30.25

From my current refinement where the data are bin 2, resulting in 3.378A/pix; then 6.14A/3.378A/pix=1.82pix

Would it be - - postxf=h96:1:30.25:1.82 ,0,0,1.82

Can the final number, the z translation, be non-integer pixels?

But what about all the subunits where z goes down rather than up?

Also,  I am unclear how to get my mask (which is currently in the big box) properly put into a smaller box for this new approach.   What command would that use?

thanks very much.

Sincerely,

Esther

To view this discussion visit https://groups.google.com/d/msgid/eman2/C84D6D33-67DC-4F75-8DFA-2271A7C59F64%40gmail.com.

[Steve Ludtke]

Hi Esther,

Point symmetry is a somewhat tricky issue. Helical symmetry is worse. 

Let's say you have an object with a simple C4 symmetry, and one particle image you want to insert in the correct orientation into Fourier space. You have 3 choices: 1) determine the orientation and insert the particle 4 times, once in each symmetry-related orientation, 2) replicate the particle 4 times and put each in one symmetry-related orientation before refining the orientations, then insert each of the 4 copies in its one individual updated orientation (which will no longer strictly follow C4 symmetry) or 3) insert the particle in a single orientation (any one of the 4 symmetry-related orientations), then when you finish inserting all of the other particles (also one time each), you impose C4 symmetry on the final structure.  

How are these three approaches different?  The second is the most computationally intensive, and simultaneously assumes that you both do and don't have true C4 symmetry. The first is the next most computationally intensive, requiring roughly 4x the computation of the final approach. 

Mathematically, in terms of the 3-D structure, approach 1 and 3 are completely identical, and approach 2 is not internally self-consistent. The reason sometimes insertion in multiple orientations is used instead of averaging at the end is that particle quality assessment is often happening at insertion-time, and you _may_ get more accurate per-particle quality values if you include the multiple orientations in the calculation. It isn't really a very good reason, and if strict symmetry has been imposed on the 3-D map before quality assessment, even this doesn't make much mathematical sense.

In your case, I could see some potential benefit in something like h6:1:30.25:1.82 , maybe as much as h12, just to ensure that you don't get issues with missing wedge not getting filled in properly. The reason this might make sense is due to the very specific geometry you tend to have with filaments (lying flat on the grid). Beyond that, I don't see much potential benefit in either approach. 

If a helical particle's orientation is being very accurately determined and the helical parameters don't vary, there is really no benefit to over-sampling the length of the helix any finer than one "step" in the final imposed 3-D symmetry. That is, if you impose h96:1:30.25:1.82 symmetry and the spacing of boxes along the filament is much less than 96*1.82 pixels, you aren't likely to gain much of anything (as long as 96*1.82 is less than 1/2 the box size). This is all to say, as long as you impose the (correct) helical symmetry at the end of each iteration to use as a reference for the next iteration, there really shouldn't be any need to symmetry-replicate your particles.

--

Having said all that, any of the 3 approaches above might then be _followed_ by a subtraction approach where you attempt to extract the density for only 1 of the 4 subunits from each particle image as a separate particle image with no symmetry. You then do a symmetry-free refinement of those (4x as many) particles. This followup refinement can sometimes improve the resolution of the individual subunit structure if the symmetry isn't strictly obeyed. You can even take it further and apply classification/dynamics to these individual subunits to improve homogeneity.  

While this approach does make mathematical sense, it is _very_ sensitive to the size of the subunit you are trying to extract, as there often just isn't enough signal present in an individual subunit to clearly classify the particles. Frequently this method produces worse structures than the helical segment approach. Furthermore, when you do the subtraction process, it's the average structure that you're subtracting, which obviously won't be perfectly accurate if the particles themselves aren't perfectly symmetric, so the residual unsubtracted density may influence the quality or classification of the subtracted individual subunits.

To view this discussion visit https://groups.google.com/d/msgid/eman2/SA3PR03MB7516A22F3143E5C1D2D23368C9B92%40SA3PR03MB7516.namprd03.prod.outlook.com.

[Muyuan Chen]

Hi Esther, 

On vacation earlier. Gradually catching up. 

I agree with Steve that h96 would be way too much. You can reduce it by separating the particle duplicate and the symmetry applying process. Starting from your initial point (spt_14), if you just apply a lower symmetry number with e2proclst, for example

e2proclst.py sets/spt14_06_sym.lst --sym h4:1:274.59:16.38 

So essentially 9x subunits per copy. I think for helical symmetry, specifying h4 will make 4x copies above 0 and 4x copies below, so 9x copies total. This should cover 131 pixels, probably enough in most cases. 

Still, double check if everything works correctly and you have the right number of particles after applying symmetry to the list. 

Similar to the expansion of c symmetry (https://blake.bcm.edu/emanwiki/doku.php?id=eman2:e2tomo_atpsyn&s[]=atp#refinement_of_atp_synthase_monomers), gather metadata to make sure everything works first. 

e2spt_gathermeta.py --ptcls sets/spt14_06_sym.lst --ali2d spt_14/aliptcls2d_03.lst

e2spt_refine_new.py --path spt_15 --continuefrom 0.5 --localrefine --sym h4:1:30.51:1.82  .....

The reconstruction in spt_15/threed_01_even/odd should be same as your spt_14 result with symmetry applied. Here I specify --sym h4:1:30.51:1.82 so it fills the 8x subunit step size in e2proclst. So here you essentially have 81 copies. 9 from duplicated particles and 9 from the applying symmetry step in post process. You can also use a lower symmetry or keep it c1 for some validation. 

In theory you can also use the GMM based focused refinement to target one subunit without the subtraction. There are some documentation in the link above. I haven't used it on helical stuff before. In theory it should be the same but no guarantee...

 

Muyuan