A question about "New 3D Refinement"

[Liang Chen]

Dear EMAN2 Community,

After EMAN2 finished 10 iterations of the "New 3D Refinement" function, I tried many times with PCA-based classification. However, the following error always occurs:

IndexError: list index out of range

But EMAN2 has no such problems with the same data set finished using the old 3D Refinement. Could this be a python list bug?

Best Regards,

Liang Chen

[Ludtke, Steven J.]

Oops, looks like this question slipped through the cracks. This program was part of the old subtomogram averaging pipeline, and I doubt it was ever updated. Frankly, due to missing wedge effects, PCA was never an exceptionally good approach for subtomogram classification. There are several other classification programs which do work with the new pipeline. I would suggest using one of those instead:

e2spt_refinemulti_new.py 

e2spt_classify_byproj.py

I think I may be forgetting one other...

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/f067bd90-13d6-412a-8613-986db72c554bn%40googlegroups.com.

[Liang Chen]

I tried "e2spt_classify_byproj.py --ncls 2 --path spt_06 --iter 10 --threads 12", getting similar error as follows. I guess maybe there is a index bug in the new 3D refinement function.

crun:  0
Traceback (most recent call last):
  File "/home/lc20fk/yes/bin/e2spt_classify_byproj.py", line 288, in <module>
    main()
  File "/home/lc20fk/yes/bin/e2spt_classify_byproj.py", line 196, in main
    if prjs[0]==0 : print("ERROR: likely trying to use an incomplete or nonexistent iteration")

IndexError: list index out of range

[Ludtke, Steven J.]

Could you send the full output of e2version.py (on the machine you're running this on, obviously)? Also 'ls -al spt_06'

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/dad324fe-4322-4ac5-8ed4-504be609c8abn%40googlegroups.com.

[Liang Chen]

Hi,

Here are the result:

EMAN 2.99 ( GITHUB: 2022-03-01 08:48 - commit: 9a2f3ee )
Your EMAN2 is running on: Linux-5.3.0-28-generic-x86_64-with-glibc2.27 5.3.0-28-generic
Your Python version is: 3.9.10

-rw-rw-r--  1 lc20fk lc20fk      820 Jul  7 22:20 0_spt_params.json
-rw-rw-r--  1 lc20fk lc20fk 10574561 Jul  7 22:33 aliptcls2d_01.lst
-rw-rw-r--  1 lc20fk lc20fk 10541924 Jul  7 22:56 aliptcls2d_02.lst
-rw-rw-r--  1 lc20fk lc20fk 10639835 Jul  7 23:23 aliptcls2d_03.lst
-rw-rw-r--  1 lc20fk lc20fk 10574561 Jul  7 23:56 aliptcls2d_04.lst
-rw-rw-r--  1 lc20fk lc20fk 13903535 Jul  8 00:25 aliptcls2d_05.lst
-rw-rw-r--  1 lc20fk lc20fk 13903535 Jul  8 01:09 aliptcls2d_06.lst
-rw-rw-r--  1 lc20fk lc20fk 16840865 Jul  8 07:40 aliptcls2d_07.lst
-rw-rw-r--  1 lc20fk lc20fk 10574561 Jul  8 08:41 aliptcls2d_08.lst
-rw-rw-r--  1 lc20fk lc20fk 10639835 Jul  8 09:54 aliptcls2d_09.lst
-rw-rw-r--  1 lc20fk lc20fk 17265146 Jul  8 10:08 aliptcls2d_10.lst
-rw-rw-r--  1 lc20fk lc20fk   274126 Jul  7 22:33 aliptcls3d_01.lst
-rw-rw-r--  1 lc20fk lc20fk   276104 Jul  7 22:56 aliptcls3d_02.lst
-rw-rw-r--  1 lc20fk lc20fk   278082 Jul  7 23:23 aliptcls3d_03.lst
-rw-rw-r--  1 lc20fk lc20fk   278082 Jul  7 23:56 aliptcls3d_04.lst
-rw-rw-r--  1 lc20fk lc20fk   276104 Jul  8 08:41 aliptcls3d_08.lst
-rw-rw-r--  1 lc20fk lc20fk   276104 Jul  8 09:54 aliptcls3d_09.lst
-rw-rw-r--  1 lc20fk lc20fk     6958 Jul  8 11:17 .browsercache.json
-rw-rw-r--  1 lc20fk lc20fk     2375 Jul  7 22:39 fsc_masked_01.txt
-rw-rw-r--  1 lc20fk lc20fk     2297 Jul  7 23:02 fsc_masked_02.txt
-rw-rw-r--  1 lc20fk lc20fk     2266 Jul  7 23:29 fsc_masked_03.txt
-rw-rw-r--  1 lc20fk lc20fk     2255 Jul  8 00:01 fsc_masked_04.txt
-rw-rw-r--  1 lc20fk lc20fk     2270 Jul  8 00:30 fsc_masked_05.txt
-rw-rw-r--  1 lc20fk lc20fk     2270 Jul  8 01:15 fsc_masked_06.txt
-rw-rw-r--  1 lc20fk lc20fk     2253 Jul  8 07:45 fsc_masked_07.txt
-rw-rw-r--  1 lc20fk lc20fk     2240 Jul  8 08:46 fsc_masked_08.txt
-rw-rw-r--  1 lc20fk lc20fk     2232 Jul  8 10:00 fsc_masked_09.txt
-rw-rw-r--  1 lc20fk lc20fk     2252 Jul  8 10:14 fsc_masked_10.txt
-rw-rw-r--  1 lc20fk lc20fk     2375 Jul  7 22:39 fsc_maskedtight_01.txt
-rw-rw-r--  1 lc20fk lc20fk     2297 Jul  7 23:02 fsc_maskedtight_02.txt
-rw-rw-r--  1 lc20fk lc20fk     2267 Jul  7 23:29 fsc_maskedtight_03.txt
-rw-rw-r--  1 lc20fk lc20fk     2246 Jul  8 00:01 fsc_maskedtight_04.txt
-rw-rw-r--  1 lc20fk lc20fk     2264 Jul  8 00:30 fsc_maskedtight_05.txt
-rw-rw-r--  1 lc20fk lc20fk     2274 Jul  8 01:15 fsc_maskedtight_06.txt
-rw-rw-r--  1 lc20fk lc20fk     2247 Jul  8 07:45 fsc_maskedtight_07.txt
-rw-rw-r--  1 lc20fk lc20fk     2249 Jul  8 08:46 fsc_maskedtight_08.txt
-rw-rw-r--  1 lc20fk lc20fk     2238 Jul  8 10:00 fsc_maskedtight_09.txt
-rw-rw-r--  1 lc20fk lc20fk     2259 Jul  8 10:14 fsc_maskedtight_10.txt
-rw-rw-r--  1 lc20fk lc20fk     2389 Jul  7 22:39 fsc_unmasked_01.txt
-rw-rw-r--  1 lc20fk lc20fk     2309 Jul  7 23:02 fsc_unmasked_02.txt
-rw-rw-r--  1 lc20fk lc20fk     2284 Jul  7 23:29 fsc_unmasked_03.txt
-rw-rw-r--  1 lc20fk lc20fk     2254 Jul  8 00:01 fsc_unmasked_04.txt
-rw-rw-r--  1 lc20fk lc20fk     2280 Jul  8 00:30 fsc_unmasked_05.txt
-rw-rw-r--  1 lc20fk lc20fk     2275 Jul  8 01:14 fsc_unmasked_06.txt
-rw-rw-r--  1 lc20fk lc20fk     2271 Jul  8 07:45 fsc_unmasked_07.txt
-rw-rw-r--  1 lc20fk lc20fk     2252 Jul  8 08:46 fsc_unmasked_08.txt
-rw-rw-r--  1 lc20fk lc20fk     2248 Jul  8 10:00 fsc_unmasked_09.txt
-rw-rw-r--  1 lc20fk lc20fk     2279 Jul  8 10:14 fsc_unmasked_10.txt
-rw-rw-r--  1 lc20fk lc20fk  1113765 Jul  8 10:14 mask.hdf
-rw-rw-r--  1 lc20fk lc20fk  1048704 Jul  8 10:14 mask_tight.hdf
-rw-rw-r--  1 lc20fk lc20fk  7474046 Jul  7 22:20 particle_info_2d.lst
-rw-rw-r--  1 lc20fk lc20fk   339400 Jul  7 22:20 particle_info_3d.lst
-rw-rw-r--  1 lc20fk lc20fk        2 Jul 14 19:13 particle_parms_08.json
-rw-rw-r--  1 lc20fk lc20fk        2 Jul 14 19:12 particle_parms_10.json
-rw-rw-r--  1 lc20fk lc20fk 55308976 Jul  7 22:20 threed_00_even.hdf
-rw-rw-r--  1 lc20fk lc20fk 55308968 Jul  7 22:20 threed_00.hdf
-rw-rw-r--  1 lc20fk lc20fk 55308976 Jul  7 22:20 threed_00_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  7702144 Jul  7 22:39 threed_01_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  7473841 Jul  7 22:39 threed_01.hdf
-rw-rw-r--  1 lc20fk lc20fk  8609595 Jul  7 22:39 threed_01_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  6728756 Jul  7 23:02 threed_02_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  6330957 Jul  7 23:02 threed_02.hdf
-rw-rw-r--  1 lc20fk lc20fk 10765136 Jul  7 23:02 threed_02_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  8000377 Jul  7 23:29 threed_03_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5733152 Jul  7 23:29 threed_03.hdf
-rw-rw-r--  1 lc20fk lc20fk 11352866 Jul  7 23:29 threed_03_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  6367255 Jul  8 00:02 threed_04_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5460736 Jul  8 00:02 threed_04.hdf
-rw-rw-r--  1 lc20fk lc20fk 11605339 Jul  8 00:02 threed_04_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  6431507 Jul  8 00:30 threed_05_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5642422 Jul  8 00:30 threed_05.hdf
-rw-rw-r--  1 lc20fk lc20fk 12117169 Jul  8 00:30 threed_05_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  6432421 Jul  8 01:15 threed_06_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5409207 Jul  8 01:15 threed_06.hdf
-rw-rw-r--  1 lc20fk lc20fk 12183136 Jul  8 01:15 threed_06_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  9395406 Jul  8 07:46 threed_07_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5546042 Jul  8 07:46 threed_07.hdf
-rw-rw-r--  1 lc20fk lc20fk 12843659 Jul  8 07:46 threed_07_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  9318533 Jul  8 08:46 threed_08_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5184470 Jul  8 08:47 threed_08.hdf
-rw-rw-r--  1 lc20fk lc20fk 13293537 Jul  8 08:47 threed_08_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  9937373 Jul  8 10:00 threed_09_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  5191785 Jul  8 10:00 threed_09.hdf
-rw-rw-r--  1 lc20fk lc20fk 13288020 Jul  8 10:00 threed_09_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk  9924236 Jul  8 10:15 threed_10_even.hdf
-rw-rw-r--  1 lc20fk lc20fk  8526127 Jul  8 10:15 threed_10.hdf
-rw-rw-r--  1 lc20fk lc20fk 55306832 Jul  9 18:54 threed_10_masked_even.hdf
-rw-rw-r--  1 lc20fk lc20fk 55306832 Jul  9 18:54 threed_10_masked_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk 13209492 Jul  8 10:15 threed_10_odd.hdf
-rw-rw-r--  1 lc20fk lc20fk 55306648 Jul  8 10:14 threed_even_unmasked.hdf
-rw-rw-r--  1 lc20fk lc20fk 55306648 Jul  8 10:14 threed_odd_unmasked.hdf

You received this message because you are subscribed to a topic in the Google Groups "EMAN2" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eman2/sairSVWRoQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/4B6E2A17-0E81-4ECF-BB2D-2115A2174B4A%40bcm.edu.

[Ludtke, Steven J.]

e2spt_classify_byproj was updated to work with the _new refinement on May 25. The _new programs are definitely an improvement, but they are new, so if you're working with them you may need to update EMAN2 every so often to get the latest changes.

cheers

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAMD%3DT6QeoOsLvW%2B5zhdxNEL6V%2BaRSwfkDsYHcyBwJdaBZ%3D7ckw%40mail.gmail.com.

[Liang Chen]

Thank you! I think it worked. Another question related to the new program, have you tested the new 3D refinement with helical symmetry? I also got similar error as "IndexError: list index out of range" . The entire error message is very long. The new function does well with symmetry "c", but "h" causes problems.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/4FA02E75-F4FF-49B6-8180-25F122218112%40bcm.edu.

[Muyuan Chen]

I think as long as you can apply the symmetry with e2proc3d, it should work fine. The helical symmetry should not affect the orientation search anyway.

On Jul 15, 2022, at 5:10 PM, Liang Chen <liangche...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAMD%3DT6RqB7ATRg88ZxjDpuJEFYyF29fKNWyM1BHJ3z%2B-pX%2BZWA%40mail.gmail.com.

[Ludtke, Steven J.]

Please note that we discovered last week that the nstart parameter is broken for helical symmetry. Make sure nstart=1if imposing helical symmetry.

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/4B0FD779-2DF3-4CEB-A1DB-938392F012FB%40gmail.com.

[Liang Chen]

Would you please elaborate a little more? I need to do helical reconstruction from my subtomograms and I used the command described by "https://blake.bcm.edu/emanwiki/EMAN2/Symmetry", something like "--sym H16:4:33:68" but only got errors.

By the way, have you tested the latest new initial model generator? It used to work but now produces many errors like "IndexError: list index out of range", no matter I apply symmetry or not.

[Liang Chen]

I changed nstart to 1, like "H16:1:33:68", but still got the long "IndexError: list index out of range" error.

[Ludtke, Steven J.]

Yes, the second parameter for H symmetry is nstart. We discovered that if nstart>1 it produces incorrect results. The code has not been fixed, but we set it up to produce an error or warning. If you try to use a symmetry with nstart!=1 right now, it will complain. At some point someone will fix this, but the person who wrote this code has been gone for perhaps 10 years, and we haven't had a chance to look into it yet. 

On the bright side, it works fine if nstart=1, and nstart is equivalent to C symmetry. So you as a stopgap measure you can simply symmetrize with

H16:1:33:68

and then

C4

and achieve the same thing.

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/3080efca-06fa-4308-a241-c242090ccf41n%40googlegroups.com.

[Ludtke, Steven J.]

While it's good that you're trying to tell us what the error is, "IndexError: list index out of range" is meaningless in and of itself. You need to send the exact command you ran, and the full error message (multiple lines usually), for us to be able to figure out what the problem is...

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/5336792e-1427-42b2-9b22-9f93d3b35d1an%40googlegroups.com.

[Liang Chen]

Here are everything I tried:

e2spt_refine_new.py --ptcls=sets/curve_02.lst --ref=cylinder_diameter100_box240_apix2.148.hdf --startres=50.0 --goldstandard --sym=H16:1:33:68 --iters=p --keep=0.95 --maxres=0.0 --minres=0.0 --parallel=thread:22 --threads=10

Writing in spt_24...
Gathering metadata...
 41/41
Preparing references...
e2proc3d.py cylinder_diameter100_box240_apix2.148.hdf spt_24/threed_00_even.hdf --process filter.lowpass.gauss:cutoff_freq=0.0200 --process filter.lowpass.randomphase:cutoff_freq=0.0200
e2proc3d.py cylinder_diameter100_box240_apix2.148.hdf spt_24/threed_00_odd.hdf --process filter.lowpass.gauss:cutoff_freq=0.0200 --process filter.lowpass.randomphase:cutoff_freq=0.0200
e2proc3d.py spt_24/threed_00_even.hdf spt_24/threed_00.hdf --addfile spt_24/threed_00_odd.hdf --mult 0.5
######## iter 1 ##########
### Subtomogram alignment....
e2spt_align_subtlt.py spt_24/particle_info_3d.lst spt_24/threed_00.hdf --path spt_24 --iter 1 --maxres 50.00 --sym H16:1:33:68 --parallel thread:22  --fromscratch --goldcontinue
41 jobs on 22 CPUs
Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fb512c49ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error running task :  19
e2spa_make3d.py --input spt_24/aliptcls2d_01.lst --output spt_24/threed_01_even.hdf --keep 0.95 --clsid even --outsize 240 --pad 480 --sym H16:1:33:68  --parallel thread:22
Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fa6b4d86ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fdb81158ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f73b5bc6ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f8dc4033ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f448fe39ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fc61dcd6ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f156314aee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f4e28c1cee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fdee4983ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f8bac957ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f087906cee0>
Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fa011462ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fc6b021fee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f26d7989ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f83ddaaaee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f356dfccee0>
Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f095036fee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f4500c51ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f9a9f51fee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f7a12ec3ee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f0bb885cee0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 473, in execute
    xfout=Transform(curxfs[0])

IndexError: list index out of range

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spa_make3d.py", line 439, in <module>
    main()
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2916, in db_emd_init
    self.read_image(*parms)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2975, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2673, in read_into_image
    n,fsp,jsondict=self.read(N)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2625, in read
    if n>=self.n : raise Exception("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
Exception: Attempt to read record 0 from #LSX spt_24/aliptcls2d_01.lst with 0 records
e2spa_make3d.py --input spt_24/aliptcls2d_01.lst --output spt_24/threed_01_odd.hdf --keep 0.95 --clsid odd --outsize 240 --pad 480 --sym H16:1:33:68  --parallel thread:22

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spa_make3d.py", line 439, in <module>
    main()
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spa_make3d.py", line 63, in main
    tmp=EMData(options.input,0,True)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2916, in db_emd_init
    self.read_image(*parms)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2975, in db_read_image
    return LSXFile(fsp).read_into_image(self,*parms)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2673, in read_into_image
    n,fsp,jsondict=self.read(N)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2625, in read
    if n>=self.n : raise Exception("Attempt to read record {} from #LSX {} with {} records".format(n,self.path,self.n))
Exception: Attempt to read record 0 from #LSX spt_24/aliptcls2d_01.lst with 0 records
e2refine_postprocess.py --even spt_24/threed_01_even.hdf   --threads 10 --restarget 50.00 --sym H16:1:33:68
Use iteration number 1.

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2refine_postprocess.py", line 561, in <module>
    main()
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2refine_postprocess.py", line 115, in main
    hdr=EMData(evenfile,0,1)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2921, in db_emd_init
    self.read_image_c(*parms)
RuntimeError

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_refine_new.py", line 377, in <module>
    main()
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_refine_new.py", line 290, in main
    r=calc_resolution(f"{path}/fsc_masked_{itr:02d}.txt")
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_refine_new.py", line 360, in calc_resolution
    fsc=np.loadtxt(fscfile)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/numpy/lib/npyio.py", line 961, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/numpy/lib/_datasource.py", line 195, in open
    return ds.open(path, mode, encoding=encoding, newline=newline)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/numpy/lib/_datasource.py", line 535, in open
    raise IOError("%s not found." % path)
OSError: spt_24/fsc_masked_01.txt not found.

[Muyuan Chen]

I see the problem here. At some point I changed the orientation generator from "eman" to "saff" (at line ~200 in e2spt_align_subtlt.py), since "saff" does a slightly better job in covering the edge of asymmetrical units at some symmetry (probably only visible in apoferritin). However, while the "eman" generator treats H symmetry as c1, "saff" does not deal with H symmetry at all, leading to the error. The easier fix would be to change it back to "eman", since there aren't many opportunities to deal with apoferritin (or anything of oct symmetry) anyway. It won't be too hard to fix the H symmetry in "saff" too, but I don't know who wrote it and don't want to upset people by changing their code. For now, the easiest way would be just to change your code locally, replacing saff with eman. I will let Steve decide what is a better fix for this.

Muyuan

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/a34311b7-ecda-44f6-b0cf-fe28b703be78n%40googlegroups.com.

[Liang Chen]

Thank you for the solution! It fixed the New 3D Refinement when doing helical symmetry. But the New Initial Model Generator is still not working, with or without symmetry. It used to work fine in the "2022-03-01" version .

e2spt_sgd_new.py sets/curve_02.lst --res=20.0 --niter=50 --shrink=1 --parallel=thread:22 --ncls=1 --batch=12 --learnrate=0.2 --ref=cylinder_diameter100_box240_apix2.148.hdf --sym=c4

Gathering metadata...
 41/41
17 jobs on 22 CPUs
Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f922121cf10>

Traceback (most recent call last):
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 228, in execute
    data=self.data[0]

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fbe16111f10>

Traceback (most recent call last):
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 228, in execute
    data=self.data[0]

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7fc9e6f12f10>

Traceback (most recent call last):
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 228, in execute
    data=self.data[0]

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7ff8465c0f10>

Traceback (most recent call last):
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 228, in execute
    data=self.data[0]

IndexError: list index out of range

Error executing:  <e2spt_align_subtlt.SptAlignTask object at 0x7f246ab3ff10>

Traceback (most recent call last):
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2parallel.py", line 96, in main
    ret=task.execute(empty_func)

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_align_subtlt.py", line 228, in execute
    data=self.data[0]

IndexError: list index out of range

Error running task :  17

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_sgd_new.py", line 240, in <module>
    main()
  File "/home/lc20fk/eman2-sphire-sparx/bin/e2spt_sgd_new.py", line 114, in main
    ali2d=load_lst_params(f"{path}/aliptcls2d_00.lst")
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2863, in load_lst_params
    lsx=LSXFile(fsp,True)
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2503, in __init__
    if ifexists: raise Exception("Error: lst file {} does not exist".format(path))
Exception: Error: lst file sptsgd_12/aliptcls2d_00.lst does not exist
Exception ignored in: <function LSXFile.__del__ at 0x7f17c85ca0d0>

Traceback (most recent call last):

  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2569, in __del__
    self.close()
  File "/home/lc20fk/eman2-sphire-sparx/lib/python3.9/site-packages/EMAN2.py", line 2580, in close
    if self.ptr!=None :
AttributeError: 'LSXFile' object has no attribute 'ptr'

[Muyuan Chen]

Maybe just set threads to 17? I think it is just that you give it more cpus than the number of particles in each batch.

On Jul 15, 2022, at 8:26 PM, Liang Chen <liangche...@gmail.com> wrote:

Thank you for the solution! It fixed the New 3D Refinement when doing helical symmetry. But the New Initial Model Generator is still not working, with or without symmetry. It used to work fine in the "2022-03-01" version .

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/d8a18f9b-a2e2-449f-ad8f-64f74f505889n%40googlegroups.com.

[Liang Chen]

Thank you very much! It worked!

[Liang Chen]

There is another problem coming up with the New 3D Refinement. It cannot do "Map particles to tomograms". The old 3D Refinement has no such problems with the same set of data.

e2spt_mapptclstotomo.py --path=spt_06 --iter=5 --tomo=tomograms/tomo__bin4.hdf

Using averaged map spt_06/threed_05.hdf
Apix from tomogram 8.59, apix from average 2.15. Shrinking factor 4.0
0 particles total.
No particles. Exiting

On Friday, July 15, 2022 at 11:40:50 PM UTC-4 g5v...@gmail.com wrote:

[Ludtke, Steven J.]

Hi Liang,

e2spt_mapptclstotomo.py --help

you need the --new option

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/6205f2ac-fd4c-4906-9896-3ad8138a38ean%40googlegroups.com.

[Liang Chen]

Thank you so much! I will read more carefully about the --help content!