Classification from e2spt_refine_new.py

[Jacob Croft]

Hello,

I'm trying to run 3D classification from the results of a refinement run with e2spt_refine_new.py in eman/nightly, and I'm getting errors.  Is classification supported for the outputs of the new refinement program yet?

For example, ortho-proj classification:

e2spt_classify_byproj.py --path=spt_103/ --iter=6 --ncls=5 --layers=2 --shrink=1 --threads=12 --hp=-1.0 --lp=20.0

crun:  0
Traceback (most recent call last):
  File "/opt/applications/eman/nightly/bin/e2spt_classify_byproj.py", line 288, in <module>
    main()
  File "/opt/applications/eman/nightly/bin/e2spt_classify_byproj.py", line 196, in main
    if prjs[0]==0 : print("ERROR: likely trying to use an incomplete or nonexistent iteration")
IndexError: list index out of range

And PCA classification:

e2spt_pcasplit.py --path=spt_103 --iter=-1 --nclass=5 --maxres=30.0 --sym=c1 --nbasis=5 --clean

Traceback (most recent call last):
  File "/opt/applications/eman/nightly/bin/e2spt_pcasplit.py", line 336, in <module>
    main()
  File "/opt/applications/eman/nightly/bin/e2spt_pcasplit.py", line 94, in main
    threed = EMData("{}/threed_{:02d}.hdf".format(options.path,options.iter))
  File "/opt/applications/eman/nightly/lib/python3.9/site-packages/EMAN2.py", line 2910, in db_emd_init
    self.__initc(*parms)
RuntimeError

......

Thanks! 

Jacob Croft

University of Washington

[Steve Ludtke]

Hi Jacob,

sorry for the slow reply, your message slipped through the cracks. However, as you suspected in the message below, there isn't much overlap between the "old pipeline" tools and the "new pipeline" tools. It probably wouldn't be a tremendous amount of work to get e2spt_classify_byproj to work, at least... I'll take a look.

Note that there is a:

e2spt_refinemulti_new.py

to go along with

e2spt_refine_new.py

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/0f67562f-6aff-4ca7-81ed-fad11af16ebcn%40googlegroups.com.

[Jacob Croft]

Hi Steve,

Thanks for the reply.  Looks like e2spt_refinemulti_new.py is what I'm after!

Best,

Jake

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/1D64291C-0E58-4C5C-BA57-A16883BD5998%40gmail.com.

[Ludtke, Steven J.]

I went ahead and fixed e2spt_classify_byproj to work with new or old style refinement jobs. It is now available on github if you install from source. I find that one convenient because it is fast and easy to assess the results. At the very least it gives a good starting point for the full refinement runs.

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAOa2YHA8aEFt%3DZEat9Sbe2ZppCL7tXfyxQGbPR16PPcYajXeWg%40mail.gmail.com.

[Jacob Croft]

Thanks Steve! I agree, it's nice to use that one quickly to figure out how many classes might be found in the data and get some starting references.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/7DC6C01D-3D3A-411E-8A37-75BA607B7C7C%40bcm.edu.

[Maia Azubel]

I updated source code today to use e2spt_classify_byproj.py with the new style of refinement (e2spt_refine_new.py) but I'm getting an error message

(eman2) [maia2@sr20-395c92b4ea eman]$ e2version.py

EMAN 2.99 ( GITHUB: 2022-08-04 13:35 - commit: 8a24bb112 )

Your Python version is: 3.7.9

(eman2) [maia2@sr20-395c92b4ea eman]$ e2spt_classify_byproj.py --path=spt_72 --iter=1 --ncls=5 --layers=2 --shrink=1 --threads=72 --hp=100.0 --lp=7.5

crun:  0

Traceback (most recent call last):

  File "/data/maia2/miniconda3/envs/eman2/bin/e2spt_classify_byproj.py", line 288, in <module>

    main()

  File "/data/maia2/miniconda3/envs/eman2/bin/e2spt_classify_byproj.py", line 196, in main

    if prjs[0]==0 : print("ERROR: likely trying to use an incomplete or nonexistent iteration")

IndexError: list index out of range

Any advice?

Thanks!

Maia

---

From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Jacob Croft <jac...@uw.edu>
Sent: Wednesday, May 25, 2022 8:39 AM
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] Classification from e2spt_refine_new.py

 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAOa2YHDLk5kkspOB2EeSeJwyPPaa-ynoihTEV7e9ZmEJ-vBajw%40mail.gmail.com.

[Ludtke, Steven J.]

Hi Maia,

sorry for the slow reply, but our term has already started, I'm teaching, and I wanted to try this myself before responding. What sort of iteration did you try to use from e2spt_refine_new?  Here is an example that worked for me:

e2spt_refine_new.py --ptcls sets/post06-mono2.lst --ref r_96_a.hdf --startres 60 --goldstandard --minres 150 --iters p,t,t,r --keep 0.5,0.8,0.9 --parallel thread:64:/home2/stevel/tmp --threads 64 --tophat global --mask mask_168_spoke_soft.hdf --m3dthread 24

e2spt_classify_byproj.py --path spt_05 --iter 1 --ncls 2 --nbasis 3 --layers 15 --mask mask_168_badmemrgn.hdf --threads 32

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB66570FC5D99756A9071D1543DF9F9%40BY5PR02MB6657.namprd02.prod.outlook.com.

[Maia Azubel]

Hi Steve,

Here's my e2spt_refine_new (outputs to spt_73)

e2spt_refine_new.py --ptcls sets/ptcls.lst --ref spt_72/threed_04.hdf --goldcontinue --parallel=thread:72 --threads=72 --tophat local --iter p,p  --smoothN 250 --smooth 200

After ran it I updated the version

EMAN 2.99 ( GITHUB: 2022-08-04 13:35 - commit: 8a24bb112 )
Your Python version is: 3.7.9

and when I try

e2spt_classify_byproj.py --path spt_73 --iter 1 --ncls 2 --nbasis 3 --layers 15 --mask whole_mask.hdf --threads 72

I get the following error message

crun:  0

Traceback (most recent call last):

  File "/data/maia2/miniconda3/envs/eman2/bin/e2spt_classify_byproj.py", line 288, in <module>

    main()

  File "/data/maia2/miniconda3/envs/eman2/bin/e2spt_classify_byproj.py", line 196, in main

    if prjs[0]==0 : print("ERROR: likely trying to use an incomplete or nonexistent iteration")

IndexError: list index out of range

Thanks!

Maia

---

From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Ludtke, Steven J. <slu...@bcm.edu>
Sent: Tuesday, August 9, 2022 6:06 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/B0F7DC0F-07D7-429F-8914-0C7D338AFF58%40bcm.edu.

[Ludtke, Steven J.]

Sorry, teaching still taking a lot of my time. My initial guess is that --goldcontinue is causing your issues. I vaguely recall Muyuan saying it wasn't working properly at one point. Try to do a full refinement sequence in one run, rather than breaking the sequence into bits...

--------------------------------------------------------------------------------------

Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine

Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 

Deputy Director, Advanced Technical Cores                   and Molecular Biology

Academic Director, CryoEM Core

Co-Director CIBR Center

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657F2F50F2F7210F9F94286DF659%40BY5PR02MB6657.namprd02.prod.outlook.com.