e2spt_trajfromrefine.py error

13 views
Skip to first unread message

Maia Azubel

unread,
Aug 24, 2021, 7:10:07 PM8/24/21
to EMAN2
Hi Muyuan, 
I'm following your tutorial on spt focused refinement. 


Everything worked fine until I got to e2spt_trajectformrefine. 
Command and error below. 

Also, once you have the result how do you visualize the motion in Chimera?
Many thanks, 
Maia



EMAN 2.91 final ( GITHUB: 2021-03-08 11:36 - commit: 81caed2 )
Your Python version is: 3.7.9

Command
 e2spt_trajfromrefine.py --path spt_118 --iter 5
Refine stats saved to spt_118/refinestats_05.txt. The three columns are:
  Translation - mean: 23.92  std: 8.02
  Rotation    - mean: 101.05  std: 62.84
  Score       - mean: -0.30  std: 0.05
Generating eigen-trajectory...
Eigen values saved to spt_118/eigval_05.txt
Trajectory 0:
  Rotation    - x: -19.73  y: 1.99  z: -0.70
  Translation - x: -10.22  y: 6.34  z: -13.00
Trajectory file written to spt_118/traj_it05eg00.hdf
Trajectory 1:
  Rotation    - x: -4.74  y: -14.19  z: -14.02
  Translation - x: -11.42  y: -1.48  z: -3.08
Trajectory file written to spt_118/traj_it05eg01.hdf
Making averages along trajectories...
[-2. -1.  0.  1.  2.]
Traceback (most recent call last):
  File "/programs/x86_64-linux/eman2/2.91/bin/e2spt_trajfromrefine.py", line 163, in <module>
    main()
  File "/programs/x86_64-linux/eman2/2.91/bin/e2spt_trajfromrefine.py", line 143, in main
    dcout={keys[i]:dic00[keys[i]] for i in idx}
  File "/programs/x86_64-linux/eman2/2.91/bin/e2spt_trajfromrefine.py", line 143, in <dictcomp>
    dcout={keys[i]:dic00[keys[i]] for i in idx}
NameError: free variable 'dic00' referenced before assignment in enclosing scope

Muyuan Chen

unread,
Aug 24, 2021, 7:37:02 PM8/24/21
to em...@googlegroups.com
I think it is expecting another particles_params_xx.json file to be input through --parminit option, which would be the alignment parameters before the focused refinement. In theory, if the particles are extracted from an existing alignment, it should also work without --parminit. However, it seems that there is a bug in the version you are on, that triggers the error you have. Unfortunately I cannot fix the bug on my side, because the program has been updated significantly since March, to work with the new spt pipeline, and I have not made it compatible with previous versions yet. So the choices are: 1. find the json file for the global alignment and use as input. 2. edit the program on your side locally, simply changing "dic00" to "dic" in that line (File "/programs/x86_64-linux/eman2/2.91/bin/e2spt_trajfromrefine.py", line 143, in main
) should work.

To visualize, I use the volume series function in chimera. I have only recently noticed that it may have issue ordering the movie correctly if you open a long enough one (>10 frames). Still trying to figure out the best solution to that...

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/BY5PR02MB6657448BC5287280365D2666DFC59%40BY5PR02MB6657.namprd02.prod.outlook.com.

Maia Azubel

unread,
Aug 25, 2021, 2:42:43 PM8/25/21
to em...@googlegroups.com
Thanks for the fast response, Muyuan!

I tried
e2spt_trajfromrefine.py --path spt_118 --iter 5 --parminit spt_118/particle_parms_05.json
And it seems that worked (however I'm not 100% sure which particle_parms_xx.json I should use). No error message. (I'm pasting below the entire in-terminal message). I'm also confused regarding the *99* files I got.

I got as output the following files
refinestats_05.txt
eigval_05.txt
traj_it05eg00.hdf
traj_it05eg01.hdf
class_it05eg00.hdf
particle_parms_99.json
threed_99_even.hdf
threed_99_odd.hdf
fsc_unmasked_99.txt
class_it05eg01.hdf

A few more questions (I hope you don't mind)

traj_it05eg00.hdf and traj_it05eg01.hdf, differ in the number of eigval? Which one did you use to display trajectory in the example of have on e2tomo_more? (In my case traj_it05eg00.hdf exhibits larger movement than traj_it05eg01.hdf)

Is there a good test(rational) to change --nframe (I believe default is 5), --nptcl, or even --nstd.?
In which cases would you recommend to use --replace3d ?

And finally,  do the files class_it05egxx.hdf correspond to the 3D volumes before the trajectories are built?

Many thanks,
Maia



e2spt_trajfromrefine.py --path spt_118 --iter 5 --parminit spt_118/particle_parms_05.json
Refine stats saved to spt_118/refinestats_05.txt. The three columns are:
  Translation - mean: 0.00  std: 0.00
  Rotation    - mean: 0.00  std: 0.00
  Score       - mean: -0.30  std: 0.05
Generating eigen-trajectory...
Eigen values saved to spt_118/eigval_05.txt
Trajectory 0:
  Rotation    - x: 0.00  y: -0.00  z: -10.23
  Translation - x: 0.00  y: 0.00  z: -0.00
Trajectory file written to spt_118/traj_it05eg00.hdf
Trajectory 1:
  Rotation    - x: -0.00  y: 0.00  z: -1.17
  Translation - x: 0.00  y: 0.00  z: 0.00
Trajectory file written to spt_118/traj_it05eg01.hdf
Making averages along trajectories...
[-2. -1.  0.  1.  2.]
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.19
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.18
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.16
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.18
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.17
500  threads
[-2. -1.  0.  1.  2.]
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.16
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.17
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.18
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.18
500  threads
e2spt_average.py --path spt_118 --iter 99 --skippostp --threads 4 --keep 1
Keeping 100% particles with score below -0.18
500  threads




From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Muyuan Chen <g5v...@gmail.com>
Sent: Tuesday, August 24, 2021 4:36 PM
To: em...@googlegroups.com <em...@googlegroups.com>
Subject: Re: [EMAN2] e2spt_trajfromrefine.py error
 

Muyuan Chen

unread,
Aug 25, 2021, 4:10:47 PM8/25/21
to em...@googlegroups.com
The _99_ files are the temporary files when creating the averages. They should be the same as the last average generated. Don’t worry about them. 

Yes eg00 and eg01 are different in eigen value. In theory the scale of movement should decrease as the eg number increases. 

nframe is just for visualization, depending on how many frames you want to show per movie. Nptcl controls how many particles in each frame. You can increase the number if the maps look noisy, but different classes may overlap and makes it harder to figure out the motion. You can use a larger nstd if the particles are sparsely distributed in the pca space. It can cover a larger range of the movement but averages at the edge of the distribution will have lower quality. 

I don’t know the answer to the rest of the questions from the top of my head. As I mentioned, I have made significant changes to the protocol since 2.91, and some of the mechanisms have been changed. You can find a short description of the later version of the program if you search e2tomo_new in the eman2 wiki. However there is still compatibility issues between the two versions so beware if you want to try the new one.

Maia Azubel

unread,
Aug 25, 2021, 4:37:38 PM8/25/21
to em...@googlegroups.com
Many thanks!
Maia

Sent: Wednesday, August 25, 2021 1:10 PM

Maia Azubel

unread,
Sep 13, 2021, 3:05:27 PM9/13/21
to em...@googlegroups.com
Hi Muyuan, 

I want to give it a try to the new protocols you implemented for spt_trajectfromrefine.py 

So I after building EMAN2/SPARX/SPHIRE from source 

>e2version.py
EMAN 2.99 ( GITHUB: 2021-09-13 10:49 - commit: ba39969d8 )
Your Python version is: 3.7.9

Below you mentioned 'However there is still compatibility issues between the two versions so beware if you want to try the new one.'
Would you recommend to reprocess? (For internal reasons I'll have to transfer data and want to know if I should aim to transfer all, or just original tiltseries) 

Thanks!
Maia

From: em...@googlegroups.com <em...@googlegroups.com> on behalf of Maia Azubel <maz...@stanford.edu>
Sent: Wednesday, August 25, 2021 1:37 PM

MuyuanChen

unread,
Sep 13, 2021, 3:42:53 PM9/13/21
to em...@googlegroups.com
If you are on the latest version, it is recommended to run the “new” spt refinement routine here:

And you can pull out the continuous movement from the refinement result using the same program.

The tilt series alignment and tomogram reconstruction is compatible between the two versions, so you do not have to re-process everything. It might be better to rerun the particle extraction and the refinement, since there may be some metadata changes that can cause problems. There is also now a script in examples/e2spt_json2lst.py, that convert the particle_parms_xx.json file from the previous spt routine to the new one. It will keep the initially determined angles, but it is still safer to run one iteration of e2spt_refine_new to have the downstream analysis working properly. 

Muyuan

Maia Azubel

unread,
Sep 13, 2021, 3:53:45 PM9/13/21
to em...@googlegroups.com
Thanks! I'll try it. 
Maia

From: em...@googlegroups.com <em...@googlegroups.com> on behalf of MuyuanChen <g5v...@gmail.com>
Sent: Monday, September 13, 2021 12:42 PM
Reply all
Reply to author
Forward
0 new messages