re-extracting particles with offset

[Madeline Rollins]

Hi all,
Using the p22 tutorial as a guide, I am trying to re-extract a subset of my particles with an offset from my previous refinement job, which seems to be centered on the ribosome. My current approach for finding the new center for the particles is to:

1. Open my volume using the "Show 3D" button in the e2display browser
2. Insert a sphere
3. Under the "basic" tab (for the sphere), change the Tx, Ty, and Tz values until the sphere is located at the desired position

Then, I take those values for Tx, Ty, and Tz and run the extraction job (adding the coordinates to the --postxf option):
e2spt_extract.py --boxsz_unbin=384 --newlabel=sec61 --threads=32 --maxtilt=100 --padtwod=2.0 --shrink=1.0 --tltkeep=1.0 --jsonali=sptcls_25/aliptcls3d_10_01.lst --compressbits=8 --postxf=c1,5,-60,15 --mindist=100

After I run the extraction job, I used e2spt_sgd_new.py to create a new initial model using the re-extracted particles. I didn't get any error messages but the recentering didn't work - when I opened the output map from the spt_sgd job, inserted a sphere and changed the coordinates for the sphere to Tx=0, Ty=0, and Tz=0, the sphere was roughly positioned in the center of the ribosome . Is there something wrong with how I re-centered or re-extracted the particles? Also, is there a way to verify that I am applying the proper offset before extracting the particles?

Thanks,

Madeline

(Here is the ouput from e2version.py)
EMAN 2.99.66 ( GITHUB: 2025-03-13 19:17 - commit: ce0afb465-dirty )
Your EMAN2 is running on: Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-glibc2.28 4.18.0-305.57.1.el8_4.x86_64
Your Python version is: 3.12.9

[Muyuan Chen]

Hi Madeline,

I normally find the x,y,z from the slice or projection view, and sometimes in filtertool with mask.soft. I never used the sphere thing from show 3D widget but I suppose it should work too? Hopefully it is using the coordinate system of the volume instead of the rendering. Maybe Steve will comment on that. 

Using initial model generation to confirm the centering may not be a good approach though. I believe the e2spt_sgd_new program centers the reference by itself every few iterations, so it is not surprising that the center shifts back even if the coordinates provided by --postxf is correct. 

If you are following the p22 tutorial, one way to test is to run the e2spt_gathermetadata.py on the new particle set, then run a single iteration of refinement following that. It should make reconstruction at the coordinates defined by --postxf. 

Another perhaps easier way is to run 

e2proc3d.py sets/ptcls.lst avg.hdf --average --avg_byxf

It will just rotate the 3d particles to the pre-defined orientations and average them. The result isn't as good as reconstructing the 2d particles from tilt images, but you can check whether the transform is correct quickly. The average probably need to be lowpass filtered a bit, and it might also be slow if you have many particles since it is not parallelized. If the goal is just to check whether the coordinate is correct, you can just take a small subset to test. 

Muyuan

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/eman2/2913ca73-9eca-4605-bd1a-acdc1ef7f818n%40googlegroups.com.

[Madeline Rollins]

Thanks so much. I did try running e2spt_gathermetadata.py on the new particle set and I used the 2d alignment information from my initial subtomogram averaging run and a mask focused on my area of interest. The densities were distorted compared to the same e2spt_gathermetadata.py on the original particles, so I think the coordinates that I'm using from the show 3D widget are not correct.

I tried the using filter tool to find x,y,z but when I open it, I do not get a 3D view of my particle as in the tutorial - just the 2D view and a black screen (if I leave scale=0). If I click "Add 3D View" (from the View dropdown menu at the top of the filter tool window), the window freezes before closing. I don't get any errors in the terminal when this happens, so I'm not sure how to troubleshoot this. Do you know if this has happened before?

Madeline

[Muyuan Chen]

Typically, an incorrect —postxf shouldn’t lead to distorted density, only with shifted center. If you see distortion at the first iteration, without the masking, then something else might be wrong. Masks too tight can cause distorted densities at later iterations though. 

Holding shift while clicking filtertool on e2display browser can enter a safe mode where it might be less buggy. Select the isosurface and drag the threshold a bit may bring up more densities, and start from a large soft mask at center before shifting it.

Muyuan

On Jul 10, 2025, at 7:46 PM, Madeline Rollins <mgro...@uchicago.edu> wrote:

Thanks so much. I did try running e2spt_gathermetadata.py on the new particle set and I used the 2d alignment information from my initial subtomogram averaging run and a mask focused on my area of interest. The densities were distorted compared to the same e2spt_gathermetadata.py on the original particles, so I think the coordinates that I'm using from the show 3D widget are not correct.

To view this discussion visit https://groups.google.com/d/msgid/eman2/57f3ca56-443a-4ddf-be14-cf373cedeae9n%40googlegroups.com.

[Madeline Rollins]

Hi Muyuan,

Using filtertool with mask.soft worked very well for determining the x,y,z and successfully recentering my particles - thanks so much for that suggestion. I do have a follow-up question about running the refinement job for the recentered particles:

In the tutorial, the next step is to run e2spt_gathermetadata.py on the recentered particle set and then run a refinement job. My commands look like this:

e2spt_gathermetadata.py --ptcls sets/sec61.lst --ali2d spt_01/aliptcls2d_03.lst

e2spt_refine_new.py --path=spt_42 --continuefrom=0.5 --sym=c1 --parallel=thread:32:/scratch/beagle3/mgrollins --startres=20.0 --tophat=local --localrefine --iters=p,p,t --keep=0.95 --threads=32 --mask=masks/custom_mask.mrc

For the gather metadata step, the ptcls are my recentered particles and the ali2d input is the 2d alignment information from my initial subtomogram averaging (when my particles were centered on the ribosome). I think there might be some heterogeneity in the ribosome interaction at the membrane, which might cause some misalignment of the other densities that I'm interested in at the membrane. As a result, I think the reconstruction that is generated for the refinement using continuefrom=0.5 is misaligned. I think this makes sense because my reconstruction does not really improve over multiple iterations.

My idea was to just run the refinement job from scratch (without using e2spt_gathermetadata.py to get the 2d alignment information), but I'm not sure how to generate a new initial model. I tried using e2spt_sgd_new.py to generate a new initial model, but it seems like the center of the reference gradually shifts back to the ribosome.

Would the best approach be to run e2spt_gathermetadata.py using the 2d alignment information from one of the first iterations of my initial subtomogram averaging? My idea is that since the particles were not that well-aligned in the earlier iterations, that it would provide an opportunity to improve the alignment over multiple iterations. Do you have any suggestions on the best way to generate an initial model for recentered particles?

All the best,

Madeline

[Muyuan Chen]

The first iteration structure, (spt_42/threed_01.hdf in this case), should be basically a shifted version of the structure which you used to re-extract particles. If there is significant difference, please double check if something went wrong. 

Refinement of protein-membrane interface tends to be difficult. The membrane contributes most of the signal in the particles, and the fluctuation of lipid affects the alignment. I don't think running refinement from scratch helps, but you can indeed explore the heterogeneity at the interface using e2spt_sgd_new. Run something like

e2spt_sgd_new.py spt_42/aliptcls3d_01.lst --refine --ncls 3 --classify

So it will take the initial orientation, do some local refinement and classification. Adding --skipali makes it only do classification. If you have a good mask to focus, in theory it can also be used by specifying --mask=xx --realspace --avgunmask, but I am less certain about the performance. 

More recently I use the GMM based pipeline to deal with the heterogeneity. It is implemented for SPT but is not very user friendly. Use e2gmm_spt_refine.py to refine orientations, and use e2gmm_spt_heter_refine.py to explore the heterogeneity. It seems that I also implemented a mask-free focused refinement for SPT (e2gmm_spt_rigidbody.py), but somehow did not write a wrapper to run it iteratively. I really don't know what am I thinking sometimes... 

Muyuan

To view this discussion visit https://groups.google.com/d/msgid/eman2/25ca2802-fd99-4778-899e-7b54d0873250n%40googlegroups.com.