I create an artifact map of teamtomo.org EMPIAR-10164 dataset in 1.6Apx

[yuqi zhang]

 I did the teamtomo.org EMPIAR-10164. After relion_reconstruct_mpi 1.6Apx refinement, the map was only noise. I used relion 3.0.6. The missing wedge affected the orientation distribution.

command:   

mpirun -np 40 `which relion_reconstruct_mpi` --i subtomograms_1.6Apx.star --o reconstruction_1.6Apx.mrc --3d_rot --ctf

I desire that someone can help me check the whole workflow about EMPIAR-10164 teamtomo.org. I didn't know which step went wrong. This tutorial has confused me for several months. Very very thank you.

The warp(version 1.0.9) preprocessing settings:

And I deselected 2 micrographs as the toturial said.

To generate the .st .rawtlt

press "create stack for imod"

dautoalign2warp in dynamo (I use the IMOD version : 4.12.6)

dautoalign4warp('./imod', 1.35, 10, 85.3, './dynamo_alignments')

To import the aligned tilt series

press "import"

3dCTF & handness check

I knew that the handness was very important. So I pressed the process only this item ctf and check the tilt handness one by one. They are all positive value and I never invert them.

[Amédée des Georges]

After relion_reconstruct, you need to low-pass filter the maps to their measured resoultion. Otherwise, they are just balls of noise like this. 

On Jun 18, 2021, at 11:24 AM, 'yuqi zhang' via Warp <war...@googlegroups.com> wrote:

 I did the teamtomo.org EMPIAR-10164. After relion_reconstruct_mpi 1.6Apx refinement, the map was only noise. I used relion 3.0.6. The missing wedge affected the orientation distribution.

command:   

mpirun -np 40 `which relion_reconstruct_mpi` --i subtomograms_1.6Apx.star --o reconstruction_1.6Apx.mrc --3d_rot --ctf

<reconstruct_1.6A_mrc_view1.jpg><reconstruct_1.6A_mrc_view2.jpg>

I desire that someone can help me check the whole workflow about EMPIAR-10164 teamtomo.org. I didn't know which step went wrong. This tutorial has confused me for several months. Very very thank you.

The warp(version 1.0.9) preprocessing settings:

And I deselected 2 micrographs as the toturial said.

<1624028532.jpg>

To generate the .st .rawtlt

press "create stack for imod"

<1624028804(1).jpg>

dautoalign2warp in dynamo (I use the IMOD version : 4.12.6)

dautoalign4warp('./imod', 1.35, 10, 85.3, './dynamo_alignments')

To import the aligned tilt series

press "import"

<1624029277(1).jpg>

3dCTF & handness check

I knew that the handness was very important. So I pressed the process only this item ctf and check the tilt handness one by one. They are all positive value and I never invert them.

<1624029411(1).jpg>

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/36405006-cdcf-45f6-bd3c-bbe0de37756dn%40googlegroups.com.
<reconstruct_1.6A_mrc_view2.jpg><1624028532.jpg><1624028804(1).jpg><1624029277(1).jpg><1624029411(1).jpg><reconstruct_1.6A_mrc_view1.jpg>

[yuqi zhang]

Then, I create the full tomograms in warp.

The dimension was 8000*8000*6000. It was same setting as the last figure(3D ctf)

Pick particles and extract in dynamo.

In matlab R2020a in dynamo folder:

warp2catalogue('../frames/reconstruction', 10); dcm warp_catalogue

I only chose the intact VLP and apply the dipole2vesicle script.

TS_01: 9             TS_03:  6               TS_43:  5               TS_45:  5              TS_54:   2

And then, I extract two VLP in TS_01 and TS_03 without dBox mode. 

The folder inimodelData.Data contains 2649 particles.

I used the dcp in dynamo and the result was like this:(I think it was good.)

Symmetry expansion:

align_symmetry_axis

And then, I continued to align all particles.

The folder findallData.Data contains 31561 particles  without dBox mode. 

I used the dcp in dynamo and the result was like this:

[yuqi zhang]

Thank you! And I am so sorry that I am still updating my statement.

[yuqi zhang]

And then I use some script given in the tutorial to clean trash particles.

view_particles('refined_table_ref_001_ite_0001.tbl')
remove_duplicates

check_radial_distribution

subset_table_based_on_neighbours

dynamo2warp -i result_10Apx_nodup_neighbourcleaning.tbl -tm ../../../../findallData.Data/indices_column20.doc -o result_10Apx_nodup_neighbourcleaning_data.star

And then, I extract particles in warp.

To avoid the warp not to mismatch the tomostar files, I copied the file result_10Apx_nodup_neighbourcleaning_data.star to the frames/ folder. And then extract:

Then I reconstruct a map with random 500 particles

relion_reconstruct --i random_subset_5Apx.star --o random_subset_5Apx.mrc --3d_rot --ctf --sym C6

I display 2 different views. I think the middle of hexamal lattice has a artifact density.

mpirun -np 5 `which relion_refine_mpi` --o Refine3D/job002/run --auto_refine --split_random_halves --i subtomograms_5Apx.star --ref random_subset_5Apx.mrc --firstiter_cc --ini_high 30 --dont_combine_weights_via_disc --pool 30 --pad 2  --ctf --ctf_corrected_ref --particle_diameter 320 --flatten_solvent --zero_mask --oversampling 1 --healpix_order 3 --auto_local_healpix_order 3 --offset_range 2 --offset_step 2 --sym C6 --low_resol_join_halves 40 --norm --scale  --j 7 --gpu ""

I think the map became more and more strange.

Maybe the symmetry expansion leads to the artifact of the middle density of hexamal lattice

[yuqi zhang]

And then, I extracted particles with run_it027_data.star (the last iteration)

btw, I was afraid that the 3DCTF .xml file was changed (because the revised date time was different from that before last 5Apx extract).

So I copy the 3DCTF .xml as backup files. And before 1.6Apx extract, I overlapped the files with the backup files.

And then, you can see the map at the top of this topic with relion_reconstruct_mpi.

All I wanted to report was in the end.

[yuqi zhang]

PLZ PLZ PLZ, does anyone can help me check the whole process? This tutorial has confused me for several month. Maybe I was a little stupid. Hahaha.

Thanks for all of you.

Relion version : 3.0.6.

Warp version : 1.0.9

IMOD version : 4.12.6

Matlab version : R2020a

dynamo version 1.1.509

[Lorenzo Gaifas]

Hey Yuqi,

You seem to be doing great until the initial model generation. Look at that breatuful lattice!

Looking at the results of the first alignment on the whole dataset, I suspect you might be skipping an important step right after the initial model generation: centering the symmetry axis (https://teamtomo.org/walkthroughs/EMPIAR-10164/ini-model.html#aligning-and-centering-the-6-fold-symmetry-axis).

When you apply a symmetry to your alignment, the axis is at the center of the box, which does not necessarily align with the real center of symmetry of your data. This is why you first need to shift your initial model so that the center of symmetry (the hole in the middle of the hexamer) is in the center of the box. At the end, your model should look like this:

Rather than like this:

Let me know if this helps!

Best,

Lorenzo

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/8d88c8cf-b18d-40bf-9d30-d8a6d284d803n%40googlegroups.com.

[yuqi zhang]

You said that I should center the symmetry axis after alignment with whole dataset which has 30,000 particles? or I should just adjust the symmetry axis after the test dataset about 2,500 particles?

I guessed what you meaned was after the whole dataset. I will try it now. Thank you very much!

[yuqi zhang]

So, I should centering the axis again, and then,  remove trash particles and go on, right?

[Lorenzo Gaifas]

Hi Yuqi,

You should do this BEFORE moving on to the whole dataset, to save time. (You can see that the step I linked in the previous email is a the bottom of the `initial model generation` section). Otherwise, the positions you get from the whole-dataset alignment would be shifted, and you would have to redo it anyways.

After you have the centered initial model, you can proceed as normal with the find-particles section (and only after that you should remove the trash).

Best,

Lorenzo

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/1a5c09bd-876c-4d8f-adaf-5ffe235e1ec4n%40googlegroups.com.

[Alister Burt]

Hi Yuqi,

Centering the symmetry axis only needs to be done on the initial model itself - not the particles. 

The general idea is this:

- use a subset of the data to generate an initial model (using a small subset allows us to move quickly)

- see that this model has clear 6-fold symmetry, we recenter it on the 6-fold axis so we can take advantage of that symmetry during refinement

- we then align all of our particles (oversampled the surfaces of vesicles) to the recentered initial model, this should allow us to find the ’true positions’ of the C6 axes in the lattices on the VLPs (we should see regular lattices when we look at the particle positions)

- finally, decide on some geometrical constraint (each particle should have a certain number of neighbours at the expected inter-particle distance in the lattice). Leverage this to decide which particles are good or bad

As Lorenzo said, it looks like the recentering the 6-fold axis of the reference didn’t work for you, you might have to modify our script slightly to get this to work in your hands (your initial model is probably just a bit shifted relative to ours). 

This means that the reference you used for alignment of the whole dataset was aligning to an artefact (you applied c6 symmetry on a map where the 6-fold axis was not correctly centered) -> your particle positions after alignment are incorrect -> your cleaning based on geometrical constraints is wrong -> you get bad results

It’s super important to have a clear idea of where your particles should be/are centered and what you’re trying to do at every step, checking for internal consistency if you can. In your case, I imagine you could have seen that you didn’t have the expected regular lattices after aligning your particles to your badly recentered initial model.

Good luck, looks like you’re almost there!

Cheers,

Alister

[yuqi zhang]

I am so sorry that I just show the same screenshot in the "initial model" and "whole dataset map".

I am so sorry.

In fact, I have done the symmetry axis for C6

After symmetry centering, the initial model was like this:

And the whole dataset alignment result was this:

I have done the symmetry alignment before. I am so sorry to mislead you.

And can you help me find out other mistakes? Thanks a lot!

[Alister Burt]

As for what you concretely need to do now

- correctly align the 6-fold axis of your reference and symmetrise -> c6 initial model

- align all particles from oversampled vesicles to this c6 initial model

- clean particles based on geometrical constraint

- reconstruct particles in warp, refine in RELION

Have fun!

Alister

[yuqi zhang]

Maybe the picture was broken.

The map after adjusting the symmetry:

The aligned map with whole dataset:

[Alister Burt]

Hi Yuqi,

The result below shows that you aligned to a reference which was not centered on the 6-fold axis - this is where your problems come from, same advice from my previous message applies :)

Cheers,

Alister

[yuqi zhang]

When I run the script "align_symmetry_axis". There was an error information:

So I do the "%" to this command line. And in the left column, there was the result in the folder "inimodel/results/ite_0008averages/".

I think the "%" may be OK. Anyway, you can tell me how to debug for this if you like.

[yuqi zhang]

Besides only run the script "align_symmetry_axis", I didn't know any other thing I should do.

[yuqi zhang]

To display the results, I change the command line like this in highlight

And then, the 4 dview map were there:

Are these results right for whole dataset alignment?

[Lorenzo Gaifas]

Hi Yuqi,

The error you see is a bug in the script, but should not a problem for you (the thing that's failing is just for visualization). We'll have to fix that though!

(EDIT: I just saw you spotted it as well!)

If indeed you aligned the symmetry using the script and proceeded with the aligned initial model, then the problem must be later down the line.

When maps are very noisy, it's often better to look at them in slices, rather than isosurfaces.

How does your `random_subset_5Apx.mrc` look like if you open it with `dview('random_subset_5Apx.mrc')` in matlab?

It should look similar to the initial model pictures we were sharing earlier. If it does not, you then we know the problem is somewhere in between. If it does, we can check some other things later on.

Anyways, did you do as suggested by Amédée (lowpass filter the map before looking at it as isosurface)?

Best,

Lorenzo

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/08980919-883d-47e7-948e-d40cc3c6da24n%40googlegroups.com.

[yuqi zhang]

  The random_subset_5Apx.mrc was noiser.

One potential problem was the relion version.

In the teamtomo.org, I saw the relion background color was purple rather than orange. So it must be relion 3.1 or later.

I used the relion 3.0.6 rather than 3.1.1.

The generated star file in WARP was an old version.

When I used this star file in relion 3.1.1, it report an error:

I was so sorry to ask a relion question in warp discussion group.

[yuqi zhang]

The picture was broken. I updated them.

[Lorenzo Gaifas]

The relion version should not be a problem here.

I think that the map you showed above probably comes from a wrongly aligned set of particles. For reference, here's the same image from my processing:

Differently from your map, this has well-defined cylindrical holes going through several slices where there's very little density (=much darker).

I think that the patterns you see in your data are simply caused by imposing symmetry on noisy data. I would do as Alister suggested, and go back a couple of steps and re-do the find-particles section to make sure that you are indeed using the initial model with symmetry axis aligned to the center.

Best,

Lorenzo

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/a916a0d0-80eb-4385-9dad-b834da4322den%40googlegroups.com.

[yuqi zhang]

Thank you very much! Was the slice I show in the symmetry_axis_align right? 

The map was in the last 6th dialogue.

your map was so good. 

I am re-running the findparticles dcp job again now. Wait some time and I will show the result.

btw, can I still use the relion 3.0.6? Maybe the star file should be revised to use relion 3.1. Can you teach me how to handle it? Thanks a lot. 

[yuqi zhang]

I have another suggestion. Can you send me a tarball of all the results you do. And I can compare the results detail to mine.

Maybe it was too large. How about the tarball without micrographs and the folder "subtomo/"?

The email:

8245...@qq.com /        zhan...@westlake.edu.cn      /   31601...@zju.edu.cn

Thank you very much!

[Lorenzo Gaifas]

Yes, the map after the symmetry alignment seemed good. Make sure you use that (average_aligned_along_x_c6) and at each step double check if you're using the right files. It's easy to mix them up!

Unfortunately I don't have quick access to the whole data in the shape that the tutorial presents it (a lot of trial and error on this side :P). I'm happy to help again if things don't improve, though, so let me know how the new refinement goes.

As for relion versions: relion 3.0 will work just fine for now, since Warp 1.0.9 does not yet accept the relion 3.1 format. I suggest you keep working with 3.0 for this dataset to avoid complications. If you want to upgrade anyways, you will sometimes have to downgrade star files so that Warp can handle them. For that, you can use `relion_star_downgrade` from `dynamo2m` as described for example here: https://teamtomo.org/walkthroughs/EMPIAR-10164/relion.html#subtomogram-reconstruction-at-1-6a-px .

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/6c2b9752-fff1-46f0-a0af-9c9381003094n%40googlegroups.com.

[yuqi zhang]

The whole dataset alignment like this: 

If it is no problem, I will remove trash particles.

[yuqi zhang]

And then, I went into the folder "findpartilces/results/ite_0001/averages/". I ran the 4 command line in the tutorial view & remove_dup & check & subset. Right?

And then, I run the dynamo2m script.

The star file generated was this:

Note that the _rlnMicrographName column:  the path ......./frames/reconstruction/TS_54.mrc.tomostar

I think the tomostar files were in the frames/ folder rather than the frames/reconstruction/ folder.

Is that OK?

I didn't revise anything to extract subtomograms.

In the tutorial, The particle diameter was 160A. However, I think, in the next relion refinement job, you mean it is 320A. Right?

[yuqi zhang]

I used relion_reconstruct to generate the ~500 particles map, like this:

[yuqi zhang]

In the tutorial, the reference of whole dataset alignment was ite_0004 rather than ite_0008. Maybe it was a little error.

[Lorenzo Gaifas]

Good catch, that's a typo, it should be ite_0008.

The star file should be fine as it is.

I'm not sure about the map's difference... Take a look at the cleaned dataset next to the non-cleaned dataset. Do they differ? Does it look like you removed the bad stuff?

Unfortunately, I don't have time to take a look at this today. If you need more help, I can take a look again tomorrow.

Best,

Lorenzo

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/d7b3262c-7796-4fd4-87c0-673a2e28138bn%40googlegroups.com.

[yuqi zhang]

I have another question when I re-read the whole dialogue

In Lorenzo's statement(you can find it above):

Yes, the map after the symmetry alignment seemed good. Make sure you use that (average_aligned_along_x_c6) and at each step double check if you're using the right files. It's easy to mix them up!

Unfortunately I don't have quick access to the whole data in the shape that the tutorial presents it (a lot of trial and error on this side :P). I'm happy to help again if things don't improve, though, so let me know how the new refinement goes.

As for relion versions: relion 3.0 will work just fine for now, since Warp 1.0.9 does not yet accept the relion 3.1 format. I suggest you keep working with 3.0 for this dataset to avoid complications. If you want to upgrade anyways, you will sometimes have to downgrade star files so that Warp can handle them. For that, you can use `relion_star_downgrade` from `dynamo2m` as described for example here: https://teamtomo.org/walkthroughs/EMPIAR-10164/relion.html#subtomogram-reconstruction-at-1-6a-px .

The symmetry axis of average_aligned_along_x_c6 map was x？ Or it was z?

Maybe I was too subtle. However, I prefer to solve this problem....

[yuqi zhang]

Can I ask this question again?

The whole datasets alignment in dynamo was good ? or it was not recentered?

[Lorenzo Gaifas]

Hi Yuqi,

That must be another typo, sorry. The symmetry axis is indeed z, and you can tell because your symmetrized maps look as expected.

Cheers,

Lorenzo

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/64bf5cce-858a-41ef-93f9-3d3372fd1dbfn%40googlegroups.com.