Subtilt refinement error in the latest version

[digvij...@gmail.com]

Hi 

I noticed that the subtilt refinement in the bleeding-edge latest version is not working. It says it can't find files from the specified spt run. Error attached. 

I tried running the subtilt-refinement from the same spt_folder using an older version of EMAN2 and it was fine. 

Cheers,

Digvijay

[Muyuan Chen]

Yes I just noticed this literally a minute ago... seems that it is affected by a folder name routine change lately. Still confirming this with Steve and hopefully fill get this fixed soon.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/172ce2d2-5f91-4b6c-be21-ba8b7460a290n%40googlegroups.com.

<subtilt_error_in_the newest version.png>

[Steve Ludtke]

You are correct, Muyuan, there was a mistake in that routine. The intent was for it to use :02d formatting still, so 1->01, 10->10, 100->100, so at least it wouldn't fail when projects got above 99. There are at least 3 different places in the EMAN code where this function is implemented and they all act in slightly different ways :^(

very sorry this didn't get taken care of properly before the release. It only impacts a handful of programs though right now. Still, worried that we might need to push out a 2.91 patch sooner rather than later. I pushed a quick fix for the 2-digit problem to master, so the snapshots will include the fix, at least  :^(

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/5DEF00A0-2599-482C-BE1B-B342D5026992%40gmail.com.

[digvij...@gmail.com]

No worries at all,  please take your time. I have got some slightly older versions of EMAN2 as well so the subtilt is running fine on those. I am good for now. 

This may be a good time to ask a few broader questions about the subtilt refinement:

{1} Any reason why the evaluation(e2_eval) is restricted to only spt and does not work for subtilt? EMAN2 produces clean fsc and parameters files, so those can be plotted externally easily, using other programs. But it would be handy to be able to plot .e.g, FSC curves for subtlt just like spt.

{2} I guess little value in doing re-extraction from subtlt run, and that is why the re-extraction can't be done from the subtlt and only spt runs? 
{3}When choosing resume in subtilt run, does it resume from the latest subtilt run and using its latest iteration? 
{4} What do you think about starting a spt run from the .json file of the subtilt run?

Thanks,
Digvijay

[Muyuan Chen]

I have been reworking the subtilt refinement routine lately, so the answer to your questions keeps changing. Mostly to combine the spt and subtilt routine and include local ice morphing constraints. A working version should be available sometime soon. Right now a preliminary implementation can be called by adding —scipy option to e2spt_refine on git version.Hopefully it will address (1), (2) and (4). 

(3) you should be able to specify path and iter to resume. 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/1bfe5638-6f12-4693-87f1-1f43c9a8ae57n%40googlegroups.com.

[digvij...@gmail.com]

Thank you!

[digvij...@gmail.com]

Another question about the subtilt refinement: 

My subtilt refinements are running significantly faster than spt refinement, which is quite at odds with the processing time listed in your e2tomo tutorial, where the subtlt is to take much longer than spt. 

• Here's my dataset:
• Particle size 84 by 84 by 84. 
• ~5000 particles. Althought I found a similar behaviour with ~2000 particle dataset as well. 
• and the computer specs are attached. 

A faster subtlt is definitely welcome, but I was worried that the faster subtlt (over spt) may mean some underlying issue with the particles or the refinement process. Have you improved the subtlt algorithm lately to improve its speed?

Thanks 

[Muyuan Chen]

Yes in the current version subtilt refinement is faster. I haven’t updated the timing in the tutorial yet. Since I am reworking this part I am planning to update later.

The speed difference is caused by a few bugs that got fixed, and that the rotation refinement is disabled in the current version caused by some other bug introduced somehow during the python 2 to 3 transition.

The subtilt rotational refinement was not too helpful in the previous version anyway, so disabling it probably does not hurt much. I have decided to rework the routine instead of digging into the memory leaking bug, and it seems to do well so far. Hopefully will push an update soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/aaaef20a-18c3-417a-bdcc-f892abaa2d3dn%40googlegroups.com.

<ComputerSpecs.png>

[digvij...@gmail.com]

Awesome. Good to know. Thank you!

[Fabian Schneider]

Hi all,

thanks a lot for the update, really adding even more value to Eman2!

I was stumbling over the not working subtilt refinements as well and did a fresh installation of the daily build today.

It fixes the error of not finding the files from the chosen spt (creating folders like spt_9_00 from the original spt_9), but somehow I am now stuck with another error I've never seen before:

Building aligned 2D particle set from spt alignment...
e2proc3d.py spt_9_00/threed_00_even.hdf alignref.hdf --multfile None
Traceback (most recent call last):
  File "/home/fschneid/eman2-sphire-sparx/bin/e2proc3d.py", line 890, in <module>
    main()
  File "/home/fschneid/eman2-sphire-sparx/bin/e2proc3d.py", line 655, in main
    mf=EMData(options.multfile[index_d[option1]],0)
  File "/home/fschneid/eman2-sphire-sparx/lib/python3.7/site-packages/EMAN2db.py", line 400, in db_emd_init
    self.__initc(*parms)
RuntimeError: FileAccessException at /home/eman2/miniconda3/conda-bld/eman2_1613794828164/work/libEM/emutil.cpp:420: error with 'None': 'cannot access file 'None'' caught

I am not sure, whether this is something related to the new version and its subtilt / e2proc3d or whether I am somehow facing another problem.

Version info (as I said I've downloaded the continuous daily build)

EMAN 2.91 ( GITHUB: 2021-02-19 22:23 - commit: dfb041b )
Your Python version is: 3.7.9

Thanks already for any help!

Best,

Fabian

[MuyuanChen]

I think this is because the maskalign option that was newly added to e2spt_refine that was not well taken by the subtilt refinement protocol… Try giving the subtilt refinement an extra option of “—maskalign=none” and hopefully it will get through.

Steve, probably setting the default of maskalign in e2spt_refine to “” instead of None will make it safer? Will it break other things?

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/66efaf38-06b1-4b32-b699-e2e1cce454b5n%40googlegroups.com.

[Fabian Schneider]

thanks a lot

I've changed the initial maskalign from auto to none and so far it is continuing!

I'll update tomorrow whether there was any further issue!

[Steve Ludtke]

I’ll deal with it before the buggix release tomorrow. Good timing!

Sent from my iPhone

On Mar 1, 2021, at 4:02 PM, Fabian Schneider <schn...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/eaad6645-4d3d-46b4-a428-18c7b801df74n%40googlegroups.com.

[Fabian Schneider]

works perfectly fine, the only catch was that when I changed from "auto" to "none" in the command window of the graphic user interface eman dropped the --maskaling "none" after starting a subtilt run.

When I started a second subtilt run (supposing the same settings) I got an error as it was not specified anymore, but by adding the --maskaling "none" manually it worked again.

I guess it is just eman ignoring parameters that are set to none, am I right?