Error with .lst file in e2spt_refine.py

[Ricardo Righetto]

Dear EMAN2 developers,

I tried to run the following command line:

e2spt_refine.py sets/tempmatch_emd3694_top30000.lst --reference=emd_3694_thr_box64-lp40.mrc --niter=5 --sym=d4 --mass=540.0 --goldstandard=40 --goldcontinue --pkeep=0.8 --maxtilt=90.0 --threads=40

And got the following error messages:

stdout:

setting up environment
e2proc3d.py emd_3694_thr_box64-lp40.mrc spt_00/model_input.hdf --process mask.soft:outer_radius=-1 --first 0 --last 0
e2spt_align.py sets/tempmatch_emd3694_top30000.lst spt_00/alignref.hdf --parallel thread:40 --path spt_00 --iter 1 --sym d4 --minres 200 --maxres 30.0  --goldcontinue
Using particle list:
sets/tempmatch_emd3694_top30000__even.lst
sets/tempmatch_emd3694_top30000__odd.lst
0 jobs on 40 CPUs
Error running task :  0
e2spt_average.py --parallel thread:40 --path spt_00 --sym d4 --keep 0.800 --iter 1 --skippostp
Cannot find a spt_00/particle_parms* file

stderr:

Traceback (most recent call last):
  File "/fs/home/righetto/eman2-sphire-sparx/bin/e2spt_refine.py", line 357, in <module>
    main()
  File "/fs/home/righetto/eman2-sphire-sparx/bin/e2spt_refine.py", line 262, in main
    data=EMData(combine)
  File "/fs/home/righetto/eman2-sphire-sparx/lib/python3.7/site-packages/EMAN2db.py", line 400, in db_emd_init
    self.__initc(*parms)
RuntimeError: FileAccessException at /home/eman2/miniconda3/conda-bld/eman2_1615224802990/work/libEM/io/hdfio2.cpp:464: error with 'spt_00/threed_01.hdf': 'cannot access file 'spt_00/threed_01.hdf'' caught

The highlighted part is particularly intriguing because this path does not exist in my machine.

In the command line above, sets/tempmatch_emd3694_top30000.lst is a file pointing to 30000 subtomogram particles as follows:

#LSX
# This file is in fast LST format. All lines after the next line have exactly the number of characters shown on the next line. This MUST be preserved if editing.
# 63
0 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
1 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
2 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
3 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
4 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
5 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf    
6 particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf

(...)

I managed to find a workaround by directly supplying the file particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf as input to e2spt_refine.py, which worked. I noticed it also works with an .lst file if this points to individual files containing one tomogram each, for example:

#LSX
# This file is in fast LST format. All lines after the next line have exactly the number of characters shown on the next line. This MUST be preserved if editing.
# 37
0 subtomos_all/L3Tomo3-00001.mrc    
0 subtomos_all/L3Tomo3-00002.mrc    
0 subtomos_all/L3Tomo3-00003.mrc    
0 subtomos_all/L3Tomo3-00004.mrc    
0 subtomos_all/L3Tomo3-00005.mrc    
0 subtomos_all/L3Tomo3-00006.mrc    
0 subtomos_all/L3Tomo3-00007.mrc

(...)

...but it doesn't work if the .lst file points to several subtomograms stored in the same file, as explained above. Is this a bug?

EMAN 2.91 final ( GITHUB: 2021-03-08 11:36 - commit: 81caed2 )
Your Python version is: 3.7.7

Thanks!

--
Ricardo Diogo Righetto

[Ludtke, Steven J.]

Hi Ricardo,

I suspect you are using your own naming convention here. The likely cause of the problem is the use of "__" (double underscore) in the particles3d filenames. Double underscore is reserved in EMAN2 for describing variants of the same file, if they are, for example, filtered in a different way or somesuch. If the __ was produced automatically by EMAN2, we'd need to know what you did to produce it...

--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine 

Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology                      (www.bcm.edu/biochem)

Academic Director, CryoEM Core                                        (cryoem.bcm.edu)
Co-Director CIBR Center                                    (www.bcm.edu/research/cibr)

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAB7-Rruc_te_jHZQh5vfcgbiBH50sT7D%3Db4%3Df3%3DeKisfi16bCg%40mail.gmail.com.

[Ricardo Righetto]

Hi Steve,

The double underscore was generated by EMAN2. My tilt series was called L3Tomo3_orig.st, hence the first part of the file name.

Then I ran template matching as follows:

e2spt_tempmatch.py tomograms/L3Tomo3_orig__bin4.hdf --reference=emd_3694_thr_box64-lp40.mrc --label=tempmatch_emd3694_top30000 --nptcl=30000 --dthr=-1.0 --vthr=2.0 --delta=30.0 --sym=d4 --rmedge --rmgold --boxsz=-1 --threads=16

Extracted particles:

e2spt_extract.py tomograms/L3Tomo3_orig__bin4.hdf --boxsz_unbin=-1 --label=tempmatch_emd3694_top30000 --threads=16 --maxtilt=100 --padtwod=2.0 --shrink=1.0 --tltkeep=1.0 --rmbeadthr=-1.0 --curves=-1 --curves_overlap=0.5 --compressbits=8

Built a set:

e2spt_buildsets.py particles3d/L3Tomo3_orig__tempmatch_emd3694_top30000.hdf --label=tempmatch_emd3694_top30000 --spliteo

The sets/tempmatch_emd3694_top30000.lst input for e2spt_refine.py (which did not work) is the output of e2spt_buildsets.py above. Everything was done through the e2projectmanager.py GUI, I just copied the corresponding command lines.

Thanks,

--
Ricardo Diogo Righetto

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/E853D75F-A025-42EA-BA78-71DC743A70FC%40bcm.edu.

[Ludtke, Steven J.]

Hmm, ok, have to take a closer look... that was just a knee jerk reaction...

did the split files get generated correctly? :

Using particle list:
sets/tempmatch_emd3694_top30000__even.lst
sets/tempmatch_emd3694_top30000__odd.lst

They should be created automatically, but the indication seems to be that it's seeing empty files as inputs.

--------------------------------------------------------------------------------------
Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine 

Charles C. Bell Jr., Professor of Structural Biology
Dept. of Biochemistry and Molecular Biology                      (www.bcm.edu/biochem)

Academic Director, CryoEM Core                                        (cryoem.bcm.edu)
Co-Director CIBR Center                                    (www.bcm.edu/research/cibr)

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CAB7-Rrtq8o7W67KAT5MK0%3D892cL%3Dd1RATn7gJcTgvak4Z%3D-mrA%40mail.gmail.com.

[Muyuan Chen]

I would just uncheck the --spliteo option and try it again. It is working a while ago but I haven't touched it since the last upgrade. There might be some incompatibility issue with the downstream workflow...

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/317A1AE0-424B-4F9A-9983-551A9D697865%40bcm.edu.

[MuyuanChen]

ok, I think I just fixed this. There were something wrong in the buildset step when spliteo is checked. Although the fix won’t really appear until the next release, and I imaging the metadata will be handled quite differently in the next release anyway. For now, it is easier to just not use that option.

[Ricardo Righetto]

Thanks Steve and Muyuan!

Indeed the even and odd .lst files were empty, looks like it's this bug with --spliteo. After deleting the empty even and odd .lst files, the input  sets/tempmatch_emd3694_top30000.lst worked!

Best wishes,

--
Ricardo Diogo Righetto

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/B5C1F840-AD9B-43B3-9715-51727FD80163%40gmail.com.