calculation of CCC between two maps

[Jianhao Cao]

Hi specialists,

I have two maps(maybe three or more) in mrc file, and I want to perform a global orientation search and to calculate the CCC among them.
Is there a script or command-line to do that?

Thank you in advances!

Regards,
Jianhao

[Paul Penczek]

sximgstat.py -h

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Ludtke]

Hi Jianhao,

e2proc3d.py <map1> <aligned_map1> --align rotate_translate_3d_tree --alignref <map2>

will perform an alignment of one map to another.

It isn't quite clear to me what value a cross correlation score between two maps has by itself? We don't have a stand alone program which just does a ccc and prints the number, though it's trivial to do. If you want to compute something like the FSC curve, THAT you can do with e2proc3d:

e2proc3d.py <map1> fsc.txt --calcfsc <map2>

Alternatively from Python:

from EMAN2 import *

img1=EMData("image1.mrc")

img2=EMData("image2.mrc")

img1a=img1.align("rotate_translate_3d_tree",img2)

print "ccc: ",img1a.cmp("ccc",img2)

There are several different alignment algorithms available, and several different ways of computing the similarity of two volumes other than a CCC:

e2help.py cmps -v 1

e2help.py aligners -v 1

On Sep 8, 2017, at 8:28 AM, Jianhao Cao <jian...@gmail.com> wrote:

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

[Jianhao Cao]

Thank you both for Paul and Steven.

My goal is to combine these similar classes after 3D Classification, but I don't know any better way to do that.

regards,

Jianhao

To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---

You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com

To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---

You received this message because you are subscribed to a topic in the Google Groups "EMAN2" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eman2/uEYFqzcX1-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eman2+unsubscribe@googlegroups.com.

[Steve Ludtke]

Are the 3-D maps in question subtomograms, subtomogram averages, or are they a large number of maps produced using single particle methods?

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

[C.J.]

They are production of single particles methods.

Best regards!

C.J.

[Steve Ludtke]

Please note that in the following, file format doesn't matter unless you plan to make a 3-D stackfile (one file containing multiple volumes) in which case you must convert to HDF. Output files will normally be HDF in any case.

Assuming the maps are all of a similar structure:

1) pick one map to serve as an alignment reference

2) You need to get all of your volumes (including the reference) into a 3-D stack. This can be a virtual stack (.lst file) or a HDF file containing all of the data. For example:

mkdir sets

e2proclst.py <image1> <image2> <...> --create sets/allparticles.lst

3) determine the optimal alignment of all of the particles to the reference:

e2spt_align.py sets/allparticles.lst <reference>  --saveali --threads <threads>

This will create a folder called spt_01. Each folder can also contain multiple "iterations". That is, you can align everything to a reference, then make a 3-D average, and realign everything to the new average. The second alignment will be a second iteration in the same spt_01 folder. You can control --path and --iter with most of these commands.

4) If you wish to create an average containing the "best" (most similar to the reference) particles:

e2spt_stat.py --gui 

This defaults to the last iteration in the last spt_XX folder. It will show you a histogram of the similarity of the individual particles to the reference. This assumes you have enough particles to make a histogram worthwhile. If you have only a few particles, you can look at the

spt_01/particle_parms_01.json

file (it is a text file). The 'score' values indicate the similarity of each particle to the reference. Note, SMALLER IS BETTER (this is ccc*-1).

5) You can then make an average using only the particles with a score lower than a threshold value:

e2spt_average.py --simthr <threshold>

6) If you'd like to classify the particles into different groups, you can run MSA on the aligned stack (spt_01/aliptcls.hdf) saved in step 2. 

e2msa.py spt_01/aliptcls.hdf spt_01/basis.hdf  --nbasis 8

e2basis.py project spt_01/basis.hdf spt_01/aliptcls.hdf spt_01/basisproj.hdf --mean1 --normproj

e2classifykmeans.py spt_01/basisproj.hdf --ncls <Nclasses> --average --normavg -C --original spt_01/aliptcls.hdf --fastseed -v 2

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

[C.J.]

Steve, thank you very much. This really is helpful!

Best regards!

Jianhao

[C.J.]

Hi Steve,

There is some problem when I ran e2spt_align.

$ e2spt_align.py sets/allparticles.lst 1.hdf --saveali --threads 95 --verbose 9
Traceback (most recent call last):
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 558, in open_db
    try : ret=JSDict(path)
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 629, in __init__
    self.sync()
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 680, in sync
    file_lock(jfile,readonly=False)
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 201, in file_lock
    else : fcntl.lockf(fileobj.fileno(),fcntl.LOCK_EX)
IOError: [Errno 38] Function not implemented
===========================
Traceback (most recent call last):
  File "/home/programs/EMAN2/bin/e2spt_align.py", line 121, in <module>
    main()
  File "/home/programs/EMAN2/bin/e2spt_align.py", line 84, in main
    angs=js_open_dict("{}/particle_parms_{:02d}.json".format(options.path,options.iter))
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 73, in js_open_dict
    return JSDict.open_db(url)
  File "/home/programs/EMAN2/lib/EMAN2jsondb.py", line 563, in open_db
    raise Exception,"Unable to open "+path
Exception: Unable to open spt_02/particle_parms_01.json

$ cat sets/allparticles.lst
#LSX
# This file is in fast LST format. All lines after the next line have exactly the number of characters shown on the next line. This MUST be preserved if editing.
# 20
0    1.hdf           
0    2.hdf           
0    3.hdf           
0    4.hdf           
0    5.hdf           
0    6.hdf           


$ e2version.py
EMAN 2.12 (CVS 2015/10/19 09:00:04)
Your EMAN2 is running on:  Linux-2.6.32-573.12.1.el6_lustre.x86_64-x86_64-with-centos-6.7-Final 2.6.32-573.12.1.el6_lustre.x86_64 x86_64
Your Python version is:  2.7.3

Is there something I have missed?

Regards,

Jianhao

[C.J.]

I try to update EMAN2. There are still some other problems.

$ e2version.py
EMAN 2.2 (GITHUB: 2017-09-20 00:00 - commit: 4aa69dd )

Your EMAN2 is running on: Linux-2.6.32-573.12.1.el6_lustre.x86_64-x86_64-with-centos-6.7-Final 2.6.32-573.12.1.el6_lustre.x86_64

Your Python version is: 2.7.13

$ e2proclst.py

Traceback (most recent call last):

  File "/home/jianhaoc/s/EMAN2/bin/e2proclst.py", line 311, in <module>
    main()
  File "/home/jianhaoc/s/EMAN2/bin/e2proclst.py", line 43, in main
    parser = EMArgumentParser(usage=usage,version=EMANVERSION)
NameError: global name 'EMANVERSION' is not defined

So, I produced a list with old version. And I tried to run e2spt_align.py with new version.

$ e2spt_align.py sets/allparticles.lst 1.hdf --saveali --threads 95 --verbose 9
Traceback (most recent call last):

  File "/home/jianhaoc/s/EMAN2/bin/e2spt_align.py", line 150, in <module>
    main()
  File "/home/jianhaoc/s/EMAN2/bin/e2spt_align.py", line 59, in main
    parser = EMArgumentParser(usage=usage,version=EMANVERSION)
NameError: global name 'EMANVERSION' is not defined

While other programmes seem to be fine.

$ e2proc3d.py
usage: e2proc3d.py [options] <inputfile> <outputfile>
    Generic 3-D image processing and file format conversion program.
    All EMAN2 recognized file formats accepted (see Wiki for list).

    To create a new image, rather than reading from a file, specify ':<nx>:<ny>:<nz>:<value>'
    as an input filename.

    Examples:

    Convert MRC format to HDF format:
    e2proc3d.py test.mrc test.hdf

    Apply a 10 A low-pass filter to a volume and write output to a new file:
    e2proc3d.py threed.hdf threed.filt.hdf --process=filter.lowpass.gauss:cutoff_freq=0.1

    Extract a reconstruction from a refinement directory as an HDF file usable with Chimera:
    e2proc3d.py bdb:refine_02#threed_filt_04 map_02_04.hdf

    Create a new 64x64x64 volume, initialize it as 1, then apply a hard spherical mask to 0:
    e2proc3d.py :64:64:64:1 myvol.hdf --process mask.sharp:outer_radius=25

    'e2help.py processors -v 2' for a detailed list of available procesors


Please run 'e2proc3d.py -h' for detailed options

Regards,

Jianhao

--

Best regards!

C.J.

[Steve Ludtke]

Hi CJ, 

from some indicators in your message, it looks like you installed the CentOS7 version of EMAN2.2 on a machine running CentOS6. Is that a possibility?

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

--- 
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

[C.J.]

Hi Steve,
I installed a new version of EMAN2.2 for CentOS6, but there is still some little problem.
Am I still miss something?

$ e2version.py
EMAN 2.2 (GITHUB: 2017-09-20 00:31 - commit: None )

Your EMAN2 is running on: Linux-2.6.32-573.12.1.el6_lustre.x86_64-x86_64-with-centos-6.7-Final 2.6.32-573.12.1.el6_lustre.x86_64
Your Python version is: 2.7.13

$ ls -l
total 359580
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 1.hdf
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 2.hdf
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 3.hdf
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 4.hdf
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 5.hdf
-rw-rw-r-- 1 jianhaoc jianhaoc 28324496 Sep 20 07:18 6.hdf
drwxrwxr-x 2 jianhaoc jianhaoc     4096 Sep 22 01:46 sets

$ cat sets/allparticles.lst
#LSX
# This file is in fast LST format. All lines after the next line have exactly the number of characters shown on the next line. This MUST be preserved if editing.
# 20
0    1.hdf           
0    1.hdf           
0    2.hdf           
0    3.hdf           
0    4.hdf           
0    5.hdf           
0    6.hdf           

$ e2spt_align.py sets/allparticles.lst 1.hdf --saveali --threads 3

Traceback (most recent call last):

  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 566, in open_db
    try : ret=JSDict(path)
  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 637, in __init__
    self.sync()
  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 688, in sync
    file_lock(jfile,readonly=False)
  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 209, in file_lock

    else : fcntl.lockf(fileobj.fileno(),fcntl.LOCK_EX)
IOError: [Errno 38] Function not implemented
===========================
Traceback (most recent call last):

  File "/home/jianhaoc/s/EMAN2/bin/e2spt_align.py", line 150, in <module>
    main()

  File "/home/jianhaoc/s/EMAN2/bin/e2spt_align.py", line 113, in main

    angs=js_open_dict("{}/particle_parms_{:02d}.json".format(options.path,options.iter))

  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 72, in js_open_dict
    return JSDict.open_db(url)
  File "/home/jianhaoc/s/EMAN2/lib/python2.7/site-packages/EMAN2jsondb.py", line 571, in open_db

    raise Exception,"Unable to open "+path

Exception: Unable to open spt_01/particle_parms_01.json

Regards,
Jianhao

To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

--- 
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards!

C.J.

[Steve Ludtke]

Hi CJ,

was this in the earlier message as well?  If so I missed it.  

This error indicates that you are trying to run a job on a filesystem which doesn't support file locking. This can lead to all sorts of problems with file/image corruption and other bad things, so we intentionally do not support it. Usually this happens when someone is trying to run using a shared filesystem with poor Linux compatibility, or using an older NFS protocol without configuring the locking server properly.  In any case, running image processing jobs on remote filesystems is generally a very bad idea except on clusters, which have dedicated high-speed networks for such.

[C.J.]

Hi Steve,
Thank you for your quick reply. Maybe I should try on another workstation.

Regards,

Jianhao

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards!

C.J.

[Steve Ludtke]

Hi CJ,

are your files stored on a local physical hard drive or RAID array, or are they stored remotely on another computer?  I'm assuming you aren't doing something like running a refinement on a portable external hard drive. The key is to have your project on an internal hard drive/RAID array formatted with a native Linux filesystem, like EXT3/4, XFS, ZFS, etc.  

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[C.J.]

Hi Steve,
Thank you. I did success when change a computer. Maybe the reason is like what you said.

One more question, are the scores in result of e2spt_align calculated by CCC method? They are different from the values calculated by CCC method.

Regards,

Jianhao

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best regards!

C.J.

[Steve Ludtke]

Hi Jianhao,

I remember responding to this, but it isn't marked as such... The scores are missing-wedge compensated, so you can't relate them directly to CCC values. The only guarantees about similarity metrics is that smaller means more similar. You will only find a few situations where normalized CCC's are used in EMAN2 by default.

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

[Jianhao Cao]

Thank you again!

Regards,

Jianhao

You received this message because you are subscribed to a topic in the Google Groups "EMAN2" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eman2/uEYFqzcX1-c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eman2+unsubscribe@googlegroups.com.