Simulate Gaussian noise micrograph modulated by given CTF parameters

[je...@crystal.harvard.edu]

Hi Steve,

Is there a reasonably easy way to simulate a micrograph with noise that is modulated by the CTF parameters obtained from an original real micrograph (I have the original micrograph and extracted particles with CTF entry in the HDF header, plus the corresponding *.json file; I also can make images with noise)?

For EMAN1 I see on the wiki:

"To fairly realistically simulate a micrograph use 'applyctf flipphase addnoise'. This assumes that the images are already filtered to an appropriate resolution. Note that the scale of added noise may be off and need adjustment."

This is bascially what I would like to do, but all my files are in EMAN2.

Any hints on would be appreciated,

Simon

[Steven Ludtke]

This is something that should really be easy to do, but nobody has gotten around to doing it properly ... I think.  There is a processor called filter.CTF_ which may do something like what you're after, except for the noise.  

e2help.py processor CTF -v 2

For the noise, you could make flatband Gaussian noise then filter it using any profile you like.

e2help.py processor noise

This should really get added to a simple command  :^(

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[je...@crystal.harvard.edu]

OK, I run this in python:

#!/programs/x86_64-linux/eman2/2.11/extlib/bin/python

from EMAN2 import *

n = EMData(256,256,True,1)

n.process_inplace("testimage.noise.gauss",{"mean":0.0, "sigma":1.0})

n.write_image('noise.hdf', 0)

n.process_inplace("filter.CTF_",{

"apix":0.62,

"b_factor":200.0,

"cs":2.0,

"cutoff_abs":0.5,

"defocus":20000.0,

"npow":1.0,

"sigma":0.2,

"sign":1.0,

"voltage":200.0,

"wgh":0.07

})

n.write_image('ctf.hdf', 0)

The image statistics of noise.hdf are as excepted:

maximum: 3.844

mean: 0.005

minimum: -4.346

The image statistics of ctf.hdf are strange (max < min) and I don't see anything if I open the image:

maximum: -3.403e+38

mean: 0.0

minimum: 0.0

I'm not sure what's going on here.

Simon 

[Steven Ludtke]

I'm afraid you'll have to ask Pawel about this one. That processor was added as part of SPARX, and doesn't follow most EMAN conventions. I checked, and the code uses Pawel's EMFourierFilterInPlace function, so I'm not sure about its proper use...  That is, the way you are trying to call it is correct, but I don't know if there is something funny with your parameters.

[Paul Penczek]

Indeed, it is one of the never resolved issues.
I will provide explanation later...
Regards,

Pawel

[Paul Penczek]

Hi,

there are two ways to use multiplication by the CTF: easy and hard.

Easy:

from sparx import *

m = model_gauss_noise(1.0,256,256)

c = generate_ctf([defocus, cs, voltage, apix, bfactor, ampcont, astigmatism_amplitude, astigmatism_angle])

Units:  [ microns, mm, kV, Angstroms, A^2, microns, radians]

f = filt_ctf(m,c)

Hard:

you can parse body of film_ctf, it is written in python and refers only to EMAN2 functions:

def filt_ctf(img, ctf, dopad=True, sign=1, binary = 0):

"""

Name

filt_ctf - apply Contrast Transfer Function (CTF) to an image in Fourier space

Input

image: input image, it can be real or complex

ctf: an CTF object, please see CTF_info for description.

pad: apply padding with zeroes in real space to twice the size before CTF application (Default is True, if set to False, no padding, if input image is Fourier, the flag has no effect).

sign: sign of the CTF. If cryo data had inverted contrast, i.e., images were multiplied by -1 and particle projections appear bright on dark background, it has to be set to -1). (Default is 1).

binary: phase flipping if set to 1 (default is 0).

Output

image multiplied in Fourier space by the CTF, the output image has the same format as the input image.

"""

from EMAN2 import Processor

assert img.get_ysize() > 1

dict        = ctf.to_dict()

dz          = dict["defocus"]

cs          = dict["cs"]

voltage     = dict["voltage"]

pixel_size  = dict["apix"]

b_factor    = dict["bfactor"]

ampcont     = dict["ampcont"]

dza         = dict["dfdiff"]

azz         = dict["dfang"]

if dopad and not img.is_complex():  ip = 1

else:                               ip = 0

params = {"filter_type": Processor.fourier_filter_types.CTF_,

"defocus" : dz,

"Cs": cs,

"voltage": voltage,

"Pixel_size": pixel_size,

"B_factor": b_factor,

"amp_contrast": ampcont,

"dopad": ip,

"binary": binary,

"sign": sign,

"dza": dza,

"azz":azz}

tmp = Processor.EMFourierFilter(img, params)

tmp.set_attr_dict({"ctf":ctf})

return tmp

Depending on your taste, you may also consider easy what I consider hard.

Regards,

Pawel.

---

From: Steven Ludtke <slud...@gmail.com>
To: em...@googlegroups.com
Cc: Pawel A. Penczek <Pawel.A...@uth.tmc.edu>
Sent: Sunday, September 13, 2015 3:05 PM
Subject: Re: [EMAN2] Simulate Gaussian noise micrograph modulated by given CTF parameters

[Steven Ludtke]

Huh, after digging a bit, I remembered that we had done something about easy CTF simulation some time ago, specifically:

https://groups.google.com/forum/?hl=en#!searchin/eman2/apply$20ctf/eman2/-TJKFA9WDXU/KnVcWXebAMMJ

Interestingly, however, this method seems no longer to exist in the list of active processors. I will need to dig a bit more to see what happened  :^/

[Paul Penczek]

If the issue is how to generate a micrograph that would contain CTF-modified projections of a structure with two kinds of noise (with and without

CTF, see my review on image formation for detail), it is done with:

sxprocess.py input_structure.hdf data mic --generate_projections --format="bdb":apix=5.2:CTF=True:boxsize=64

A working example is included in sparx run_through_example

RunThroughExample - SPARX Wiki

		RunThroughExample - SPARX Wiki Run-through example using simulated ribosome data 08/28/2015 Download file demo_0828_2015.tar.gz and unzip it using the following command: tar -xvf demo_0828_2015.tar.gz cd demo/mpi_bdb_ctf
		View on sparx-em.org	Preview by Yahoo

As it is a demo, the amount of noise is ridiculously low.  To have exactly what you want, you would have to modify some hardwired

parameters, such as amount of noise, envelopes, and most of all defocus and voltage values.  It is all in sxprocess code.

Pawel.

[Steven Ludtke]

Ok, after some research, I discovered that the 'math.simulatectf' processor, which does pretty much exactly what you requested, was accidentally disabled by one of my programmers about a week after I made it available. Literally, he just turned it off.  I just re-enabled it and tested it out, and it works quite well. This processor doesn't allow you to select a specific spectral profile for the noise, but has 2 user adjustable parameters, one for flatband Gaussian noise, and one for noise with a micrograph-like spectral profile. 

Please follow the instructions for the processor in the original posting if you want to play with it. With e2filtertool, you can easily interactively adjust the parameters to achieve the desired results, then use e2proc2d for bulk processing.

[je...@crystal.harvard.edu]

Thank you very much Steve and Pawel for all these valuable suggestions. I did the implementation and it works fine. All I need to figure out now is what the appropriate noise parameters for my application should be.

Thanks again, Simon