.gain reference file not usable

[Sebastien Moniot]

Hi all,

I recently obtained a cryo-EM dataset collected on a Falcon 4 detector and thus with .eer files.

Warp handles them fine but I cannot get to use the reference.gain file for preprocessing. Warp only accepts .dm4, .mrc, or .em extensions.

I tried to replace the extension of my .gain file to the others without success, Warp then complaining that the file cannot be read.

Any Idea how to solve my problem?

Thanks.

Sebastien

[Takanori Nakane]

Hi,

An EER gain reference with an .gain extension is internally a TIFF file.

An important point is that Warp (and RELION) divide raw detector values by EER's gain reference in the MRC format.

This is the convention used by earlier versions of the Falcon4 controller. In contrast to this, a newer gain reference in the TIFF container

is a multiplicative gain, as in K2 and K3. That is, the raw values must be multiplied by the gain reference.

So, what you should do is:

1. convert .gain into .mrc

2. take the reciprocal of the file

Both can be done with e2proc2d.py in EMAN2.

Best regards,

Takanori Nakane

[Florian Roßmann]

Hi,

Sorry to open this up again. We followed these steps from Takanori, but we still see vertical lines on my averaged images. Preprocessing of the same .eer frames works fine in cryoSPARC(Live) and RELION.

Any ideas what else we could try?

Thanks,

Flo

[Takanori Nakane]

Hi,

Are your vertical lines very sharp? If so, they came from dead lines on your detector.

These pixels are marked by 0 in the gain reference and RELION tries to conceal them

by replacing them with random values taken from neighboring good pixels.

I don't know if Warp has such a defect correction algorithm.

Best regards,

Takanori Nakane

[Florian Roßmann]

Hi Takanori,

Thanks for the quick reply. Yes, it is a pixel defect (3 pixels in a row) and it was also concealed in the camera software, which is why the line does not appear in mrc files. I thought the gain ref would take care of it...

Flo

[Takanori Nakane]

Hi,

Dead pixels have the value of zero, so you cannot conceal them by multiplying them with anything...

I keep saying TFS that the defects should be dealt with by the camera software before writing a file,

because the best correction algorithm depends on the details of the camera hardware and can

vary from model to model. It is not practical for RELION, Warp etc to implement different algorithms for

every new camera.

Best regards,

Takanori Nakane

[Max Renner]

Hi,

>So, what you should do is:

>1. convert .gain into .mrc

>2. take the reciprocal of the file

>Both can be done with e2proc2d.py in EMAN2.

Sorry if this is obvious, but what is the command to take the reciprocal in e2proc2d.py?

Thanks! :)

[3P1L]

Dear All,

is there any workable solution? In fact, I tried all of the above, but the vertical lines remain.

I've tried making a separate defects file from the initial .gain, too. I guess 0s are dead pixels?

(My current workaround is using Relion's implementation)


Maybe we can find a solution and archive it here for posterity! ;-)

Best,

Philipp.

[Zhengyi Yang]

I'm still looking for the best way to process the .eer and .gain file. 

There are two ways that work for me, but I'd like some people to confirm as well.

1. convert the .gain file to .mrc, either use newstack -in *gain -ou gain.mrc, or simply rename the suffix should do the job. 

2. find the reciprocal. one way is the relion_estimate_gain, since it only takes star file as input, you will need to create a relion project and import the above gain.mrc. 

alternatively, you can open the gain.mrc in FIJI, Process -> Math -> reciprocal, then save it. 

I found neither way elegant. so any other suggestion would be very welcome. 

[Max Renner]

Hi Zhengy and Philipp,

This is what I have been doing to prepare the gain ref for Warp, starting from a .gain file:

tif2mrc GainReference.gain gain.mrc

e2proc2d.py --process math.reciprocal gain.mrc gain_reciprocal.mrc

I got ~4.7A for STA from a small set of ribosome tilt series (using WARP-RELION-M) using this gain ref, so I think it should be ok...

Hope that helps :)

Max

[Zhengyi Yang]

Dear Max, Thanks for the reply. For anyone's information, now there is a very good explanation on relion documentation.

The confusion here is because historically Falcon is producing .eer files with .mrc gain reference. To apply gain reference, the .eer needs to be divided by the .mrc, which is the opposite of what you need to do with K2/K3 raw tiffs (multiplying). 

The .gain reference is a recent development, which you only need to multiply to the raw .eer, so similar to K2/K3, which should make our life easier, except Warp doesn't accept .gain format yet. If you convert .gain to .mrc, Warp thinks this is the old version that it needs dividing, hence giving you the wrong result.

Best,

Zhengyi

[3P1L]

Hi @All!

Just to come back to this ... I am not sure if the above is still true ...

At least not for eer data with a falcon4i (recorded with the newest SerialEM) and Warp 1.1 beta.

tif2mrc + 1/gain does not look OK -> see attached image

tif2mrc looks good

Does anybody know if the .gain was changed (to multiplicative), or does Warp handle it differently now?

🤷‍♂️

Philipp.

[Jinyi Zhu]

Hi Philipp,

If you use the commands from Max Renner to generate gain.mrc, I would suggest you try to:

1) flip y axis for the gain in warp 1.1 beta,

2) turn on "use movie sum" in CTF section. 

My data were captured by EPU directly, so I am not sure if those apply to your SerialEM collected data

It is odd for flip Y axis as the same .gain file was used in Relion 4 and CryoSparc without any flipping and never caused any trouble. However, without flipping, I always see the bright vertical lines.

Jinyi

[3P1L]

Hey Jinyi,

thanks. The flipping does remove the bright line.

Philipp.

[Tom]

Hi all,

this is a bit of a different topic, but to all I see for now, it relates to the gain reference inversion. I am seeing small white rectangular patches in my gain corrected .eer files (I used the eman2 commands to convert/invert the gain file). They translate to white stripes orthogonal to the image plane, after alignment. Did anyone see this in their data (and could find a reason/fix for it)?

Best,

Tom