White Spots Created in Warp, but not RELION5

[Luke]

Hello all, 

When processing a series of tomograms in warp, I'm observing some artificial bright white spots shown here:

The white spots spread along the rotation axis as the tilt increases.

When I process in RELION5 using the same gain, reconstruction dimensions, and dose parameters the white dots are not present.  I have also used both a .gain and tif2mrc converted gain thinking that could be the issue. 

Does anyone have any idea what might be different between the two reconstructions?  

[Huy Bui]

Hi Luke,
You can look at the stack from Warp & Relion if you see very bright pixels there (or even the motion corrected micrographs). There are certain times that our camera produces a lot of hot pixels that cannot be corrected using the gain reference and might/might not be addressed using IMOD Xray removal.

So, at least you can conclude whether Relion corrects for those hot pixels better.

Best,
H

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/22476e4b-cbc3-4546-81c3-670f6477ed0an%40googlegroups.com.

[Marten Chaillet]

Hi Luke,

Those hot pixels don't get corrected by the gain file. I think IMOD and RELION do some automatic thresholding to filter out these pixels while Warp does not. The ringing you see around them happens because of the CTF correction during reconstruction. In the averaged frames they should visible as single pixels. 

I had the same issue and managed to correct it by creating a defects file myself (I probably have some notes on it somewhere but need to search). But to get it working I had to make a bug fix and make a custom installation of WarpTools. And my PR has not been merged yet: https://github.com/warpem/warp/pull/45 .

If I find the notes I will post them :)

Cheers,
Marten

Op dinsdag 7 april 2026 om 20:06:03 UTC+2 schreef huy...@gmail.com:

[Marten Chaillet]

I found it: https://gist.github.com/McHaillet/dfddd32eff547198c5d6e091b9e2ac32 

I think masking above 8 standard deviations is what IMOD does by default.

Op woensdag 8 april 2026 om 15:51:33 UTC+2 schreef Marten Chaillet:

[Luke]

Hello Marten and Huy, 

Thank you both for pointing this out.  Specifically thank you Marten for sending me your script, I'll give that a shot and report back. 

--Luke 

[Warp Bot]

Hi Luke,

Huy and Marten are correct -- this is a hot pixel issue. To add some detail from the codebase:

In the WarpTools CLI, the automatic X-ray pixel correction (GPU.Xray) that the old Warp GUI used is disabled. Hot pixel removal only happens if you provide a defects file explicitly. RELION, by contrast, does its own automatic outlier thresholding during preprocessing, which is why you don't see the spots there.

To fix this in WarpTools, you can supply a defects file when creating your settings:

WarpTools create_settings [...] --defects_path path/to/defects.mrc

The defects file should be an MRC image the same size as your frames, with non-zero values marking bad pixels. Marten's script (linked above) that identifies hot pixels by thresholding at several standard deviations above the mean is a good approach for generating one.

Note that a bug in defects file handling was fixed in PR #65 (https://github.com/warpem/warp/pull/65), which has been merged into the current version.

— Warp Bot
THIS IS AN AUTOMATED MESSAGE GENERATED BY AN LLM. IT MAY OR MAY NOT SOLVE YOUR PROBLEM. IF YOU'D LIKE TO SPEAK TO A HUMAN, SAY SO IN YOUR MESSAGE.

--

[teg...@gmail.com]

I'll add that this should be ideally solved via gain reference. Outlier removal existed in earlier versions, but using a threshold that works well for SPA data (like 8) generated too many false positives in much noisier tilt series frames. I think I set it to 20 sigmas at some point. I don't have hands-on experience with old CCD data, but my impression was that direct detectors were much less susceptible to the assumed source of non-persistent hot pixels in images: X-rays. And since every persistent hot pixel should be captured by the gain reference, I just disabled outlier detection completely.

Cheers,

Dimitry

[Luke]

Hello everyone, 

I followed Martens advice and created a series of defects files by masking the hot pixels. Beginning with a SD of 8, and then eventually going all the way down to SD of 2.5 to ensure that every potential hot pixel would be masked.  An example of the defects file is shown here.  I also flipped all the defect files on the X and Y axis, the X and Y together, and no flip at all  but  each attempt  was unsuccessful.

Interestingly, the defects are different in each tilt --I'm not sure if that could be indicative of the issue.  Below are thumbnail outputs from WarpTools fs_motion_and_ctf after adding in a defects file. Each time, the defects are in the same position regardless of the flip of the defect file.

 

[teg...@gmail.com]

Hi Luke,

What are the processing commands you're currently using? This looks like the images were scaled down significantly judging by the ringing around the bright white spot. The random hot pixels you're detecting are just false positives due to the very low (2.5) threshold you chose. I see a single actual camera defect. Does it not show up in your gain reference?

Cheers,

Dimitry

[Luke]

Hey Dimitry, 

The images are in fact scaled down because I was having warp generate thumbnails after fs_motion_and_ctf. 

You are correct, there is a single defect. The gain reference  i was originally using is shown here:

I noticed that the defect traveled with the gain reference, so flipping the gain on the x-axis resulted in the artifact flipping. 

After some troubleshooting, I ran this script, on the gain reference mrc file. Which gave this output which solved the issue. 

[Luke]

I wanted to quickly elaborate on what I think caused the issue. 

For our falcon4 camera, the output is a “.gain” file. Typically, we have converted the .gain file to a “.mrc’ file using IMOD’s tif2mrc command, which has not been an issue in the past.

For some reason the tif2mrc converted gain, did not effectively mask the hot pixels, in fact it made the hot pixel issue worse. But treating the tif2mrc converted gain with the script, which masked hot pixels using a SD threshold (similar to what Marten recommended, but applied to the gain) reverted the .gain file back to successfully masking the hot pixels. 

So if anyone is looking at this in the future, and uses a falcon4 with a converted gain, and the defects follow the flip of your gain, you may try using the script mentioned above on your gain reference file to correct the issue.

You received this message because you are subscribed to a topic in the Google Groups "Warp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp-em/OLJhGAyXrsY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/537503cf-b9c0-4503-b79d-38c81d2a628bn%40googlegroups.com.

[teg...@gmail.com]

Hi Luke,

Thanks for confirming! 

There are 2 common conventions for gain references: One takes 1/gain to produce a reference that the images must be multiplied by for correction. The other just reports the gain pattern, so you have to divide the images by those values to correct. 

Gatan's software has always produced type #1. However, I'm still not sure what the situation is in EPU: When the EER format just came out, the example data I used to develop support for the new format had a .gain file of type #2. However, all Falcon 4 gain references I've seen since then have been type #1, so Warp interprets .gain files same as .tif or .mrc gain references: multiplicative. 

If applying your original gain reference amplified the hot pixel, that sounds like type #2.

If someone knows more about the current state of gain reference conventions, please chime in.

Cheers,

Dimitry