Defect map does not match image dimensions

[Niharika Shukla]

Hi everyone,

I am trying to apply a defect map during motion correction after pointing to it during creation of the settings files.

WarpTools fs_motion_and_ctf --settings warp_frameseries.settings --m_grid 1x1x6 --c_grid 2x2x1 --c_range_max 6 --c_range_min 40 --c_use_sum --out_averages

However, I consistently get the following error:

System.Exception: Image dimensions don't match those of the defect map\n

When I check the headers of the frames, the gain reference, and the defect map, they all appear to be the same dimensions in X and Y. Both the gain reference and defect map were originally exported as .dm4 files, but I continue to get this error if I convert to .tif as well. When I convert the defect map to a .mrc, the motion correction job instead gets stuck at:

Connecting to workers...

I am able to process this dataset when I omit the defect map.

Thank you for any help!

Best,
Niharika

[Alister Burt]

Hi Niharika,

Dimitry made some updates recently which I believe included EER upsampling by default - this might be the cause.

Don’t have an immediate fix for you but you could try to either:

- process without defect map

- go back to 2.0.0dev34

- check for options related to EER upsampling (not 100% on the strategy/defaults/how this was exposed if at all)

Cheers,

Alister

Sent from mobile - apologies for brevity

On Sep 26, 2025, at 15:26, Niharika Shukla <ns.nihari...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/398e6d6e-da55-4dee-806c-4f2997ca9592n%40googlegroups.com.

[Niharika Shukla]

Hi Alister,

Thank you for the quick response!

I have tried reconstruction through Warp without the defect map, but I find that the alignment is quite poor. When manually reconstructing the same tilt series in IMOD after manual replacement of the defective pixels during preprocessing, the reconstruction is much improved.

I am currently running 2.0.0dev34.

The images were collected with a K3 detector in super-resolution mode as 6 frames and exported as .tif files, so I have not been using any options related to EER fractionation. I found that removing the pre-processing bin factor in the warp_frameseries.settings does allow motion correction to run without error. 

However, though the defect map now appears to be applied, it does not seem to effectively replace the pixels with the average value of surrounding pixels, and the alignment and reconstruction do not appear much improved. I have attached a screenshot of a motion corrected average as an example. 

I would greatly appreciate any suggestions you may have.

Thank you,

Niharika

[Alister Burt]

Hi Niharika,

Agh, I'm sorry about this - we don't routinely process data with defect files, is there a reason your gain does not include these lines?

I hear from a colleague here that in their experience these defects do not negatively impact motion correction too much.

If we take that as true, the problem we have to deal with is fixing the motion corrected averages. 

There's no easy command line tool which will solve this directly for you but I have a Python package called torch-cryoeraser which can be used to inpaint masked regions similar to Warp's EraseDirt method. You could either try to generate a mask from your defect file programmatically or paint one using something like napari (uvx --with napari-mrcfile-reader --with pyqt5 napari)

There will be other less python-y methods to solve this but that's how I would do it.

Cheers,

Alister

[teg...@gmail.com]

Hi Niharika,

Unfortunately, there isn't a single, widely accepted format for describing defects, and it had never been enough of an issue for me to add support for more formats. 

Can you share a single movie and your defects map?

Cheers,

Dimitry

[Niharika Shukla]

Hi Dimitry and Alister,

Thank you for your helpful responses. My gain reference does include this line, but gain correction alone does not seem to be enough to prevent the defect from affecting the alignment. I have applied torch-cryoeraser to the motion-corrected averages and filled in the defect. The alignment and reconstruction appear much improved when I use these defect-corrected averages.

I have attached a .zip file containing a test movie, the defect map, and gain reference in the format they are exported from our microscope. Hopefully these can prove helpful. I believe Warp desktop has an option to input a .txt file with the coordinates of the defective pixels that is then written into a .tiff by Warp. I was unable to successfully input a .txt file as a defect map in WarpTools, but perhaps adding in support for that format would be enough for most users.

Thank you again for all of your help!

Best,

Niharika

 DefectMapTestFiles.zip

To view this discussion visit https://groups.google.com/d/msgid/warp-em/7df44798-afed-42f5-a4d5-17685a7180c8n%40googlegroups.com.

[Hamidreza Rahmani]

Hi Niharika,

For another project that used montaged micrographs I wrote a code that finds areas where values are uniform and replaces them with random numbers. The code not well developed yet, but I can test it on your data and help you fix the whole dataset if you are interested.

Best,

Hamid

To view this discussion visit https://groups.google.com/d/msgid/warp-em/CAK6K4rNwFPcOxYxeBCo6mp2P_%3DgsDpuQxEUTjEWstLUbvSsdXg%40mail.gmail.com.

[Alister Burt]

Hey Niharika,

Really glad you were able to use our packages to solve your issue - thanks also for sharing your data!

Cheers,

Alister 

Sent from mobile - apologies for brevity

On Sep 30, 2025, at 15:33, Niharika Shukla <ns.nihari...@gmail.com> wrote: