cuFFT error in WarpTools template matching on large tomogram

[Irene Hu]

I am trying to run template matching at different pixel sizes (WarpTools v2.0.0dev34) but encounter a cuFFT error: CUFFT_INTERNAL_ERROR on a tomogram reconstructed at 2070x2070x1020 px, while the same parameters run fine on the same tomogram reconstructed with dimensions 1242×1242×606 px (24 px template). With the larger tomogram, I tried the following settings:
• --batch_angles: 1, 4, 16, 32
• --subdivisions: 2, 3
• --subvolume_size: 64 , 128, 192
• one or two A40s
• template with box sizes 24, 32 and 40 px

All produce the following error
Log excerpt (full log attached):
Received "TomoMatch"... 45215 MB free
Using 18432 orientations
0%
Using 192 sub-volumes
Loading...
0%
terminate called after throwing an instance of 'std::runtime_error'
what(): cuFFT error: CUFFT_INTERNAL_ERROR
at .../gtom/src/FFT/FFT.cu:23

Has anyone experienced anything similar or have advice on running template matching with larger volumes?

Thanks,
Irene Hu
PhD Candidate, Fung Lab
Life Sciences Institute / Department of Cell and Developmental Biology
University of Michigan

[Hamidreza Rahmani]

In my case this always has been a memory issue. I got better resutls when passed the gpuids explicitly or changed paramrets to use less GPU memory. 

Best,

Hamid

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/fc8c32df-5bef-442a-9a7a-6a816e8b8b25n%40googlegroups.com.

[Irene Hu]

Hi Hamid,

Thanks for the insight! I’ve explicitly passed the GPU IDs as well, and since I’m using two A40s (so 96 GB total GPU memory), I was surprised it still ran into the same issue. Do you mind sharing which parameters you changed to reduce GPU memory usage? I tried a few flags, but none of them seemed to actually lower the GPU memory consumption when this error occurred.

Best,
Irene

[Alister Burt]

Hi Irene

One major factor dictating memory in TM is the size of your tomogram along Y, try matching at a larger pixel size?

This one can sneak up on you if you collected at a larger pixel size and didn’t downscale too much for reconstruction

Cheers,

Alister

Sent from mobile - apologies for brevity

On Oct 17, 2025, at 07:20, Irene Hu <hui...@umich.edu> wrote:



To view this discussion visit https://groups.google.com/d/msgid/warp-em/02163561-bb42-4379-854d-d24d478600aen%40googlegroups.com.

[Irene Hu]

Hi Alister,

Thanks so much for the reply — that makes a lot of sense! I was benchmarking to see whether using a smaller pixel size could help improve template matching performance on our dataset.

I also noticed that PyTom has the --volume-split option for splitting large tomograms to fit within GPU memory. Do you think there’s any possibility of a similar feature being incorporated into Warp or the current template matching pipeline in the future?

Best,
Irene