Crash in M

[Qiang Su]

Hi  Dimitry,

I am studying the amazing Warp & M software. I am using the tile series data  EMPIAR-10164 you suggest. I strictly follow the procedure. Everything goes well in Warp, IMOD and RELION. I used bin4 pixel size (1.35X4=5.4) not you suggest in the paper. However, when I imported the data into M, M is crashed. The crash information is as below. I also attached the parameter of the refine. Could you please help me on this problem? Is there any GPU memory or parameter problem? 

CPU: Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz   2.59 GHz

RAM 128G

GPUV100, 32G

Thanks,

Qiang

[Qiang Su]

The lastclash information

2020/11/16 1:32:25:

System.Exception: System.Exception: cudaErrorIllegalAddress

   at Warp.GPU.CheckGPUExceptions() in D:\Dev\warp2\WarpLib\GPU.cs:line 1183

   at Warp.TiltSeries.PerformMultiParticleRefinement(String workingDirectory, ProcessingOptionsMPARefine optionsMPA, Species[] allSpecies, DataSource dataSource, Image gainRef, DefectModel defectMap, Action`1 progressCallback) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3277

   at WarpWorker.Program.Main(String[] args) in D:\Dev\warp2\WarpWorker\Program.cs:line 294

   在 Warp.WorkerWrapper.GetSuccessStatus() 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 96

   在 Warp.WorkerWrapper.MPARefine(String path, String workingDirectory, String logPath, ProcessingOptionsMPARefine options, DataSource source) 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 214

   在 M.MainWindow.<>c__DisplayClass30_6.<PerformRefinementIteration>b__22() 位置 D:\Dev\warp2\M\MainWindow.xaml.cs:行号 817

   在 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   在 System.Threading.ThreadHelper.ThreadStart()

   在 Warp.WorkerWrapper.GetSuccessStatus() 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 96

   在 Warp.WorkerWrapper.MPARefine(String path, String workingDirectory, String logPath, ProcessingOptionsMPARefine options, DataSource source) 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 214

   在 M.MainWindow.<>c__DisplayClass30_6.<PerformRefinementIteration>b__22() 位置 D:\Dev\warp2\M\MainWindow.xaml.cs:行号 817

   在 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   在 System.Threading.ThreadHelper.ThreadStart()

[Qiang Su]

System.Exception: System.OverflowException: Value was either too large or too small for a Decimal.

   at System.Decimal..ctor(Single value)

   at Warp.TiltSeries.<>c__DisplayClass87_2.<PerformMultiParticleRefinement>b__24(Double[] input, TiltSeries series, Boolean setParticles) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3882

   at Warp.TiltSeries.<>c__DisplayClass87_2.<PerformMultiParticleRefinement>b__26(Double[] input) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 4006

   at Accord.Math.Optimization.BaseOptimizationMethod.<>c__DisplayClass27_0.<Maximize>b__0(Double[] x)

   at Accord.Math.Optimization.LBFGS.<>c__DisplayClass2_0.<NewMethod>b__0(Object instance, Double[] x, Double[] g, Int32 n, Double step)

   at Accord.Math.Optimization.LBFGS.line_search_morethuente(Int32 n, Double[] x, Double& f, Double[] g, Double[] s, Double& stp, Double[] xp, Double[] gp, Double[] wa, callback_data_t& cd, lbfgs_parameter_t& param)

   at Accord.Math.Optimization.LBFGS.lbfgs(Int32 n, Double[] x, Double& ptr_fx, lbfgs_evaluate_t proc_evaluate, lbfgs_progress_t proc_progress, Object instance, lbfgs_parameter_t _param)

   at Accord.Math.Optimization.LBFGS.main(Double[] start, Func`2 fn, Func`2 gn, EventHandler`1 progress, lbfgs_parameter_t param)

   at Accord.Math.Optimization.BroydenFletcherGoldfarbShanno.Optimize()

   at Accord.Math.Optimization.BaseOptimizationMethod.Maximize()

   at Accord.Math.Optimization.BaseGradientOptimizationMethod.Maximize()

   at Warp.TiltSeries.PerformMultiParticleRefinement(String workingDirectory, ProcessingOptionsMPARefine optionsMPA, Species[] allSpecies, DataSource dataSource, Image gainRef, DefectModel defectMap, Action`1 progressCallback) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 5071

   at WarpWorker.Program.Main(String[] args) in D:\Dev\warp2\WarpWorker\Program.cs:line 294

   在 Warp.WorkerWrapper.GetSuccessStatus() 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 96

   在 Warp.WorkerWrapper.MPARefine(String path, String workingDirectory, String logPath, ProcessingOptionsMPARefine options, DataSource source) 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 214

   在 M.MainWindow.<>c__DisplayClass30_6.<PerformRefinementIteration>b__22() 位置 D:\Dev\warp2\M\MainWindow.xaml.cs:行号 817

   在 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   在 System.Threading.ThreadHelper.ThreadStart()

   在 Warp.WorkerWrapper.GetSuccessStatus() 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 96

   在 Warp.WorkerWrapper.MPARefine(String path, String workingDirectory, String logPath, ProcessingOptionsMPARefine options, DataSource source) 位置 D:\Dev\warp2\WarpLib\WorkerWrapper.cs:行号 214

   在 M.MainWindow.<>c__DisplayClass30_6.<PerformRefinementIteration>b__22() 位置 D:\Dev\warp2\M\MainWindow.xaml.cs:行号 817

   在 System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   在 System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   在 System.Threading.ThreadHelper.ThreadStart()

[teg...@gmail.com]

Hi Qiang,

Does this error occur in the very first tilt series, or further down the road during refinement? You posted 2 different error messages: the first one (from lastcrash.txt) relates to a memory allocation problem. How much RAM does this computer have? Did you create your species with 1.35 A/px references, or 4x binned ones? If there is not much RAM, you can try disabling "Use CPU memory for particles" since your GPU has plenty of memory. The second error message can have many causes. You can rule out a few by disabling all refinement options except "particle poses".

Cheers,

Dimitry

[Jesse Hansen]

hi all,

I am encountering the same error (below).

All was working well in M. I had refined particle poses, image warp, doming, stage angles, volume warp, and CTF without issue.  Now I’m refining movies and I guess there were some problematic tilt series because it would always get stuck on the same series.  So I removed particles from that series from the star file, then setup a new species in M. I tried running again with movie refinement and I get the error below (occurs after about 50 tilt series processed, not at the start).

As Dimitry suggested I ran with only particle poses refining, and I received the same error.  

any help appreciated.  Thank you!

Jesse  

System.OverflowException: Value was either too large or too small for a Decimal.
   at System.Decimal..ctor(Single value)
   at Warp.TiltSeries.<>c__DisplayClass87_2.<PerformMultiParticleRefinement>b__24(Double[] input, TiltSeries series, Boolean setParticles) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3882
   at Warp.TiltSeries.<>c__DisplayClass87_2.<PerformMultiParticleRefinement>b__26(Double[] input) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 4006
   at Accord.Math.Optimization.BaseOptimizationMethod.<>c__DisplayClass27_0.<Maximize>b__0(Double[] x)
   at Accord.Math.Optimization.LBFGS.<>c__DisplayClass2_0.<NewMethod>b__0(Object instance, Double[] x, Double[] g, Int32 n, Double step)
   at Accord.Math.Optimization.LBFGS.line_search_morethuente(Int32 n, Double[] x, Double& f, Double[] g, Double[] s, Double& stp, Double[] xp, Double[] gp, Double[] wa, callback_data_t& cd, lbfgs_parameter_t& param)
   at Accord.Math.Optimization.LBFGS.lbfgs(Int32 n, Double[] x, Double& ptr_fx, lbfgs_evaluate_t proc_evaluate, lbfgs_progress_t proc_progress, Object instance, lbfgs_parameter_t _param)
   at Accord.Math.Optimization.LBFGS.main(Double[] start, Func`2 fn, Func`2 gn, EventHandler`1 progress, lbfgs_parameter_t param)
   at Accord.Math.Optimization.BroydenFletcherGoldfarbShanno.Optimize()
   at Accord.Math.Optimization.BaseOptimizationMethod.Maximize()
   at Accord.Math.Optimization.BaseGradientOptimizationMethod.Maximize()
   at Warp.TiltSeries.PerformMultiParticleRefinement(String workingDirectory, ProcessingOptionsMPARefine optionsMPA, Species[] allSpecies, DataSource dataSource, Image gainRef, DefectModel defectMap, Action`1 progressCallback) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 5071
   at WarpWorker.Program.Main(String[] args) in D:\Dev\warp2\WarpWorker\Program.cs:line 294

[Jesse Hansen]

I noticed it happened twice no the same tilt series, so I removed that tilt series and tried again.   Now it's an error on a different tilt series.

I tried replacing the xml files in the WARP folders with an earlier version which ran fine.  Still did not resolve it.

This began when I tried to refine the movie frames.

[Jesse Hansen]

update: I seem to have resolved it for now.  

as stated above:
replacing XML files didn't help, nor did reimporting the species, nor did running with just refined poses. 

I went back to WARP and reprocessed the frame series (.tif not .tomostar) and also swapped my population file for an earlier one.  So one of those two things made the difference.   

Jesse