Sorry to open this up again, I'm running the latest version patched with the nightly build files and am getting the following error which looks similar to Shyamal's? Is this the same?
Running on GPU #1 (13633 MB free) through WarpWorker0f33726d-37a9-4dd8-9aae-da47ab22d1f6
Received "SetHeaderlessParams", with 3 arguments, for GPU #1, 13633 MB free
Set headerless parameters to 2, 2, 0, float
0.000
Received "MPAPreparePopulation", with 1 arguments, for GPU #1, 13633 MB free
Preparing hex64 for refinement... 2020-08-31 15:52:41.517185: I D:\Dev\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-08-31 15:52:41.764715: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 0 with properties:
name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:17:00.0
totalMemory: 16.00GiB freeMemory: 13.32GiB
2020-08-31 15:52:41.777151: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1405] Found device 1 with properties:
name: Quadro RTX 5000 major: 7 minor: 5 memoryClockRate(GHz): 1.815
pciBusID: 0000:73:00.0
totalMemory: 16.00GiB freeMemory: 13.31GiB
2020-08-31 15:52:41.787661: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2020-08-31 15:52:44.142108: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 15:52:44.148279: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 1
2020-08-31 15:52:44.153252: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N N
2020-08-31 15:52:44.157953: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 1: N N
2020-08-31 15:52:44.163503: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12889 MB memory) -> physical GPU (device: 0, name: Quadro RTX 5000, pci bus id: 0000:17:00.0, compute capability: 7.5)
2020-08-31 15:52:44.180712: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 12885 MB memory) -> physical GPU (device: 1, name: Quadro RTX 5000, pci bus id: 0000:73:00.0, compute capability: 7.5)
2020-08-31 15:52:44.229194: I d:\dev\warp2\tfutility\graph.cpp:369] not cuda_malloc
2020-08-31 15:52:44.244059: I d:\dev\warp2\tfutility\graph.cpp:262] Loading SavedModel from: D:\WM5984_180924_DCheR_minicells\species/2d294638/hex64_noisenet
2020-08-31 15:52:44.364399: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1484] Adding visible gpu devices: 0, 1
2020-08-31 15:52:44.369927: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 15:52:44.379757: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 1
2020-08-31 15:52:44.386214: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N N
2020-08-31 15:52:44.391790: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 1: N N
2020-08-31 15:52:44.398918: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12889 MB memory) -> physical GPU (device: 0, name: Quadro RTX 5000, pci bus id: 0000:17:00.0, compute capability: 7.5)
2020-08-31 15:52:44.418976: I D:\Dev\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 12885 MB memory) -> physical GPU (device: 1, name: Quadro RTX 5000, pci bus id: 0000:73:00.0, compute capability: 7.5)
2020-08-31 15:52:44.490732: I d:\dev\warp2\tfutility\graph.cpp:182] Restoring SavedModel bundle.
2020-08-31 15:52:44.898505: I d:\dev\warp2\tfutility\graph.cpp:215] Running LegacyInitOp on SavedModel bundle.
done.
Preparing data:
Done.
16.692
Received "LoadGainRef", with 5 arguments, for GPU #1, 13259 MB free
Loaded gain reference and defect map: , False, False, False,
0.003
Received "MPARefine", with 5 arguments, for GPU #1, 13259 MB free
Loading tilt series and masks...
12716 MB after loading raw data of tomo_ts105.mrc.tomostar
Calculating spectral weights...
12714 MB after spectra estimation of tomo_ts105.mrc.tomostar
Precomputing gradient weights...
Unhandled Exception:
Unhandled Exception:
Unhandled Exception:
Unhandled Exception: System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception:
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception: System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception: System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception:
Unhandled Exception:
Unhandled Exception: Unhandled Exception: Unhandled Exception: System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception: Unhandled Exception: Unhandled Exception:
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)
Unhandled Exception: System.Exception
at Warp.TiltSeries.<>c__DisplayClass87_15.<PerformMultiParticleRefinement>b__56(Int32 iwiggle, Int32 threadID) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 3967
at Warp.Tools.Helper.<>c__DisplayClass48_0.<ForCPU>b__0(Object id) in D:\Dev\warp2\WarpLib\Tools\Helper.cs:line 679
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)