Error in MCore

[Joshua Hutchings]

Hi all,

I've encountered an error with MCore.

The tutorial is running fine, and I was able to get a good initial reconstruction without running any refinements. 

However, it crashes when performing the first round of refinements (terminal output below), and the worker log references a "System.ArithmeticException: Function does not accept floating point Not-a-Number values." error, before refinements start (also pasted below).

Does this look like a memory/computation issue, or any pointers on which inputs/files I can check?

Thanks!

Josh

Terminal output:

"""

MCore \
> --population m3/refine1_v3.population \
> --refine_imagewarp 6x4 \
> --refine_particles \
> --ctf_defocus \
> --ctf_defocusexhaustive
Loading population... Done
Creating directories... Done
Spawning workers... Done
Preparing for refinement ��� this will take a few minutes per species
Preparing refinement requisites...
1/1                                                                                                                                      
Performing refinement
Preparing population for data source refine1_v3...Done
Loading gain reference for refine1_v3... Done
Refining all series in data source...
0/493Unhandled exception. System.Exception: BadRequest: System.Net.Http.HttpConnectionResponseContent
   at Warp.WorkerWrapper.SendCommand(NamedSerializableObject command) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/WorkerWrapper.cs:line 231
   at Warp.WorkerWrapper.MPARefine(String path, String workingDirectory, ProcessingOptionsMPARefine options, DataSource source) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/WorkerWrapper.cs:line 458
   at MCore.MCore.<>c__DisplayClass16_4.<DoProcessing>b__4(Int32 ifile, Int32 threadID) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/MCore/MCore.cs:line 473
   at Warp.Tools.Helper.<>c__DisplayClass51_0.<ForCPU>b__0(Object id) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/Tools/Helper.cs:line 719
Aborted (core dumped)

"""

Worker crash log:

"""

System.ArithmeticException: Function does not accept floating point Not-a-Number values.
   at System.Math.Sign(Single value)
   at Warp.Tools.Matrix3.EulerFromMatrix(Matrix3 a) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/Tools/Matrix3.cs:line 307
   at Warp.TiltSeries.GetParticleAngleInAllTilts(float3[] coords, float3[] angle) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/TiltSeries.cs:line 9715
   at Warp.TiltSeries.<>c__DisplayClass100_2.<PerformMultiParticleRefinement>b__11(float2 shiftBias, Species Species) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/TiltSeries.cs:line 6191
   at Warp.TiltSeries.<>c__DisplayClass100_2.<PerformMultiParticleRefinement>b__12(float2 shiftBias) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/TiltSeries.cs:line 6283
   at Warp.TiltSeries.<>c__DisplayClass100_2.<PerformMultiParticleRefinement>b__26(Double[] input) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/TiltSeries.cs:line 6675
   at Accord.Math.Optimization.BaseOptimizationMethod.<>c__DisplayClass27_0.<Maximize>b__0(Double[] x)
   at Accord.Math.Optimization.LBFGS.<>c__DisplayClass2_0.<NewMethod>b__0(Object instance, Double[] x, Double[] g, Int32 n, Double step)
   at Accord.Math.Optimization.LBFGS.line_search_morethuente(Int32 n, Double[] x, Double& f, Double[] g, Double[] s, Double& stp, Double[] xp, Double[] gp, Double[] wa, callback_data_t& cd, lbfgs_parameter_t& param)
   at Accord.Math.Optimization.LBFGS.lbfgs(Int32 n, Double[] x, Double& ptr_fx, lbfgs_evaluate_t proc_evaluate, lbfgs_progress_t proc_progress, Object instance, lbfgs_parameter_t _param)
   at Accord.Math.Optimization.LBFGS.main(Double[] start, Func`2 fn, Func`2 gn, EventHandler`1 progress, lbfgs_parameter_t param)
   at Accord.Math.Optimization.BroydenFletcherGoldfarbShanno.Optimize()
   at Accord.Math.Optimization.BaseOptimizationMethod.Maximize()
   at Accord.Math.Optimization.BaseGradientOptimizationMethod.Maximize()
   at Accord.Math.Optimization.BaseOptimizationMethod.Maximize(Double[] values)
   at Warp.TiltSeries.PerformMultiParticleRefinement(String workingDirectory, ProcessingOptionsMPARefine optionsMPA, Species[] allSpecies, DataSource dataSource, Image gainRef, DefectModel defectMap, Action`1 progressCallback) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpLib/TiltSeries.cs:line 7759
   at WarpWorker.WarpWorker.EvaluateCommand(NamedSerializableObject Command) in /usr/share/miniconda/envs/package-build/conda-bld/warp_1732560643671/work/WarpWorker/WarpWorker.cs:line 731

"""

[Alister Burt]

Hi Josh,

Thanks for the complete report! I haven’t seen this one before, seems like something internally is producing a NaN which causes this problem further down the line… 

You could try to refine with one ts at a time to see if it’s a specific ts that is causing issues, that might help you narrow down the problem… you could do this by using the change_selection tool then recreating the M project.

As I’ve run things successfully with a large group on the latest conda build last week I’m inclined to say there’s something up with your inputs - if all TS fail to refine and you can’t find anything obviously wrong in the inputs please report back and we’ll take a look!

Cheers,

Alister

Sent from mobile - apologies for brevity

On Apr 11, 2025, at 22:06, 'Joshua Hutchings' via Warp <war...@googlegroups.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/warp-em/3e843c85-d8a8-443d-9d2f-281db38eafbcn%40googlegroups.com.

[Joshua Hutchings]

Hi Alister, 

Thanks for the tips, still troubleshooting but will report back soon!

Related, is there a way to check what the selection status is for a settings file, without actually selecting/deselecting anything?

Best,

Josh

[Alister Burt]

Hi Josh,

I think if you run change_selection with your settings file and no other flags it will report the status and no-op over each of them

Cheers,

Alister

[Joshua Hutchings]

That gives me "Unhandled exception. System.Exception: Choose exactly 1 of the options", but it is not a big deal anyway!

Best,

Josh

[Alister Burt]

heh - in that case run with invert twice 😂