M species creation crashes Windows - training denoising model

[Dennis Michalak]

Hello,

During the species creation in M, steps 1,2, and 3 out of 5 go well and after a few seconds at 4/5, the PC hard crashes. The screen goes black and starts a boot up again. It looks like the crash happens such that no error logs are created by the system and only after boot up does it recognize something went wrong.

Is it possible that the step that trains the denoising model could completely fill the GPU memory? I thought the training only uses two refined half-maps and not individual particles themselves. That's the only thing I can think of that might cause the crash.

Best,

Dennis

[teg...@gmail.com]

Hi Dennis,

Wow, a reboot isn't easy to cause! Is that 1.0.9 or 1.1.0? What GPU(s) are you using? Training is quite memory-heavy, though mostly not because of the half-maps but because of the intermediate results the neural net needs to keep for gradient calculation.

Cheers,

Dimitry

[Dennis Michalak]

Hi Dimitry,

I see, thanks for the info. 

I'm using:
Nightly build 2020-11-04 (Is this 1.1.0?)

NVIDIA GeForce GTX 1080 Ti - 11GB

Dennis

[teg...@gmail.com]

Hi Dennis,

1.1.0 is this one: https://groups.google.com/g/warp-em/c/dAsmMnlULJA

A single card with 11 GB is likely insufficient. If you have a second card with even 6 or 8 GB lying around, adding it to that system might solve the problem.

Cheers,

Dimitry

[Dennis Michalak]

Thanks!

---

From: war...@googlegroups.com <war...@googlegroups.com> on behalf of teg...@gmail.com <teg...@gmail.com>
Sent: Wednesday, October 5, 2022 3:06:31 PM
To: Warp <war...@googlegroups.com>
Subject: Re: M species creation crashes Windows - training denoising model

 

--
You received this message because you are subscribed to a topic in the Google Groups "Warp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp-em/GK_Nu2XDC_M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/efbc9417-2e11-4862-a13c-68af33a4ec98n%40googlegroups.com.