Denoise and particle picking bug in Warp

[Zhaoqian Wang]

Hi,

Our lab just moved from old 2080TI box to a new GPU box for Warp with 4 RTX A5000.

We currently have fatal errors on Denoise and particle picking, which all related to neutral net. 

Hard to describe in words, the screenshot is attached. In short, Warp runs fine without any error message, but the output from Denoise is totally wrong. And I assume this leads the error in particle picking (no particle has been picked). The test set is old datasets that were processed by Warp on our old machine, so all the settings and reference should be good.

Please let me know if anyone have any idea what may cause this. Does Warp not support A5000, or a specific driver or setting may be needed?

Best,

Zhaoqian Wang

[Pranav Shah]

Fascinating! Your neural net is dreaming!

What happens if you retrain the denoiser specifically on your dataset?

Best,
Pranav
--
Pranav Shah
Postdoctoral Research Fellow.

Division of Structural Biology,
Wellcome Trust Centre for Human Genetics,
University of Oxford,
Roosevelt Drive, Oxford OX3 7BN,
UK

--
You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/6b7c46c7-6d86-4ce7-ba8b-7c89cd8cb603n%40googlegroups.com.

[Daniel Mann]

I've seen the same thing happening on 3090 cards. Retrain didn't help. Same dataset on 2080TIs = no problem

Cheers,

Dan

Am 14.10.21 um 10:16 schrieb Pranav Shah:

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/CAHNYQ5quPJyZi2CCpJ0Q4ZiCAaRtT4Ua6SXn7gCXh9dhveWsKQ%40mail.gmail.com.

-- 
Dr Daniel Mann

Forschungszentrum Jülich GmbH
Ernst-Ruska Centre for Microscopy and Spectroscopy with Electrons / ER-C-3 Structural Biology
Institute for Biological Information Processing / IBI-6 Cellular Structural Biology
Wilhem-Johnen-Straße, 52425 Jülich, Germany
Phone: +49 2461 61 9499
https://go.fzj.de/erc3-ibi6
Email: d.m...@fz-juelich.de

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Dr. Astrid Lambrecht,
Prof. Dr. Frauke Melchior
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

[Pranav Shah]

Could it be a CUDA issue? What happens if you use the latest version?

To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/1a240a38-b5d8-eced-ab54-0b673ff75fc6%40fz-juelich.de.

--

[mjja...@gmail.com]

Reviving this thread. I'm testing with 2x RTX A6000s and I have the same issue. So does WARP not support the RTX A series GPUs? Or has anyone found a solution?

Thanks,

Matt

[mjja...@gmail.com]

I've updated all drivers based on warpem website.

[teg...@gmail.com]

Hi Matt, have you tried 1.1.0 https://groups.google.com/g/warp-em/c/dAsmMnlULJA ?

[mjja...@gmail.com]

Hi Dmitry,

Yes, 1.1.0 gives a different error:
Damn, Warp just crashed!
Here are the details that were also saved in
C:\Users\VendorAppData\Roaming\Warp_110_beta1Vastcrash
txt.
System.InvalidOperationException: The calling thread cannot
access this object because a different thread owns it.
at System.Windows.Threading.Dispatcher.VerifyAccess0
at System.Windows.Application.get _MainWindow0
at
Warp.Controls.MicrographDisplay.<>c_DisplayClass159_2.€<
Updatelmage>b__1>d.MoveNextOin
D:\Devlwarp5\Warp\Controls\MainUNMicrographDisplay.xaml.
cs:line 684
- End of stack trace from previous location ---
at Warp.Controls.MicrographDisplay.Updatelmage0in
D:\Devlwarp5\Warp\Controls\MainU/MicrographDisplay.xaml.
cs:line 673
System.Threading.Tasks.Task.<>c.<ThrowAsync>b_128_0(Obje
ct state)
System.Windows.Threading.ExceptionWrapper.InternalRealCal
(Delegate callback, Object args, Int32 numArgs)
System.Windows.Threading.ExceptionWrapper.TryCatchWhen(
Object source, Delegate callback, Object args, Int32 numArgs,
Delegate catchHandler)
at System.Windows.Threading.Dispatcher.VerifyAccess0
at System.Windows.Application.get_MainWindow0
Warp.Controls.MicrographDisplay.<>c_DisplayClass159.2.<<
Updatelmage>b__1>d.MoveNextOin
D:\Dev\warp5\Warp\Controls\MainUNMicrographDisplay.xaml.
c:line 684
-.- End of stack trace from previous location --.
at Warp.Controls.MicrographDisplay.Updatelmage)in
D:\Dev\warp5\Warp\Controls\MainUNMicrographDisplay.xaml.
cs:line 673
ar
System.Threading.Tasks.Task.<>c.<ThrowAsync>b_128_0(Obje
ct state)
System.Windows.Threading.ExceptionWrapper.InternalRealCal
I(Delegate callback, Object args, Int32 numArgs)
System.Windows.Threading.ExceptionWrapper.TryCatchWhen(
Object source, Delegate callback, Object args, Int32 numArgs,
Delegate catchhandler)

Thanks,

Matt

[mjja...@gmail.com]

And a second pop-up error:

Damn, Warp just crashed!
Here are the details that were also saved in
C: \Users\VendorAppData\Roaming\Warp_110_beta1Vastcras

h.txt.
System.NullReferenceException: Object reference not set to
an instance of an object.
at Warp.NoiseNet2DTorch.Dispose0 in
D:\Dev\warp5\WarpLib\NNModels\NoiseNet2DTorch.cs:line
227
at Warp.NoiseNet2DTorch.Finalize0 in
D:\Dev\warp5\WarpLib\NNModels\oiseNet2DTorch.cs:line
216
at Warp.NoiseNet2DTorch.Dispose0 in
D:\Dev\warp5\WarpLib\NNModels\NoiseNet2DTorch.cs:line
227
at Warp.NoiseNet2DTorch.Finalize0 in
D:\Dev\warp5) WarpLib\NNModels\NoiseNet2DTorch.cs:line
216

Does this provide any clues?

Thanks,

Matt

[mjja...@gmail.com]

If the RTX A series are compatible with WARP, are there recommended GPUs to have for the program?

Thanks,

Matt

[mjja...@gmail.com]

Excuse me, if RTX A series are *NOT compatible with WARP...

[Arjun Bhatta]

Hi all,

I am not sure how helpful this will be, but I can at least confirm that both Warp and M (version 1.1.0) work fine on our workstation with a single RTX A5000 GPU (with CUDA 11.7 and driver version 517.40). So, there probably isn't an outright incompatibility of (all) RTX A series GPUs with warp.

Best,

Arjun

[teg...@gmail.com]

Hi Matt,

My own workstation has 2 A6000 cards, so that's as compatible as it gets. The current beta doesn't ship with a pre-trained 2D denoiser, so the result immediately after activating denoising without first training a model is going to be garbage. However, I haven't managed to make it crash this way. The latest exception message you posted looks like a multi-threading issue, which can be hard to reproduce and to debug. Does it show up consistently on your machine?

Cheers,

Dimitry

[mjja...@gmail.com]

Hi Dimitry,

This is great news. In fact, I am able to run Warp 1.1 now. Unfortunately, I have an issue with the particle picking /extraction. The picked particles look good from the micrographs (image003). The particle stacks and 2D classifications have issues though (image002 and image001). Have you seen this before with WARP 1.1? I've run WARP 0.9 with the same data and I don't have the artifacts in imported particles and 2D classification.

Thank you,

Matt