crash during subtomogram export

[alist...@gmail.com]

Hi all,

I'm having trouble reconstructing particles at a smaller pixel size

reconstruction in a 48px box at 5a/px works fine, reconstruction of the same particles in a 192px box at 1.35A crashes. I've previously extracted >200px particles for another dataset on the same system without issue. 

I tried removing all particles which display large shifts and the particles nearest the edges of the reconstruction - neither solution changes the behaviour

The relevant errors are:

System.ObjectDisposedException: Cannot access a closed pipe.

   at System.IO.__Error.PipeNotOpen()

   at System.IO.Pipes.NamedPipeServerStream.CheckConnectOperationsServer()

   at System.IO.Pipes.NamedPipeServerStream.WaitForConnection()

   at Warp.WorkerWrapper.SendCommand(NamedSerializableObject command)

   at Warp.WorkerWrapper.Dispose()

   at Warp.WorkerWrapper.Finalize()

   at System.IO.__Error.PipeNotOpen()

   at System.IO.Pipes.NamedPipeServerStream.CheckConnectOperationsServer()

   at System.IO.Pipes.NamedPipeServerStream.WaitForConnection()

   at Warp.WorkerWrapper.SendCommand(NamedSerializableObject command)

   at Warp.WorkerWrapper.Dispose()

   at Warp.WorkerWrapper.Finalize()

and for the worker

System.Exception: cudaErrorIllegalAddress

   at Warp.GPU.CheckGPUExceptions() in D:\Dev\warp2\WarpLib\GPU.cs:line 1183

   at Warp.TiltSeries.ReconstructSubtomos(ProcessingOptionsTomoSubReconstruction options, float3[] positions, float3[] angles) in D:\Dev\warp2\WarpLib\TiltSeries.cs:line 2136

   at WarpWorker.Program.Main(String[] args) in D:\Dev\warp2\WarpWorker\Program.cs:line 238

Is this likely to be some problem with my inputs? (relion star file, from a refinement) or something in the software?

Thanks in advance!

Alister

[teg...@gmail.com]

Hi Alister,

does it still crash if you check "use CPU for some steps"? How much memory does the GPU have? The latest builds limit the aliasing-free box size to 1024 px, but that might be too much for some systems. You'll probably hit it with a 2.7 A Nyquist and moderately high defocus. Is 2.7 A a reasonable expectation? I'd consider using larger pixels otherwise.

Cheers,

Dimitry

[Alister Burt]

Hi Dimitry,

That makes sense, thanks, I'll give it a go ASAP and report back - GPUs have 16GB of memory each. It's the 5 tomo Briggs HIV subset - reprocessing to prove to myself that what i'm doing should in principle work :)

Cheers and have a nice weekend,

Alister

--
You received this message because you are subscribed to a topic in the Google Groups "Warp" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp-em/nqD-8-WQkTc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/2baa9ab9-e01f-4d7d-9baf-7bb5bfde3e73n%40googlegroups.com.

[Fabian Eisenstein]

Hey all,

I just upgraded to an RTX 3090 with 24 GB of memory. When subtomogram export still kept crashing on one tomogram, I noticed it's probably because of the higher defocus. Maybe an option would be nice to lower the 1024 px aliasing-free box size limit depending on your memory. If 24 GB can't handle it, it might be a bit high.

Best,

Fabian

[Alister Burt]

Hi Fabian,

We had similar problems with some 16GB cards, checking the memory usage during export it seems like the RAM usage (I guess it’s this ‘pinned CPU memory’,  so not all of the RAN) continues to increase through the extraction - for us moving to 128GB RAM solved all of our problems.

Another thing Dimitry suggested was that only using 1gpu for extraction halves the burden on the RAM.

Good luck!

Alister

On 12 Jan 2021, at 05:49, Fabian Eisenstein <eisen...@m.u-tokyo.ac.jp> wrote:

Hey all,

You received this message because you are subscribed to the Google Groups "Warp" group.
To unsubscribe from this group and stop receiving emails from it, send an email to warp-em+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp-em/fd026b06-f614-46af-abb8-d4ea2befd51fn%40googlegroups.com.

[张皛闶]

I had encounter the same problem even I switched to a workstation with 512GB RAM and two RTX8000 (48GB dedicated Graphics Memory).

Finally i ended up with split my star into several sub stars restart warp several time to get them through.

Is it possible to release a warp version that restarts the warpworker process each time when switch to a new tomogram?

Thanks in advance,

Best,

X