e2_make3d crash after subtilt refinement step

[Mikey Grunst]

Dear EMAN2 community,

I am refining approximately 27,000 particles on a local workstation running linux. 

I previously refined ~6,000 particles on this workstation from the same dataset, and got a good result (~14A structure). I hoped after this test run to include all 27k particles. 

I am having trouble finishing the first subtilt refinement step. I get good results for the "p" iterations (aligning well), but on the "t" iterations the program keeps crashing on the e2_make3d part. I can't seem to find an error message, because the whole terminal crashes and closes. Is this a memory/performance issue?

The one error message I have ignored that keeps repeating is: 

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

One work-around I thought could be to make the new averages outside of the e2spt_refine_new.py pipeline, then restart from here. What would be the e2_make3d.py command to make the new averages from both the aliptcls3d_*.lst and the aliptcls2d_*lst? The e2spt_refine_new.py did make the new aliptcls2d_*.lst file, it just seems to crash making the new resulting average/reference.

Here is e2version output:

EMAN 2.99.55 ( GITHUB: 2024-05-15 13:18 - commit: 2aa2b781e )
Your EMAN2 is running on: Linux-6.5.0-35-generic-x86_64-with-glibc2.35 6.5.0-35-generic
Your Python version is: 3.11.9

Thank you for any advise you may have regarding this!

-Mikey

[Mikey Grunst]

Apologies, I forgot to include the last command I ran: 

/home/grunst/.conda/envs/mikeyenv/envs/eman2/bin/e2spt_refine_new.py --ptcls=spt_03/aliptcls3d_01.lst --ref=spt_03/threed_01.hdf --startres=13.0 --goldcontinue --sym=c1 --iters=p,t,d,t,p,t,d,t --keep=0.9 --tophat=local --parallel=thread:20 --threads=20 --ppid=-2. The bold "t" is as far as it went.

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/ac59f016-c3bf-4bb0-948b-e5b5593e3bcbn%40googlegroups.com.

[Muyuan Chen]

Can you provide the full command line output? I don’t think the warning you posted is the cause of the problem…

On Jun 14, 2024, at 8:02 PM, Mikey Grunst <mikey....@yale.edu> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/CACisnzrtYUoJTQzj19LNVvrP_cDehmV%3DGFFft_xWvJN2To0%2Bkg%40mail.gmail.com.

[Mikey Grunst]

Thanks Muyuan! 

My entire terminal crashes,  so I am not able to provide the exact output displayed in the terminal. Is this information stored in an output file somewhere? Sorry if I missed that.

I can also run it again and try to capture the entire terminal output using other means.

-Mikey

[Muyuan Chen]

You can always just add “> xxx” at the end of the command to redirect the command line output to a file. 

In .eman2log.txt, you may find the last command it ran before the crash, but not the error message.

Muyuan

On Jun 14, 2024, at 9:19 PM, Mikey Grunst <mikey....@yale.edu> wrote:

Thanks Muyuan! 

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/81f22aa0-11e8-4081-806a-d12bd5ff1e8cn%40googlegroups.com.

[Steve Ludtke]

If its an error you may need 2>xxx to collect the output

Sent from my iPhone

On Jun 14, 2024, at 9:36 PM, Muyuan Chen <g5v...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/4C499FAB-9FEB-4625-849F-17783A3F1608%40gmail.com.

[Mikey Grunst]

This is handy! Once I rerun and acquire the output I will post here.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/C456C96A-F242-4357-9C45-4B27AC62A12B%40gmail.com.

[Mikey Grunst]

Hi All,

I ran the refinement a few times with the 2>xxx and >xxx at the end of the command. I used cat to read the "xxx" files.

"2>xxx" repeated the same error several times as before (no other info):

/home/grunst/.conda/envs/mikeyenv/envs/eman2/lib/python3.11/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)

">xxx" only wrote:

Writing in spt_06...
Gathering metadata...
 5688/27910

Once it did actually make it through one "t" iteration (and the structure improved!), but then it crashed at the subsequent "d" iteration. I can't directly use --goldcontinue from a "t" or "d" iteration ("t" requires a "p" iteration first, and "d" requires both a "p" and a "t" first).

I noticed when running "top" during the "d" iteration that kswapdo was activated instead of seeing the individual python threads running. I suspect it may be running out of RAM. We are running a AMD 16 core CPU with 128 Gb RAM (and 2x 4090 Nvidia GPUs).

I'll transfer the data and run it on our cluster as well. Happy to try any other suggestions you may have.

Thanks again!

-Mikey