e2proc2d.py to create a large stack of particles

[Xinghong Dai]

Dear Steve,

I was trying to use e2proc2d.py to re-pack a list of over 160,000 particles into a .mrcs stack. The input is a LST file. Each particle is 1280x1280. The program always crashes when the output reaches 401G bytes. I monitored the progress from the beginning (looking at the increasing number of particles processed), it seems that it always crashes at 65,536 or 2^16 particles, which will give the 401GB size of the output stack. So is there a limit for the number of particles in a 2D mrcs stack in EMAN2? I have used relion_stack_create to repack the bin4 and bin2 particles and there were no problem. But relion_stack_create cannot process the unbinned particles because relion does it in the memory, and my dataset is apparently too large.

Here is the e2version.py output: 

>$ e2version.py

EMAN 2.12 (CVS 2015/10/19 09:00:04)

Your EMAN2 is running on:  Linux-2.6.32-504.12.2.el6.x86_64-x86_64-with-redhat-6.6-Carbon 2.6.32-504.12.2.el6.x86_64 x86_64

Your Python version is:  2.7.3

And here is my command and error message:

 

>$ e2proc2d.py combine_bin1.lst combine_bin1.mrcs

169279 images, processing 0-169278 stepping by 1

Traceback (most recent call last):

  File "/home/daixh/software/EMAN2/bin/e2proc2d.py", line 1006, in <module>

    main()

  File "/home/daixh/software/EMAN2/bin/e2proc2d.py", line 965, in main

    d.write_image(outfile, -1, out_type, False, None, out_mode, not_swap)

  File "/home/daixh/software/EMAN2/lib/EMAN2db.py", line 492, in db_write_image

    return self.write_image_c(fsp,*parms)

RuntimeError: ImageReadException at /build/co/eman2.daily/libEM/mrcio.cpp:119: error with 'combine_bin1.mrcs': 'invalid MRC' caught

Thank you very much for your help!

Best regards,

Xinghong Dai 

(Hong Zhou group @ UCLA)

[Ricardo Righetto]

Hi,

I have exactly the same issue. How to generate .mrcs stacks with more than 2^16 particles with EMAN2?

$ e2version.py

EMAN 2.12 (CVS 2015/10/19 09:00:04)

Your EMAN2 is running on:  Linux-3.10.0-514.el7.x86_64-x86_64-with-redhat-7.3-Maipo 3.10.0-514.el7.x86_64 x86_64

Your Python version is:  2.7.3

A workaround is to generate a .mrc stack using the --twod2threed option in e2proc2d.py, and later rename the .mrc file to .mrcs
However I'm afraid some EM packages might not like this.

Thanks!

--
Ricardo Diogo Righetto

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steve Ludtke]

Hi Ricardo.  We will check into it again. I was traveling when the last report came in, and didn't hear what the outcome was.

To unsubscribe from this group, send email to eman2+un...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.

To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

[Wei Huang]

Hi Steve,

I've come across the same issue. I think it is related to the size of the mrcs stack. I only got this error when I merge a large set of particles (280K 160 x 160 pix) into a large mrcs stack.

Thanks,

Wei

--
Ricardo Diogo Righetto

[Steve Ludtke]

Hi Wei, I realize you reported this something like 6 weeks ago. I did ask someone to look at it, but apparently fell through the cracks. Upon investigation it seems like this was supposedly fixed last November. Were you using 2.2 or a current snapshot when you had this problem?