Error importing SER files into EMAN2.1

[nancy meyer]

The raw images I am trying to import are SER files generated by FEI's TIA software and I expected them to import as my MRCs had previously, but I get the following errors upon launching Import Micrographs. Below that, I pasted in the separate error I receive when I try Evaluate and Import Micrographs.

My version is:
EMAN 2.1 beta4 (CVS 2014/10/22 19:23:02)
Your EMAN2 is running on:  Linux-2.6.32-131.17.1.el6.x86_64-x86_64-with-redhat-6.6-Santiago 2.6.32-131.17.1.el6.x86_64 x86_64
Your Python version is:  2.7.7

# Attempting: Import Micrographs -> Browse -> choose files -> Launch
# Files to import are of format "CCD_Acquire_00xx_1.ser" generated by FEI TIA software.
# Files are in subdirectory within working directory.
# Able to open these SER files in ImageJ with TIA Reader plugin.
# This is what I get...

NOT Writing notes, ppid=-2
Traceback (most recent call last):
  File "/progs/EMAN2-141028/bin/e2proc2d.py", line 905, in <module>
    main()
  File "/progs/EMAN2-141028/bin/e2proc2d.py", line 239, in main
    [nxinp, nyinp, nzinp] = gimme_image_dimensions3D(infile)
  File "/progs/EMAN2-141028/lib/EMAN2.py", line 1017, in gimme_image_dimensions3D
    e.read_image(imagefilename,0, read_header_only)
  File "/progs/EMAN2-141028/lib/EMAN2db.py", line 428, in db_read_image
    return self.read_image_c(fsp,*parms, **kparms)
RuntimeError: ImageReadException at /progs/EMAN2-141028/src/eman2/libEM/serio.cpp:134: error with 'raw_data/CCD_Acquire_0034_1.ser': 'SER header' caught

# This last line above seems to indicate maybe byte_order, ID, or version value is off in SER header? (?)

Traceback (most recent call last):
  File "/progs/EMAN2-141028/bin/e2rawdata.py", line 181, in <module>
    main()
  File "/progs/EMAN2-141028/bin/e2rawdata.py", line 93, in main
    d=EMData(output,0)
  File "/progs/EMAN2-141028/lib/EMAN2db.py", line 374, in db_emd_init
    else : self.__initc(*parms)
RuntimeError

# If instead, I attempt: Evaluate and Import Micrographs, importing the same SER files
# This is what I get...

Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Error: all of the first 10 images are missing !
Traceback (most recent call last):
  File "/progs/EMAN2-141028/bin/e2projectmanager.py", line 518, in _on_cmd_launch
  File "/progs/EMAN2-141028/bin/e2projectmanager.py", line 1639, in getCommand
  File "/progs/EMAN2-141028/bin/e2projectmanager.py", line 1888, in getCommand
  File "/progs/EMAN2-141028/lib/EMAN2jsondb.py", line 798, in setval
  File "/progs/EMAN2-141028/lib/EMAN2jsondb.py", line 661, in sync
IOError: [Errno 24] Too many open files: '/data/meyerna/EMAN2/reducedlig_2014-12-03_grid2/info/pm/e2evalimage.py.json'

Sorry this is a bit lengthy, but wondering if anything stands out as obviously wrong with what I'm doing. Should note that I get errors upon trying to convert these SER files to MRC format using e2proc2d.py. Thanks again for any input!

Nancy

[Steven Ludtke]

Hi Nancy. That message indicates an invalid header in the SER file. Specifically it implies that the SER version number has changed. Normally companies that use a serial number in their headers only change it when something about the file format itself has changed, meaning the reader would need to be updated. Could you run

od -cx file.ser |head -20

then paste the output into a reply to this message?  Obviously you should replace file.ser with the name of one of your files. 

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[nancy meyer]

Hi Steve. Many thanks for the reply. So Byte_Order (4949) and Series_ID (0197) look right, but Series_Version is 0220 in the output below (second line), while it's specified as 0210 in the code that reads in the SER files. Could I just change it to like "greater than" 0210 in the code and solve the problem? (Wishful thinking?) Thank you again!

[meyerna@dl580g7 OHSU_Krios_12-03-14]$ od -cx CCD_Acquire_0088_1.ser |head -20
0000000   I   I 227 001     002   "   A  \0  \0   R   A  \0  \0 001  \0
           4949    0197    0220    4122    0000    4152    0000    0001
0000020  \0  \0 001  \0  \0  \0   H  \0  \0  \0  \0  \0  \0  \0 001  \0
           0000    0001    0000    0048    0000    0000    0000    0001
0000040  \0  \0 001  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
           0000    0001    0000    0000    0000    0000    0000    0000
0000060  \0  \0  \0  \0 360   ?  \0  \0  \0  \0 006  \0  \0  \0   N   u
           0000    0000    3ff0    0000    0000    0006    0000    754e
0000100   m   b   e   r  \0  \0  \0  \0   X  \0  \0  \0  \0  \0  \0  \0
           626d    7265    0000    0000    0058    0000    0000    0000
0000120 212  \0  \0 004  \0  \0  \0  \0   P 320 347   N 001 001 223 276
           008a    0400    0000    0000    d050    4ee7    0101    be93
0000140  \0 320 347   N 001 001 343   =  \0  \0  \0  \0   P 320 347   N
           d000    4ee7    0101    3de3    0000    0000    d050    4ee7
0000160 001 001 223 276  \0 320 347   N 001 001 343   =  \0  \0  \0  \0
           0101    be93    d000    4ee7    0101    3de3    0000    0000
0000200 006  \0  \0 020  \0  \0  \0 020  \0  \0 300 016  \0  \0 300 016
           0006    1000    0000    1000    0000    0ec0    0000    0ec0
0000220  \0  \0 022 017  \0  \0   : 020  \0  \0   0 020  \0  \0 242 020
           0000    0f12    0000    103a    0000    1030    0000    10a2

[nancy meyer]

Ok, so naively just changed version from 0210 to 0220 in the serio.cpp, rebuilt EMAN2 and tried to convert a SER file to MRC with e2proc2d.py just to test, and this is what we got:

[meyerna@dl580g7 OHSU_Krios_12-03-14]$ e2proc2d.py CCD_Acquire_0088_1.ser CCD_Acquire_0088_1.mrc
python: /progs/EMAN2-141028/src/eman2/libEM/serio.cpp:163: virtual int EMAN::SerIO::read_header(EMAN::Dict&, int, const EMAN::Region*, bool): Assertion `pos == (int)dict["SER.OffsetArrayOffset"]' failed.
Aborted (core dumped)
[meyerna@dl580g7 OHSU_Krios_12-03-14]$

So not just serial number issue... ?

[Steven Ludtke]

Sounds like it may be a bigger change, but thanks for testing!   I will investigate with our original contact for the SER format and see if we can deal with the new version easily.

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

[Steven Ludtke]

Hi Nancy. Could you provide a little more information on where and with what software and version these images were collected. I traced our SER specifications back to their source, but that’s a dead-end for new versions. I have some other contacts I might use, but I’d like as many details as possible before going down the rabbit hole.

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

[nancy meyer]

We're collecting at Oregon Health and Science University (Chapman lab) on a Titan Krios with Falcon II detector. Was unfortunately doing manual collection using FEI's TIA software (instead of EPU), hence the output as .emi and .ser files. I dont have access at the moment, but am pretty sure the TIA version is 4.3.7 (though may have to verify this). Hope this slightly helpful.

[Steven Ludtke]

Hi Nancy. If you could get the exact software version it would be quite useful.  Thx

----------------------------------------------------------------------------

Steven Ludtke, Ph.D.

Professor, Dept of Biochemistry and Mol. Biol.         (www.bcm.edu/biochem)

Co-Director National Center For Macromolecular Imaging        (ncmi.bcm.edu)

Co-Director CIBR Center                          (www.bcm.edu/research/cibr)

Baylor College of Medicine                             

slu...@bcm.edu

On Dec 11, 2014, at 6:10 PM, nancy meyer <ruby...@hotmail.com> wrote:

We're collecting at Oregon Health and Science University (Chapman lab) on a Titan Krios with Falcon II detector. Was unfortunately doing manual collection using FEI's TIA software (instead of EPU), hence the output as .emi and .ser files. I dont have access at the moment, but am pretty sure the TIA version is 4.3.7 (though may have to verify this). Hope this slightly helpful.

[nancy meyer]

FEI TEM Imaging and Analysis (TIA) version 4.7 SP3, build 1489

[Steven Ludtke]

Hi Nancy. Thanks. I got the updated info from a contact I have at FEI. Would it be possible to send me a representative image in this new format? I don't care what's actually in the image, I just need a sample for debugging. While the header appears to be clear, there are a few oddities in the existing code, so I'd like to test it before posting the update...

> On Dec 11, 2014, at 7:03 PM, nancy meyer <ruby...@hotmail.com> wrote:
>
> FEI TEM Imaging and Analysis (TIA) version 4.7 SP3, build 1489
>

> --
> --
> ----------------------------------------------------------------------------------------------
> You received this message because you are subscribed to the Google
> Groups "EMAN2" group.
> To post to this group, send email to em...@googlegroups.com
> To unsubscribe from this group, send email to eman2+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/eman2
>
> ---
> You received this message because you are subscribed to the Google Groups "EMAN2" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[nancy meyer]

Had problems uploading the example SER file to the group. Put it on my Google drive and you should find the link to it in your email (bcm.edu). Thanks!

[Steven Ludtke]

I’m pleased to announce that the new SER variant should be properly supported by the nightly snapshot releases of EMAN2.1 (as of the 12/13/14 snapshot release). I’ve tested it on the one sample file I got without problems. If anyone has difficulties with it, please report it here.

(Got it. Thanks. Not good to post large binaries to the group anyway.)

Nancy, since you’re compiling from source, if you want the patched files rather than waiting until tomorrow, just let me know and I can send them to you.

On Dec 12, 2014, at 1:07 PM, nancy meyer <ruby...@hotmail.com> wrote:

Had problems uploading the example SER file to the group. Put it on my Google drive and you should find the link to it in your email (bcm.edu). Thanks!

[nancy meyer]

Yes, Steve, that would be great (i.e. patched files). (Not to be impatient!) Much obliged!

[Steven Ludtke]

Attached replacements for serio.h and serio.cpp

[Mario Borgnia]

Hi Steve,

My bug attractant keeps working flawlessly, I think I ran into this one:

> e2proc2d.py EtoF_1_E_1.ser EtoF_1_E_1.mrc

Traceback (most recent call last):

  File "/usr/local/EMAN2/bin/e2proc2d.py", line 953, in <module>

    main()

  File "/usr/local/EMAN2/bin/e2proc2d.py", line 264, in main

    [nxinp, nyinp, nzinp] = gimme_image_dimensions3D(infile)

  File "/usr/local/EMAN2/lib/EMAN2.py", line 1017, in gimme_image_dimensions3D

    e.read_image(imagefilename,0, read_header_only)

  File "/usr/local/EMAN2/lib/EMAN2db.py", line 428, in db_read_image

    return self.read_image_c(fsp,*parms, **kparms)

RuntimeError: ImageReadException at /build/co/eman2.daily/libEM/serio.cpp:182: error with 'EtoF_1_E_1.ser': 'SER header' caught

> e2proc2d.py --version

EMAN 2.11

> ls /usr/local/EMAN2/build*

/usr/local/EMAN2/build_date.2015-06-12

> od -cx EtoF_1_E_1.ser |head -20

0000000   I   I 227 001 020 002   "   A  \0  \0   R   A  \0  \0 001  \0

           4949    0197    0210    4122    0000    4152    0000    0001

0000020  \0  \0 001  \0  \0  \0   D  \0  \0  \0 001  \0  \0  \0 001  \0

           0000    0001    0000    0044    0000    0001    0000    0001

0000040  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0

           0000    0000    0000    0000    0000    0000    0000    0000

0000060 360   ?  \0  \0  \0  \0 006  \0  \0  \0   N   u   m   b   e   r

           3ff0    0000    0000    0006    0000    754e    626d    7265

0000100  \0  \0  \0  \0   L  \0  \0  \0   ~  \0  \0 004 232   s   Z 035

           0000    0000    004c    0000    007e    0400    739a    1d5a

0000120   x 207 233 276  \0   p   Z 035   x 207 353   =  \0  \0  \0  \0

           8778    be9b    7000    1d5a    8778    3deb    0000    0000

0000140 232   s   Z 035   x 207 233 276  \0   p   Z 035   x 207 353   =

           739a    1d5a    8778    be9b    7000    1d5a    8778    3deb

0000160  \0  \0  \0  \0 006  \0  \0 020  \0  \0  \0 020  \0  \0   x 003

           0000    0000    0006    1000    0000    1000    0000    0378

0000200  \0  \0   l 003  \0  \0   p 003  \0  \0   < 003  \0  \0   : 003

           0000    036c    0000    0370    0000    033c    0000    033a

0000220  \0  \0 223 003  \0  \0 312 003  \0  \0 230 003  \0  \0 177 003

           0000    0393    0000    03ca    0000    0398    0000    037f