Relion 5 to eman2 + float16 mrcs files

218 views
Skip to first unread message

Matthew Belousoff

unread,
Nov 30, 2023, 9:05:30 PM11/30/23
to EMAN2
So I'm really keen to try out the new GMM refinement, but thought to test it out on a recently processed dataset from RELION. So instead of starting fresh in EMAN I wanted to convert the run_data.star and press on directly in EMAN2.

A few details first.
e2version.py
EMAN 2.99.54 ( GITHUB: 2023-12-01 10:48 - commit: 4e38f2525 )
Your EMAN2 is running on: Linux-5.15.0-89-generic-x86_64-with-glibc2.35 5.15.0-89-generic
Your Python version is: 3.11.6


When I run e3relion_convert.py (no matter what relative path in the RELION project) I get presented with the following error after it has finished parsing the STAR file:

Parsing STAR file
WARNING: Multiple optics groups detected in star file. EMAN3 does not support this at present. The first optics group will be used for all data, which may be incorrect.
Using rlnDefocusU to group particles into 30128 groups
WARNING: Unable to group particles usefully by micrograph, collapsing to a single file. Consider rerunning with sufficiently large --dftolerance
Traceback (most recent call last):
  File "/home/mbelouso/anaconda3/envs/eman2/bin/e3relion_convert.py", line 249, in <module>
    main()
  File "/home/mbelouso/anaconda3/envs/eman2/bin/e3relion_convert.py", line 214, in main
    img=EMData("../"+imfsp,imn-1) # relion starts with #1, EMAN #0
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mbelouso/anaconda3/envs/eman2/lib/python3.11/site-packages/EMAN3.py", line 2910, in db_emd_init
    self.read_image(*parms)
  File "/home/mbelouso/anaconda3/envs/eman2/lib/python3.11/site-packages/EMAN3.py", line 2983, in db_read_image
    return self.read_image_c(fsp, *parms, **kparms)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 2: invalid start byte


The MRCS files are written in 16bit float, so I thought this might be the problem.

If I run e2iminfo.py, I get the following:
e2iminfo.py -H FoilHole_31927214_Data_30931799_30931801_20231031_224012_fractions_shiny.mrcs | grep datatype
datatype: 0


And If I try to convert a single polished stack.mrcs to .hdf using e2proc2d.py:
e2proc2d.py FoilHole_31927214_Data_30931799_30931801_20231031_224012_fractions_shiny.mrcs test.hdf
455 images, processing 0-454 stepping by 1
Traceback (most recent call last):
  File "/home/mbelouso/anaconda3/envs/eman2/bin/e2proc2d.py", line 1173, in <module>
    main()
  File "/home/mbelouso/anaconda3/envs/eman2/bin/e2proc2d.py", line 608, in main
    d.read_image(infile, i, False, None, False, img_type)
  File "/home/mbelouso/anaconda3/envs/eman2/lib/python3.11/site-packages/EMAN2.py", line 2956, in db_read_image
    return self.read_image_c(fsp, *parms, **kparms)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 0: invalid continuation byte

So I guess my question mostly is, does eman2 support the 16bit float as written out in RELION4 and RELION5? Or is there something I'm doing drastically wrong here?

thanks for your time,

cheers,

matt B




Ludtke, Steven J.

unread,
Nov 30, 2023, 9:20:34 PM11/30/23
to em...@googlegroups.com
No reason what you did shouldn't have worked. We've had several reported utf-8 decoding errors reported in the last week, all of them in different parts of the code, which seems to imply that there is some problem with the dependencies provided by Anaconda right now.   Looks like you built EMAN2 from source (which is good). Could you provide the output of 
conda list

as well as the output of
ldd $CONDA_PREFIX/lib/python3.11/site-packages/libpyEMData2.so

We have not managed to reproduce this on any of our systems yet, which is making debugging very tricky...


---
Steven Ludtke, Ph.D. <slu...@bcm.edu>                      Baylor College of Medicine
Charles C. Bell Jr., Professor of Structural Biology        Dept. of Biochemistry 
Deputy Director, Advanced Technology Cores                  and Molecular Pharmacology
Academic Director, CryoEM Core
Co-Director CIBR Center


--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/07d12ccf-15f2-4dcb-9508-efd83cecc2c0n%40googlegroups.com.

Matthew Belousoff

unread,
Nov 30, 2023, 9:26:10 PM11/30/23
to em...@googlegroups.com
Thanks for getting back so quickly! See the output below:

ldd $CONDA_PREFIX/lib/python3.11/site-packages/libpyEMData2.so
linux-vdso.so.1 (0x00007fff36be4000)
libEM2.so => /home/mbelouso/anaconda3/envs/eman2/lib/python3.11/site-packages/libEM2.so (0x00007f5885e5f000)
libboost_python311.so.1.78.0 => /home/mbelouso/anaconda3/envs/eman2/lib/libboost_python311.so.1.78.0 (0x00007f5885e22000)
libpython3.11.so.1.0 => /home/mbelouso/anaconda3/envs/eman2/lib/libpython3.11.so.1.0 (0x00007f588585f000)
libstdc++.so.6 => /home/mbelouso/anaconda3/envs/eman2/lib/libstdc++.so.6 (0x00007f588567c000)
libgcc_s.so.1 => /home/mbelouso/anaconda3/envs/eman2/lib/libgcc_s.so.1 (0x00007f5885661000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5885420000)
libgsl.so.25 => /home/mbelouso/anaconda3/envs/eman2/lib/libgsl.so.25 (0x00007f5885119000)
libopenblas.so.0 => /home/mbelouso/anaconda3/envs/eman2/lib/libopenblas.so.0 (0x00007f5882f50000)
libtiff.so.6 => /home/mbelouso/anaconda3/envs/eman2/lib/libtiff.so.6 (0x00007f5882ec2000)
libjpeg.so.8 => /home/mbelouso/anaconda3/envs/eman2/lib/libjpeg.so.8 (0x00007f5882e18000)
libpng16.so.16 => /home/mbelouso/anaconda3/envs/eman2/lib/libpng16.so.16 (0x00007f5882ddc000)
libhdf5.so.310 => /home/mbelouso/anaconda3/envs/eman2/lib/libhdf5.so.310 (0x00007f58829ae000)
libfftw3f.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/libfftw3f.so.3 (0x00007f588277c000)
libfftw3.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/libfftw3.so.3 (0x00007f5882555000)
libfftw3f_threads.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/libfftw3f_threads.so.3 (0x00007f588254c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5882465000)
libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x00007f5882368000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f5882361000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f588235c000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f5882357000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5882352000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5886786000)
libgfortran.so.5 => /home/mbelouso/anaconda3/envs/eman2/lib/./libgfortran.so.5 (0x00007f58821a7000)
libwebp.so.7 => /home/mbelouso/anaconda3/envs/eman2/lib/./libwebp.so.7 (0x00007f5882105000)
libzstd.so.1 => /home/mbelouso/anaconda3/envs/eman2/lib/./libzstd.so.1 (0x00007f5881ff1000)
liblzma.so.5 => /home/mbelouso/anaconda3/envs/eman2/lib/./liblzma.so.5 (0x00007f5881fc8000)
libLerc.so.4 => /home/mbelouso/anaconda3/envs/eman2/lib/./libLerc.so.4 (0x00007f5881f05000)
libdeflate.so.0 => /home/mbelouso/anaconda3/envs/eman2/lib/./libdeflate.so.0 (0x00007f5881ef0000)
libz.so.1 => /home/mbelouso/anaconda3/envs/eman2/lib/./libz.so.1 (0x00007f5881ed5000)
libcrypto.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/./libcrypto.so.3 (0x00007f58819c5000)
libcurl.so.4 => /home/mbelouso/anaconda3/envs/eman2/lib/./libcurl.so.4 (0x00007f588190b000)
libsz.so.2 => /home/mbelouso/anaconda3/envs/eman2/lib/./libsz.so.2 (0x00007f58818fe000)
libquadmath.so.0 => /home/mbelouso/anaconda3/envs/eman2/lib/././libquadmath.so.0 (0x00007f58818c5000)
libsharpyuv.so.0 => /home/mbelouso/anaconda3/envs/eman2/lib/././libsharpyuv.so.0 (0x00007f58818bb000)
libnghttp2.so.14 => /home/mbelouso/anaconda3/envs/eman2/lib/././libnghttp2.so.14 (0x00007f588188d000)
libssh2.so.1 => /home/mbelouso/anaconda3/envs/eman2/lib/././libssh2.so.1 (0x00007f5881848000)
libssl.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/././libssl.so.3 (0x00007f58817a6000)
libgssapi_krb5.so.2 => /home/mbelouso/anaconda3/envs/eman2/lib/././libgssapi_krb5.so.2 (0x00007f5881753000)
libkrb5.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/./././libkrb5.so.3 (0x00007f588167d000)
libk5crypto.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/./././libk5crypto.so.3 (0x00007f5881663000)
libcom_err.so.3 => /home/mbelouso/anaconda3/envs/eman2/lib/./././libcom_err.so.3 (0x00007f588165d000)
libkrb5support.so.0 => /home/mbelouso/anaconda3/envs/eman2/lib/./././libkrb5support.so.0 (0x00007f588164f000)
libkeyutils.so.1 => /home/mbelouso/anaconda3/envs/eman2/lib/./././libkeyutils.so.1 (0x00007f5881648000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f5881634000)




conda list
# packages in environment at /home/mbelouso/anaconda3/envs/eman2:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.0.0              pyhd8ed1ab_0    conda-forge
aiohttp                   3.9.1           py311h459d7ec_0    conda-forge
aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
alsa-lib                  1.2.10               hd590300_0    conda-forge
anyio                     4.1.0              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py311h459d7ec_4    conda-forge
arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3              pyhd8ed1ab_0    conda-forge
async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
attr                      2.5.1                h166bdaf_1    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
babel                     2.13.1             pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
biopython                 1.81            py311h459d7ec_1    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
boost                     1.78.0          py311h92ebd52_5    conda-forge
boost-cpp                 1.78.0               h2c5509c_4    conda-forge
brotli                    1.1.0                hd590300_1    conda-forge
brotli-bin                1.1.0                hd590300_1    conda-forge
brotli-python             1.1.0           py311hb755f60_1    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.23.0               hd590300_0    conda-forge
ca-certificates           2023.11.17           hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.2              pyhd8ed1ab_0    conda-forge
cairo                     1.18.0               h3faef2a_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311hb3a22ac_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cmake                     3.27.9               hcfe8598_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
configparser              5.3.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.0           py311h9547e67_0    conda-forge
cryptography              41.0.5          py311h63ff55d_0    conda-forge
cuda-version              11.8                 h70ddcb2_2    conda-forge
cudatoolkit               11.8.0              h4ba93d1_12    conda-forge
cudnn                     8.8.0.121            hcdd5f01_4    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
debugpy                   1.8.0           py311hb755f60_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_0    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
fftw                      3.3.10          mpi_openmpi_h4a81ba8_8    conda-forge
flatbuffers               23.5.26              h59595ed_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 h77eed37_1    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
fonttools                 4.45.1          py311h459d7ec_0    conda-forge
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
frozenlist                1.4.0           py311h459d7ec_1    conda-forge
ftgl                      2.4.0                h6cc6478_2    cryoem
future                    0.18.3             pyhd8ed1ab_0    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glib                      2.78.1               hfc55251_1    conda-forge
glib-tools                2.78.1               hfc55251_1    conda-forge
google-auth               2.24.0             pyhca7485f_0    conda-forge
google-auth-oauthlib      1.0.0              pyhd8ed1ab_1    conda-forge
google-pasta              0.2.0              pyh8c360ce_0    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
grpcio                    1.54.3          py311hcafe171_0    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
gst-plugins-base          1.22.7               h8e1006c_0    conda-forge
gstreamer                 1.22.7               h98fc4e7_0    conda-forge
h5py                      3.10.0          nompi_py311h3839ddf_100    conda-forge
harfbuzz                  8.3.0                h3d44ed6_0    conda-forge
hdf5                      1.14.2          nompi_h4f84152_100    conda-forge
icu                       73.2                 h59595ed_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
ipykernel                 6.26.0             pyhf8b6a83_0    conda-forge
ipython                   8.18.1             pyh31011fe_1    conda-forge
ipywidgets                8.1.1              pyhd8ed1ab_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jax                       0.4.19             pyhd8ed1ab_0    conda-forge
jaxlib                    0.4.14          cuda112py311hf2474b9_201    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpointer               2.4             py311h38be061_3    conda-forge
jsonschema                4.20.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.11.2          pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.20.0             pyhd8ed1ab_0    conda-forge
jupyter-lsp               2.2.1              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.5.0           py311h38be061_0    conda-forge
jupyter_events            0.9.0              pyhd8ed1ab_0    conda-forge
jupyter_server            2.11.1             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyterlab                4.0.9              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.25.2             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.9              pyhd8ed1ab_0    conda-forge
keras                     2.12.0             pyhd8ed1ab_0    conda-forge
keras-preprocessing       1.1.2              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py311h9547e67_1    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lcms2                     2.15                 haa2dc70_1    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libabseil                 20230125.3      cxx17_h59595ed_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libblas                   3.9.0           20_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hd590300_1    conda-forge
libbrotlidec              1.1.0                hd590300_1    conda-forge
libbrotlienc              1.1.0                hd590300_1    conda-forge
libcap                    2.69                 h0f662aa_0    conda-forge
libcblas                  3.9.0           20_linux64_openblas    conda-forge
libclang                  15.0.7          default_h7634d5b_3    conda-forge
libclang13                15.0.7          default_h9986a30_3    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.4.0                hca28451_0    conda-forge
libdeflate                1.18                 h0b41bf4_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.2.0               h807b86a_3    conda-forge
libgcrypt                 1.10.3               hd590300_0    conda-forge
libgfortran-ng            13.2.0               h69a702a_3    conda-forge
libgfortran5              13.2.0               ha4646dd_3    conda-forge
libglib                   2.78.1               h783c2da_1    conda-forge
libglu                    9.0.0             hac7e632_1003    conda-forge
libgomp                   13.2.0               h807b86a_3    conda-forge
libgpg-error              1.47                 h71f35ed_0    conda-forge
libgrpc                   1.54.3               hb20ce57_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libjpeg-turbo             2.1.5.1              hd590300_1    conda-forge
liblapack                 3.9.0           20_linux64_openblas    conda-forge
libllvm15                 15.0.7               h5cf9203_3    conda-forge
libnghttp2                1.58.0               h47da74e_0    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.25          pthreads_h413a1c8_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     15.4                 hfc447b1_2    conda-forge
libprotobuf               3.21.12              hfc55251_2    conda-forge
libsndfile                1.2.2                hc60ed4a_1    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.44.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_3    conda-forge
libsystemd0               254                  h3516f8a_0    conda-forge
libtiff                   4.5.1                h8b53f26_1    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libuv                     1.46.0               hd590300_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.6.0                h5d7e998_0    conda-forge
libxml2                   2.11.6               h232c23b_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown                  3.5.1              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.3           py311h459d7ec_1    conda-forge
matplotlib                3.8.2           py311h38be061_0    conda-forge
matplotlib-base           3.8.2           py311h54ef318_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
ml_dtypes                 0.3.1           py311h320fe9a_2    conda-forge
mpg123                    1.32.3               h59595ed_0    conda-forge
mpi                       1.0                     openmpi    conda-forge
multidict                 6.0.4           py311h459d7ec_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
mysql-common              8.0.33               hf1915f5_6    conda-forge
mysql-libs                8.0.33               hca2cd23_6    conda-forge
nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.11.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
nccl                      2.19.4.1             h6103f9b_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
nest-asyncio              1.5.8              pyhd8ed1ab_0    conda-forge
nodejs                    20.9.0               hb753e55_0    conda-forge
nose                      1.3.7                   py_1006    conda-forge
notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.95                 h1d7d5a4_0    conda-forge
numpy                     1.23.5          py311h7d28db0_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openmpi                   4.1.6              hc5af2df_101    conda-forge
openssl                   3.1.4                hd590300_0    conda-forge
opt-einsum                3.3.0                hd8ed1ab_2    conda-forge
opt_einsum                3.3.0              pyhc1e730c_2    conda-forge
overrides                 7.4.0              pyhd8ed1ab_0    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandas                    2.1.3           py311h320fe9a_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.42                hcad00b1_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.0.0          py311h0b84326_0    conda-forge
pip                       23.3.1             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h59595ed_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              3.5.1              pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
prometheus_client         0.19.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.41             pyha770c72_0    conda-forge
protobuf                  4.21.12         py311hcafe171_0    conda-forge
psutil                    5.9.5           py311h459d7ec_1    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pulseaudio-client         16.1                 hb77b528_5    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.5.1              pyhd8ed1ab_0    conda-forge
pyasn1-modules            0.3.0              pyhd8ed1ab_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pydusa                    3.1             np123py311h5c49d3a_7    cryoem
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_0    conda-forge
pyopengl                  3.1.5                      py_0    cryoem
pyopenssl                 23.3.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.9          py311hf0fb5b6_5    conda-forge
pyqt5-sip                 12.12.2         py311hb755f60_5    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.6          hab00c5b_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.19.0             pyhd8ed1ab_0    conda-forge
python-flatbuffers        23.5.26            pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
pyu2f                     0.1.5              pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1           py311h459d7ec_1    conda-forge
pyzmq                     25.1.1          py311h34ded2d_2    conda-forge
qt-main                   5.15.8              hc47bfe8_16    conda-forge
re2                       2023.03.02           h8c504da_0    conda-forge
readline                  8.2                  h8228510_1    conda-forge
referencing               0.31.1             pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rhash                     1.4.4                hd590300_0    conda-forge
rpds-py                   0.13.2          py311h46250e7_0    conda-forge
rsa                       4.9                pyhd8ed1ab_0    conda-forge
scikit-learn              1.3.2           py311hc009520_1    conda-forge
scipy                     1.11.4          py311h64a7726_0    conda-forge
send2trash                1.8.2              pyh41d4057_0    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
sip                       6.7.12          py311hb755f60_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
sphire                    1.4.3                    pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tensorboard               2.12.3             pyhd8ed1ab_0    conda-forge
tensorboard-data-server   0.7.0           py311h63ff55d_1    conda-forge
tensorflow                2.12.1          cuda112py311h63ba870_1    conda-forge
tensorflow-base           2.12.1          cuda112py311hb4f7669_1    conda-forge
tensorflow-estimator      2.12.1          cuda112py311h9aaad64_1    conda-forge
termcolor                 2.3.0              pyhd8ed1ab_0    conda-forge
terminado                 0.18.0             pyh0d859eb_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.3.3           py311h459d7ec_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.0             pyhd8ed1ab_0    conda-forge
types-python-dateutil     2.8.19.14          pyhd8ed1ab_0    conda-forge
typing-extensions         4.5.0                hd8ed1ab_0    conda-forge
typing_extensions         4.5.0              pyha770c72_0    conda-forge
typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.12             pyhd8ed1ab_0    conda-forge
webcolors                 1.13               pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.6.4              pyhd8ed1ab_0    conda-forge
werkzeug                  3.0.1              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.9              pyhd8ed1ab_0    conda-forge
wrapt                     1.16.0          py311h459d7ec_0    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.40                 hd590300_0    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.7                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xf86vidmodeproto     2.3.1             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.9.3           py311h459d7ec_0    conda-forge
zeromq                    4.3.5                h59595ed_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

You received this message because you are subscribed to a topic in the Google Groups "EMAN2" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/eman2/HyColdT0tXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/A55322F2-4E8C-46D8-8E31-B0F9D52B9874%40bcm.edu.


--
Dr. Matthew Belousoff
Drug Discovery Biology, Monash Institute of Pharmaceutical Sciences &

Centre for Cryo-electron Microscopy of Membrane Proteins, Monash Institute of Pharmaceutical Sciences, Monash University, Parkville, 3052

AUSTRALIA

Twitter: @bluesocks81

Mail: c/-
Monash University
Department of Microbiology
23 Innovation Walk,
Clayton Campus
Melbourne 3800

shadow walker

unread,
Nov 30, 2023, 10:16:09 PM11/30/23
to em...@googlegroups.com
Can you write the list of packages to a file, so it will be easier to recreate your environment?


conda list --explicit > environment.txt


Matthew Belousoff

unread,
Nov 30, 2023, 10:30:15 PM11/30/23
to em...@googlegroups.com
Its attached mate.

Thanks!

environment.txt

shadow walker

unread,
Dec 3, 2023, 1:53:40 PM12/3/23
to em...@googlegroups.com
Thanks for the environment file. If you don't mind, can you also provide a minimal star file and the corresponding command that fails? Or, a file that fails with e2proc2d like in your original post along with the command? If you have updated your environment recently, the output of 'conda list -r' when your environment causing problems is activated. And, finally, for now, what OS distro are you on?



--
shadow_walker

Matthew Belousoff

unread,
Dec 5, 2023, 5:26:40 PM12/5/23
to EMAN2
Hey mate,

Sorry for the delay. The files are attached and the distro is linux mint 21.2, kernel 5.15.0-89-generic. I may have to transfer the particle stack another way? Its 21M and a too big to share here. Happy to send it some other way though.

Thanks so much,

matt B



subset_forEMANTroubleshoot.star

Steve Ludtke

unread,
Dec 7, 2023, 10:43:43 AM12/7/23
to em...@googlegroups.com
Ok, we have figured out what’s going on, at least why unicode errors are getting produced. When an exception is generated in C++ code, that exception is supposed to be passed to python and displayed as an error message. Sometime over the last few years, this mechanism in Boost Python (the library we use for Python <-> C++) seems to have broken, and it started printing random gibberish on the screen instead of a proper error message. It appears that sometime in Python 3.10 or 3.11, they further changed the mechanism so the string coming in from C++ was interpreted as Unicode instead of ASCII. So it’s still getting gibberish, but now it’s trying to treat it as unicode, producing another error.

SO, what that means, if you didn’t follow the above, is that any time you see one of these Unicode errors, that means that some exception/error occurred in the C++ code. If it prints a Python traceback, the point at which the exception occurred should be correct, but instead of printing a proper error message, it produces this unicode error instead. So rather than focus on the Unicode issue, the correct response is to look for what caused the underlying error.

Float16 is not a formal part of the MRC file specification (PMC4642651), so whomever decided to do this has yet again mucked up an already abused file format. While FLOAT16s can be extremely computationally efficient on GPUs, they have insufficient precision for anything other than storage of raw image data, which could be stored MUCH more efficiently in other ways (PMC9645247). 

Anyway, if you can point at wherever FLOAT16 first emerged in publication (I missed it whereever it is), we should be able to add support for it pretty trivially.


--
--
----------------------------------------------------------------------------------------------
You received this message because you are subscribed to the Google
Groups "EMAN2" group.
To post to this group, send email to em...@googlegroups.com
To unsubscribe from this group, send email to eman2+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/eman2

---
You received this message because you are subscribed to the Google Groups "EMAN2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to eman2+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/eman2/0d921117-29d6-4124-8cee-1352ed45708fn%40googlegroups.com.
<subset_forEMANTroubleshoot.star>

Reply all
Reply to author
Forward
0 new messages