Status update May 2018: Test the first version!
12 views
Skip to first unread message
Dieter Weber
May 23, 2018, 7:57:49 AM5/23/18
to libe...@googlegroups.com
Dear all,
this month's status update is coming a week later because I wanted to
include an exciting announcement: You can test a first version of the
LiberTEM processing back-end now! A huge-shout-out to Alex for this success.
It processes pixelated STEM data faster than anything else: About 6 GB/s
on our test workstation from cached file system reads, calculating a set
of arbitrary virtual detectors and center of mass shift together in a
single pass. It is designed to scale almost linearly on a cluster, i.e.
100 nodes should give you about 600 GB/s. Alex has two sample notebooks
online that use Hyperspy as a front-end and LiberTEM for processing.
The first one reads from a local HDF5 file, that means it only works on
a single node and is easy to set up:
https://github.com/LiberTEM/LiberTEM/blob/master/examples/center_of_mass.ipynb
The second one reads from raw binary blocks distributed on a Hadoop file
system and can work on a cluster, but requires some more work to set up:
https://github.com/LiberTEM/LiberTEM/blob/master/examples/hyperspy_example.ipynb
You are cordially invited to take it for a spin. Alex
(a.cl...@fz-juelich.de) is glad to help with any questions and
support. As you can imagine, the code and documentation are still work
in progress. As always, we can use the Gitter chat
https://gitter.im/LiberTEM/Lobby and GitHub
https://github.com/LiberTEM/LiberTEM to communicate and collaborate.
Your feedback and contributions are much appreciated!
Our next steps include continuing work on the NeXus file format. We'll
have a Google hangout on the 29th of May at 13:00 Berlin time. I'll send
details in a separate message soon. Furthermore we are working towards
testing the code on a real cluster, using it for real-world
applications, and developing a user-friendly front-end.
I'm looking forward to our continued development together!
With many thanks,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
this month's status update is coming a week later because I wanted to
include an exciting announcement: You can test a first version of the
LiberTEM processing back-end now! A huge-shout-out to Alex for this success.
It processes pixelated STEM data faster than anything else: About 6 GB/s
on our test workstation from cached file system reads, calculating a set
of arbitrary virtual detectors and center of mass shift together in a
single pass. It is designed to scale almost linearly on a cluster, i.e.
100 nodes should give you about 600 GB/s. Alex has two sample notebooks
online that use Hyperspy as a front-end and LiberTEM for processing.
The first one reads from a local HDF5 file, that means it only works on
a single node and is easy to set up:
https://github.com/LiberTEM/LiberTEM/blob/master/examples/center_of_mass.ipynb
The second one reads from raw binary blocks distributed on a Hadoop file
system and can work on a cluster, but requires some more work to set up:
https://github.com/LiberTEM/LiberTEM/blob/master/examples/hyperspy_example.ipynb
You are cordially invited to take it for a spin. Alex
(a.cl...@fz-juelich.de) is glad to help with any questions and
support. As you can imagine, the code and documentation are still work
in progress. As always, we can use the Gitter chat
https://gitter.im/LiberTEM/Lobby and GitHub
https://github.com/LiberTEM/LiberTEM to communicate and collaborate.
Your feedback and contributions are much appreciated!
Our next steps include continuing work on the NeXus file format. We'll
have a Google hangout on the 29th of May at 13:00 Berlin time. I'll send
details in a separate message soon. Furthermore we are working towards
testing the code on a real cluster, using it for real-world
applications, and developing a user-friendly front-end.
I'm looking forward to our continued development together!
With many thanks,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Reply all
Reply to author
Forward
0 new messages