LiberTEM version 0.2.1 released; status update
11 views
Skip to first unread message
Dieter Weber
Oct 8, 2019, 8:09:06 AM10/8/19
to libe...@googlegroups.com
Dear subscribers,
with great happiness and pride I can announce the release of LiberTEM
version 0.2.1 after almost a year of intense development work. This
version brings a number of major new features.
User-defined functions
======================
LiberTEM 0.2 offers a new API to define a wide range of user-defined
reduction functions (UDFs) on distributed data. The interface and
implementation offers a number of unique features:
* Reductions are defined as functions that are executed on subsets of
the data. That means they are equally suitable for distributed
computing, for interactive display of results from a progressing
calculation, and for handling live data¹.
* Interfaces adapted to both simple and complex use cases: From a simple
map() functionality to complex multi-stage reductions.
* Rich options to define input and output data for the reduction
functions, which helps to implement non-trivial operations efficiently
within a single pass over the input data.
* Composition and extension through object oriented programming
* Interfaces that allow highly efficient processing: locality of
reference, cache efficiency, memory handling
Introduction: https://libertem.github.io/LiberTEM/udf.html
Advanced features: https://libertem.github.io/LiberTEM/udf/advanced.html
A big shoutout to Alex (@sk1p) who developed it! 🏆
¹User-defined functions will work on live data without modification as
soon as LiberTEM implements back-end support for live data, expected in
2020.
Support for 4D STEM applications
================================
In parallel to the UDF interface, we have implemented a number of
applications that make use of the new facilities:
* Correlation-based peak finding and refinement for CBED (credit: Karina
Ruzaeva @kruzaeva)
* Strain mapping
* Clustering
* Fluctuation EM
* Radial Fourier Series (advanced Fluctuation EM)
More details and examples:
https://libertem.github.io/LiberTEM/applications.html
Extended documentation
======================
We have greatly improved the coverage of our documentation:
https://libertem.github.io/LiberTEM/index.html#documentation
Fully automated release pipeline
================================
Alex (@sk1p) invested a great deal of effort into fully automating our
release process. From now on, we will be able to release more often,
including service releases. 🚀
Basic dask.distributed array integration
========================================
LiberTEM can now generate efficient dask.distributed arrays from all
supported dataset types. That means it should be possible to use our
high-performance file readers in applications outside of LiberTEM.
File formats
============
Support for various file formats has improved. More details:
https://libertem.github.io/LiberTEM/formats.html
***
Ongoing and future activities
=============================
Change management
-----------------
With this release we will start tracking and managing changes
systematically. In particular, we have taken a considerable amount of
time and testing to stabilize the UDF interface. It should now be ready
for widespread use.
PCA
---
Strategy and prototype implementation for efficient distributed
calculation of PCA and similar decompositions by Jaeweon (@jaewshin), a
Google Summer of Code student: https://github.com/LiberTEM/LiberTEM/pull/347
To be further tested and validated before merging into the production
branch. Please contact us if you are interested in such a capability!
Production-ready cluster
------------------------
We are planning to fulfill a number of requirements that are relevant
for using LiberTEM on a cluster in a production environment:
* Authentication and authorization
* HTTPS
* Managing Python environments to ensure that client and back-end match
* Caching
* Data management
The solution will likely be a combination of new LiberTEM features and
matching configuration and deployment of third-party tools.
Full dask.distributed array integration
---------------------------------------
We are planning to allow creating a dataset from a dask.distributed
array and to create dask.distributed arrays using UDFs. That should
enable seamless integration between Hyperspy and LiberTEM, which is
particularly important to merge functionality that is present in both
pyXem and LiberTEM at the moment. Please contact us if you are interested!
Ptychography
------------
We are part of the Helmholtz Pilot Project "Ptychography 4.0" to develop
new approaches for ptychography. Our main contribution will be efficient
distribution and handling live data. As a result, LiberTEM will most
likely support ptychography in the future.
Live data processing
--------------------
As a next major feature we are now targeting processing of live data.
This will likely be the dominating task for 2020.
Miscellaneous
=============
Congratulations to Karina for finishing her Master's degree and getting
a PhD position at RWTH Aachen!
We participated in the Google Summer of Code 2019 under the umbrella of
the Python Software Foundation (PSF). Many thanks to Karina, Jaeweon,
the PSF team and Google for the support!
The ERC Proof of Concept VIDEO that funded the initial development of
LiberTEM ended successfully in June. Follow-up funding for LiberTEM is
secured.
We are hiring!
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-188-EN-ER-C%201.html
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-191-EN-ER-C%201.html
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-190-EN-ER-C%201.html
With best regards,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
with great happiness and pride I can announce the release of LiberTEM
version 0.2.1 after almost a year of intense development work. This
version brings a number of major new features.
User-defined functions
======================
LiberTEM 0.2 offers a new API to define a wide range of user-defined
reduction functions (UDFs) on distributed data. The interface and
implementation offers a number of unique features:
* Reductions are defined as functions that are executed on subsets of
the data. That means they are equally suitable for distributed
computing, for interactive display of results from a progressing
calculation, and for handling live data¹.
* Interfaces adapted to both simple and complex use cases: From a simple
map() functionality to complex multi-stage reductions.
* Rich options to define input and output data for the reduction
functions, which helps to implement non-trivial operations efficiently
within a single pass over the input data.
* Composition and extension through object oriented programming
* Interfaces that allow highly efficient processing: locality of
reference, cache efficiency, memory handling
Introduction: https://libertem.github.io/LiberTEM/udf.html
Advanced features: https://libertem.github.io/LiberTEM/udf/advanced.html
A big shoutout to Alex (@sk1p) who developed it! 🏆
¹User-defined functions will work on live data without modification as
soon as LiberTEM implements back-end support for live data, expected in
2020.
Support for 4D STEM applications
================================
In parallel to the UDF interface, we have implemented a number of
applications that make use of the new facilities:
* Correlation-based peak finding and refinement for CBED (credit: Karina
Ruzaeva @kruzaeva)
* Strain mapping
* Clustering
* Fluctuation EM
* Radial Fourier Series (advanced Fluctuation EM)
More details and examples:
https://libertem.github.io/LiberTEM/applications.html
Extended documentation
======================
We have greatly improved the coverage of our documentation:
https://libertem.github.io/LiberTEM/index.html#documentation
Fully automated release pipeline
================================
Alex (@sk1p) invested a great deal of effort into fully automating our
release process. From now on, we will be able to release more often,
including service releases. 🚀
Basic dask.distributed array integration
========================================
LiberTEM can now generate efficient dask.distributed arrays from all
supported dataset types. That means it should be possible to use our
high-performance file readers in applications outside of LiberTEM.
File formats
============
Support for various file formats has improved. More details:
https://libertem.github.io/LiberTEM/formats.html
***
Ongoing and future activities
=============================
Change management
-----------------
With this release we will start tracking and managing changes
systematically. In particular, we have taken a considerable amount of
time and testing to stabilize the UDF interface. It should now be ready
for widespread use.
PCA
---
Strategy and prototype implementation for efficient distributed
calculation of PCA and similar decompositions by Jaeweon (@jaewshin), a
Google Summer of Code student: https://github.com/LiberTEM/LiberTEM/pull/347
To be further tested and validated before merging into the production
branch. Please contact us if you are interested in such a capability!
Production-ready cluster
------------------------
We are planning to fulfill a number of requirements that are relevant
for using LiberTEM on a cluster in a production environment:
* Authentication and authorization
* HTTPS
* Managing Python environments to ensure that client and back-end match
* Caching
* Data management
The solution will likely be a combination of new LiberTEM features and
matching configuration and deployment of third-party tools.
Full dask.distributed array integration
---------------------------------------
We are planning to allow creating a dataset from a dask.distributed
array and to create dask.distributed arrays using UDFs. That should
enable seamless integration between Hyperspy and LiberTEM, which is
particularly important to merge functionality that is present in both
pyXem and LiberTEM at the moment. Please contact us if you are interested!
Ptychography
------------
We are part of the Helmholtz Pilot Project "Ptychography 4.0" to develop
new approaches for ptychography. Our main contribution will be efficient
distribution and handling live data. As a result, LiberTEM will most
likely support ptychography in the future.
Live data processing
--------------------
As a next major feature we are now targeting processing of live data.
This will likely be the dominating task for 2020.
Miscellaneous
=============
Congratulations to Karina for finishing her Master's degree and getting
a PhD position at RWTH Aachen!
We participated in the Google Summer of Code 2019 under the umbrella of
the Python Software Foundation (PSF). Many thanks to Karina, Jaeweon,
the PSF team and Google for the support!
The ERC Proof of Concept VIDEO that funded the initial development of
LiberTEM ended successfully in June. Follow-up funding for LiberTEM is
secured.
We are hiring!
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-188-EN-ER-C%201.html
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-191-EN-ER-C%201.html
https://fz-juelich.de/SharedDocs/Stellenangebote/_common/dna/2019D-190-EN-ER-C%201.html
With best regards,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Reply all
Reply to author
Forward
0 new messages