Status update September 2018

[Dieter Weber]

Dear all,

it has been busy times! LiberTEM is now very close to a 0.1 release and
I can cordially recommend it for testing. That means we are ahead of
schedule by one quarter! These are the main achievements in the last two
months:

* Pushing throughput to 14 GB/s per node from cached file system reads
by using Torch for processing and memory mapping for reading.
* Improved format support, including a high-performance reader for K2IS
raw data, currently in beta state
* Python API for simple and powerful scripting, see our documentation
https://libertem.github.io/LiberTEM/usage.html#the-python-api and
examples: https://github.com/LiberTEM/LiberTEM/tree/master/examples
* Proper documentation for installation and usage:
https://libertem.github.io/LiberTEM/index.html Please let us know if
something can be improved or clarified!

At IMC19 and in other conversation I heard a strong demand for live
acquisition and integration in other software as a processing back-end.
That is one of the things on our short list. If you have specific wishes
or comments about workflows, features etc, I'd be very happy about your
comments in the corresponding GitHub issues
https://github.com/LiberTEM/LiberTEM/issues/122
https://github.com/LiberTEM/LiberTEM/issues/123 or through other
channels. :-) Our plan is to draft up typical workflows based on your
input, send them around for your comments, and then see how to implement
that architecture-wise.

Here in Jülich we have an ongoing master thesis about precise
identification and tracing of diffraction "blobs", i.e. spots with
moderate convergence angles that have a lot of internal structure. That
is progressing nicely and we'll make a plan on integrating that soon.
More to follow.

Another action item is to use physical units instead of pixels. Together
with that we are planning to pick up work on the file format again,
which was a bit slowed down because we couldn't actually make much use
of the metadata previously. That should change when we support physical
units. An emerging point is to use the metadata independently of the
actual file, i.e. being able to use it as a sidecar file for other
formats. LiberTEM got pretty good at reading different kinds of files
very efficiently, that means we can save the time and storage space
required for conversion.

Which other features and capabilities would you like to see in LiberTEM
for the future?

We are very happy to work on deployment of LiberTEM on the Australian
cloud. :-)

Here in Jülich we are procuring a compact blade system with eight blades
that is optimized for high IO throughput and price-performance ratio
with LiberTEM. With that we can properly test and optimize cluster
deployment for optimal performance. We'd hope for 50 GB/s reading from
mass storage and 100 GB/s for cached reads. We'll share our experiences
in due time.

Another agenda item is to manage publications around LiberTEM in a way
that creates academic visibility for users and contributors. Do you have
some ideas how to do that, maybe best practice from other projects?

The big topic for the rest of the project's duration will be ensuring
long-term sustainability, hoping that LiberTEM is useful enough to keep
around and continue growing further. The options would include getting
funding through research grants, through other sources of public
funding, service contracts, donations, employment of developers, and
last but not least in-kind contributions of code and perhaps hardware.
The Jülich blade system could already count as that. Many thanks to
Joachim Mayer (ER-C-2)! We'd be very happy about help to keep the
project going and build all the nice things that will help you do more
and better science. :-)

I'm looking forward to your feedback!

With best regards,
Dieter

--
Dr. Dieter WEBER

Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany

Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

[Ian MacLaren]

Dear Dieter/all,

A standard credit line for an Acknowledgements section developed by you and downloadable from a “publications” page hosted by your institute would be helpful.  

A basic paper on LiberTEM with a DOI would be a very useful thing to attract citations for all users.  Hyperspy has a DOI, for example.  I think Juri Barthel just got a paper on Dr Probe too, for the same reason.  You need citations to build visibility in the academic community!

A formally written publications policy governing usage would be very worthwhile, basically committing all users who publish data processed with LiberTEM to putting in the correct Acknowledgements and citation would be essential.  You might also want to specify when co-authorship of a paper is necessary, or allowing paper review by yourselves before submission is required.

Best wishes

Ian

On 27 September 2018 at 12:16:54, Dieter Weber (d.w...@fz-juelich.de) wrote:

Another agenda item is to manage publications around LiberTEM in a way 
that creates academic visibility for users and contributors. Do you have 
some ideas how to do that, maybe best practice from other projects? 

Dr Ian MacLaren
BSc (Hons), PhD, FInstP, CPhys
Reader in Physics

Direct line: +44 (0)141 330 4700
Fax:: +44 (0)141 330 4464

Materials and Condensed Matter Physics
School of Physics and Astronomy
University of Glasgow
Glasgow G12 8QQ

http://www.gla.ac.uk/schools/physics/research/groups/mcmp/

http://www.researcherid.com/rid/C-1773-2010

ORCID: 0000-0002-5334-3010

The University of Glasgow, charity number SC004401