Which formats of dark image, gain map and dead and hot pixel maps or lists to support?
23 views
Skip to first unread message
Dieter Weber
May 8, 2020, 4:25:13 AM5/8/20
to libe...@googlegroups.com, Juri Barthel, Müller-Caspary, Knut, Alexander Clausen
Hi all,
right now we are integrating support for detector corrections in
LiberTEM, see https://github.com/LiberTEM/LiberTEM/pull/778. It supports
subtraction of a dark frame, multiplication with a gain map and patching
bad pixels with an average of their good neighbors "on the fly".
To develop it in a convenient fashion, a quick question for each of you:
Which data formats do you currently use to store these maps?
That will help us to develop convenient import facilities for scripting
and in the GUI.
With many thanks in advance,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
right now we are integrating support for detector corrections in
LiberTEM, see https://github.com/LiberTEM/LiberTEM/pull/778. It supports
subtraction of a dark frame, multiplication with a gain map and patching
bad pixels with an average of their good neighbors "on the fly".
To develop it in a convenient fashion, a quick question for each of you:
Which data formats do you currently use to store these maps?
That will help us to develop convenient import facilities for scripting
and in the GUI.
With many thanks in advance,
Dieter
--
Dr. Dieter WEBER
Peter Grünberg Institute, Microstructure Research (PGI-5)
Ernst Ruska-Centre for Microscopy and Spectroscopy with Electrons (ER-C)
Forschungszentrum Jülich
52425 Jülich, Germany
Email: d.w...@fz-juelich.de
Phone: +49 2461 61 85118
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Christoph Koch
May 8, 2020, 6:00:34 AM5/8/20
to Dieter Weber, libe...@googlegroups.com, Juri Barthel, Müller-Caspary, Knut, Alexander Clausen
Hi Dieter,
thank you for starting this discussion. It seems that at every workshop on the topic of (meta-)data management there seems to be a general consensus that it might be a good idea to develop an EM application definition for the NeXus format that is getting more and more accepted in the X-ray and neutron community. There is even a dedicated NeXus branch in the liberTEM project. Having studied the format rules for such an application definition a bit, and trying to maximize compatibility with what you have already done in liberTEM, we have drafted the attached EM Nexus specification for 4D-STEM that also contains a placeholder for flatfield and darkfield. Dead and hot pixels could be added to the detector as well. There are also flags that indicate whether these corrections have been applied to the data, or not.
This is currently still under development and not fixed yet, and nobody is using this format yet. And even in our own lab we are only slowly transitioning. But I think it is a good idea to store this information together with your data, so that all the corrections that are applied to the data are transparent and potentially also reversible.
We have started to circulate this proposal among only very few microscopy groups, in order to further refine it before sharing with all, but I took this opportunity to reply to all, in order to spawn a discussion on the type of format that we would like to use to exchange these large data sets. Now seems to be the time to come to an agreement.
Best regards,
Christoph.
Prof. Christoph T. Koch, PhD
Institut für Physik & IRIS Adlershof
Humboldt-Universität zu Berlin
Newtonstraße 15
12489 Berlin
Tel: +49 30 2093 7640
Tel: +49 30 2093 7652 (Sekretariat)
https://www.physics.hu-berlin.de/en/sem
-----Ursprüngliche Nachricht-----
Von: libe...@googlegroups.com <libe...@googlegroups.com> Im Auftrag von Dieter Weber
Gesendet: Freitag, 8. Mai 2020 10:25
An: libe...@googlegroups.com; Juri Barthel <Ju.Ba...@fz-juelich.de>; Müller-Caspary, Knut <k.muelle...@fz-juelich.de>
Cc: Alexander Clausen <a.cl...@fz-juelich.de>
Betreff: [libertem] Which formats of dark image, gain map and dead and hot pixel maps or lists to support?
--
You received this message because you are subscribed to the Google Groups "LiberTEM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libertem+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libertem/7c9c8149-d588-f25f-f62e-f6c56fb12213%40fz-juelich.de.
thank you for starting this discussion. It seems that at every workshop on the topic of (meta-)data management there seems to be a general consensus that it might be a good idea to develop an EM application definition for the NeXus format that is getting more and more accepted in the X-ray and neutron community. There is even a dedicated NeXus branch in the liberTEM project. Having studied the format rules for such an application definition a bit, and trying to maximize compatibility with what you have already done in liberTEM, we have drafted the attached EM Nexus specification for 4D-STEM that also contains a placeholder for flatfield and darkfield. Dead and hot pixels could be added to the detector as well. There are also flags that indicate whether these corrections have been applied to the data, or not.
This is currently still under development and not fixed yet, and nobody is using this format yet. And even in our own lab we are only slowly transitioning. But I think it is a good idea to store this information together with your data, so that all the corrections that are applied to the data are transparent and potentially also reversible.
We have started to circulate this proposal among only very few microscopy groups, in order to further refine it before sharing with all, but I took this opportunity to reply to all, in order to spawn a discussion on the type of format that we would like to use to exchange these large data sets. Now seems to be the time to come to an agreement.
Best regards,
Christoph.
Prof. Christoph T. Koch, PhD
Institut für Physik & IRIS Adlershof
Humboldt-Universität zu Berlin
Newtonstraße 15
12489 Berlin
Tel: +49 30 2093 7640
Tel: +49 30 2093 7652 (Sekretariat)
https://www.physics.hu-berlin.de/en/sem
-----Ursprüngliche Nachricht-----
Von: libe...@googlegroups.com <libe...@googlegroups.com> Im Auftrag von Dieter Weber
Gesendet: Freitag, 8. Mai 2020 10:25
An: libe...@googlegroups.com; Juri Barthel <Ju.Ba...@fz-juelich.de>; Müller-Caspary, Knut <k.muelle...@fz-juelich.de>
Cc: Alexander Clausen <a.cl...@fz-juelich.de>
Betreff: [libertem] Which formats of dark image, gain map and dead and hot pixel maps or lists to support?
You received this message because you are subscribed to the Google Groups "LiberTEM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to libertem+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/libertem/7c9c8149-d588-f25f-f62e-f6c56fb12213%40fz-juelich.de.
Ian MacLaren
May 8, 2020, 6:27:25 AM5/8/20
to libe...@googlegroups.com, Juri Barthel, christo...@hu-berlin.de, Müller-Caspary, Knut, Dieter Weber, Magnus Nord, Gary Paterson, Alexander Clausen
I’m too much of an idiot on technical details to have anything worthwhile to say, but Magnus or Gary are probably far more useful on such points…
Best wishes
Ian
To view this discussion on the web visit https://groups.google.com/d/msgid/libertem/00b301d6251f%2481a86900%2484f93b00%24%40hu-berlin.de.
Dr Ian MacLaren
(he, him, his)
BSc (Hons), PhD, FInstP, CPhys
Reader in Physics
Phone contact currently impossible due to Covid-19 and not being in my office
Materials and Condensed Matter Physics
School of Physics and Astronomy
University of Glasgow
Glasgow G12 8QQ
ORCID: 0000-0002-5334-3010
The University of Glasgow, charity number SC004401
Andy.Stewart
May 8, 2020, 8:28:43 AM5/8/20
to d.w...@fz-juelich.de, libe...@googlegroups.com, Juri Barthel, Müller-Caspary, Knut, Alexander Clausen
Hi Dieter et al.,
Are you developing this for every/generic detector or a specific set of defectors?
I'm presuming we develop a format that can be widely adopted and those that are willing will adopt, and once enough momentum has been built others will hopefully follow, or as open data requirements encourage / force them to.
Personally, I like the NeXus based format (https://www.nexusformat.org ), it has a strong user community and is well established and covers most if not all of our requirements, in terms of openness, APIs, community support, and we don't have to reinvent the wheel.
As the size of data sets explode with 4D and bigger and faster detectors, on the fly lossless compression should be considered as an integral part of the format in order to help with moving and processing the data. Formats such as MRCZ (https://github.com/em-MRCZ/python-mrcz) and ReCoDe (https://github.com/NDLOHGRP/ReCoDe) are good examples of this.
It would be prudent to talk with the CCPEM and consult with them about how they have handled the development of their formats and the challenges evolving the format and how best we can avoid these difficulties by making wise choices at this early stage. (https://www.ccpem.ac.uk/mrc_format/mrc_format.php )
Personally I would like to see some sort of blockchain added to these formats for data tracing and accountability especially as we begin to develop databases of data for physical sciences data, whither they are centralised or decentralised is a topic for another time, but it is a development the field requires.
Currently we use the standard data formats provided by the manufacturers and convert to hdf5 or nexus depending up on what we are using, a single solution is highly desirable.
Encouraging users to input the missing metadata especially about the sample and its preparation would be highly desirable.
Best regards,
Andy
Dr Andy Stewart (he/him/his)
Lecturer & Principal Investigator
Department of Physics & Bernal Institute
AD 3-016
Analog Devices Building
University of Limerick
T +353 (0) 61 237733
W bernalinstitute.com/our_people/andy-stewart
W https://www.ul.ie/scieng/dr-andy-stewart
On 08/05/2020, 09:25, "libe...@googlegroups.com on behalf of d.w...@fz-juelich.de" <libe...@googlegroups.com on behalf of d.w...@fz-juelich.de> wrote:
EXTERNAL EMAIL: This email originated from outside of the University of Limerick. Do not click on links or open attachments unless you recognize the sender's email address and know the content is safe.
Are you developing this for every/generic detector or a specific set of defectors?
I'm presuming we develop a format that can be widely adopted and those that are willing will adopt, and once enough momentum has been built others will hopefully follow, or as open data requirements encourage / force them to.
Personally, I like the NeXus based format (https://www.nexusformat.org ), it has a strong user community and is well established and covers most if not all of our requirements, in terms of openness, APIs, community support, and we don't have to reinvent the wheel.
As the size of data sets explode with 4D and bigger and faster detectors, on the fly lossless compression should be considered as an integral part of the format in order to help with moving and processing the data. Formats such as MRCZ (https://github.com/em-MRCZ/python-mrcz) and ReCoDe (https://github.com/NDLOHGRP/ReCoDe) are good examples of this.
It would be prudent to talk with the CCPEM and consult with them about how they have handled the development of their formats and the challenges evolving the format and how best we can avoid these difficulties by making wise choices at this early stage. (https://www.ccpem.ac.uk/mrc_format/mrc_format.php )
Personally I would like to see some sort of blockchain added to these formats for data tracing and accountability especially as we begin to develop databases of data for physical sciences data, whither they are centralised or decentralised is a topic for another time, but it is a development the field requires.
Currently we use the standard data formats provided by the manufacturers and convert to hdf5 or nexus depending up on what we are using, a single solution is highly desirable.
Encouraging users to input the missing metadata especially about the sample and its preparation would be highly desirable.
Best regards,
Andy
Dr Andy Stewart (he/him/his)
Lecturer & Principal Investigator
Department of Physics & Bernal Institute
AD 3-016
Analog Devices Building
University of Limerick
T +353 (0) 61 237733
W bernalinstitute.com/our_people/andy-stewart
W https://www.ul.ie/scieng/dr-andy-stewart
On 08/05/2020, 09:25, "libe...@googlegroups.com on behalf of d.w...@fz-juelich.de" <libe...@googlegroups.com on behalf of d.w...@fz-juelich.de> wrote:
EXTERNAL EMAIL: This email originated from outside of the University of Limerick. Do not click on links or open attachments unless you recognize the sender's email address and know the content is safe.
Reply all
Reply to author
Forward
0 new messages