APT-HDF5 vote
68 views
Skip to first unread message
Daniel Haley
Oct 27, 2020, 6:20:12 AM10/27/20
to atompr...@googlegroups.com
Hi all,
As required by our charter, I am calling for a renewed vote for the
following action:
To disseminate the current draft standard on APT-HDF, for a "Request for
comments" phase from the community, where we invite the wider community
to review the document.
The proposed standard and the associated software links will be provided
as part of the mailout. Dissemination will be via the ISC, likely the
IFES mailing list.
The publicly visible standard document is here:
https://docs.google.com/document/u/2/d/e/2PACX-1vRxcJ_xF_jiNS77CoeZdQdDXD8l2BebL-DoOBkDrAsGTrkArdjLHEMCXAifBieeS8pTO9jJ9xnstKxs/pub
Option 1: Disseminate the Draft standard
Option 2: Withhold the Draft standard for further revisions.
Please indicate in your response:
"Vote : Option X", where X is 1 or 2
If there are small errors that should be corrected, please directly
modify the document, rather than on-list discussion. If you believe
further revisions are needed, we can discuss this separately to your vote.
I'm told that the document above can be hosted on the IFES website, but
am waiting for the SC to actually upload.
Please do submit your vote as soon as possible.
Thanks,
Daniel
As required by our charter, I am calling for a renewed vote for the
following action:
To disseminate the current draft standard on APT-HDF, for a "Request for
comments" phase from the community, where we invite the wider community
to review the document.
The proposed standard and the associated software links will be provided
as part of the mailout. Dissemination will be via the ISC, likely the
IFES mailing list.
The publicly visible standard document is here:
https://docs.google.com/document/u/2/d/e/2PACX-1vRxcJ_xF_jiNS77CoeZdQdDXD8l2BebL-DoOBkDrAsGTrkArdjLHEMCXAifBieeS8pTO9jJ9xnstKxs/pub
Option 1: Disseminate the Draft standard
Option 2: Withhold the Draft standard for further revisions.
Please indicate in your response:
"Vote : Option X", where X is 1 or 2
If there are small errors that should be corrected, please directly
modify the document, rather than on-list discussion. If you believe
further revisions are needed, we can discuss this separately to your vote.
I'm told that the document above can be hosted on the IFES website, but
am waiting for the SC to actually upload.
Please do submit your vote as soon as possible.
Thanks,
Daniel
Dieter Isheim
Oct 27, 2020, 9:32:12 AM10/27/20
to atompr...@googlegroups.com
Vote: Option 1
"Disseminate the Draft standard"
Comment: this is ready for discussion in the broader community, pending some smaller updates and clarifications.
Best to all - Dieter
> On Oct 27, 2020, at 5:20 AM, Daniel Haley <daniel...@materials.ox.ac.uk> wrote:
>
> Hi all,
>
> As required by our charter, I am calling for a renewed vote for the
> following action:
>
> To disseminate the current draft standard on APT-HDF, for a "Request for
> comments" phase from the community, where we invite the wider community
> to review the document.
>
> The proposed standard and the associated software links will be provided
> as part of the mailout. Dissemination will be via the ISC, likely the
> IFES mailing list.
>
> The publicly visible standard document is here:
>
> https://urldefense.com/v3/__https://docs.google.com/document/u/2/d/e/2PACX-1vRxcJ_xF_jiNS77CoeZdQdDXD8l2BebL-DoOBkDrAsGTrkArdjLHEMCXAifBieeS8pTO9jJ9xnstKxs/pub__;!!Dq0X2DkFhyF93HkjWTBQKhk!AhOwjWrrjx5RHbOBKBK35z9P4Pk17czxYuybqXMnR9hy0_b0tYRUV4w7GNfXbMz2ojV4$
> You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
> To view this discussion on the web, visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/atomprobe-tc/49f5dd26-81b8-d61c-7935-82fe79169321*40materials.ox.ac.uk__;JQ!!Dq0X2DkFhyF93HkjWTBQKhk!AhOwjWrrjx5RHbOBKBK35z9P4Pk17czxYuybqXMnR9hy0_b0tYRUV4w7GNfXbDvQ_ghP$ .
"Disseminate the Draft standard"
Comment: this is ready for discussion in the broader community, pending some smaller updates and clarifications.
Best to all - Dieter
> On Oct 27, 2020, at 5:20 AM, Daniel Haley <daniel...@materials.ox.ac.uk> wrote:
>
> Hi all,
>
> As required by our charter, I am calling for a renewed vote for the
> following action:
>
> To disseminate the current draft standard on APT-HDF, for a "Request for
> comments" phase from the community, where we invite the wider community
> to review the document.
>
> The proposed standard and the associated software links will be provided
> as part of the mailout. Dissemination will be via the ISC, likely the
> IFES mailing list.
>
> The publicly visible standard document is here:
>
>
> Option 1: Disseminate the Draft standard
> Option 2: Withhold the Draft standard for further revisions.
>
> Please indicate in your response:
>
> "Vote : Option X", where X is 1 or 2
>
> If there are small errors that should be corrected, please directly
> modify the document, rather than on-list discussion. If you believe
> further revisions are needed, we can discuss this separately to your vote.
>
> I'm told that the document above can be hosted on the IFES website, but
> am waiting for the SC to actually upload.
>
> Please do submit your vote as soon as possible.
>
> Thanks,
>
> Daniel
>
> --
> Option 1: Disseminate the Draft standard
> Option 2: Withhold the Draft standard for further revisions.
>
> Please indicate in your response:
>
> "Vote : Option X", where X is 1 or 2
>
> If there are small errors that should be corrected, please directly
> modify the document, rather than on-list discussion. If you believe
> further revisions are needed, we can discuss this separately to your vote.
>
> I'm told that the document above can be hosted on the IFES website, but
> am waiting for the SC to actually upload.
>
> Please do submit your vote as soon as possible.
>
> Thanks,
>
> Daniel
>
> You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
> To view this discussion on the web, visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/atomprobe-tc/49f5dd26-81b8-d61c-7935-82fe79169321*40materials.ox.ac.uk__;JQ!!Dq0X2DkFhyF93HkjWTBQKhk!AhOwjWrrjx5RHbOBKBK35z9P4Pk17czxYuybqXMnR9hy0_b0tYRUV4w7GNfXbDvQ_ghP$ .
London, Andy
Oct 27, 2020, 1:31:05 PM10/27/20
to atompr...@googlegroups.com
Hi all,
I've made some changes directly as encouraged, the main one was adding an "overview" section that explains the terminology of fields and groups - people with little HDF5 knowledge would otherwise find later parts of the document confusing.
Three points to raise:
(1) for the coordinate systems, would it be helpful to have diagrams for these? Might be tricky in 3D but I can try.
I also don't quite understand the bit about "If the electrode moves relative to the laboratory frame, then the space should be defined such that, in tip space, the electrode’s position does not change." - surely the electrode *would* move in tip space?
(2) Did we decide whether 3-by-n or n-by-3 resulted in better performance? There are comments on the document but I'm not sure if it's concluded
(3) We should ship the document with a complete example HDF5 file with all the fields populated with plausible data.
After clarification on the points above, I will vote for option 1.
Andy
I've made some changes directly as encouraged, the main one was adding an "overview" section that explains the terminology of fields and groups - people with little HDF5 knowledge would otherwise find later parts of the document confusing.
Three points to raise:
(1) for the coordinate systems, would it be helpful to have diagrams for these? Might be tricky in 3D but I can try.
I also don't quite understand the bit about "If the electrode moves relative to the laboratory frame, then the space should be defined such that, in tip space, the electrode’s position does not change." - surely the electrode *would* move in tip space?
(2) Did we decide whether 3-by-n or n-by-3 resulted in better performance? There are comments on the document but I'm not sure if it's concluded
(3) We should ship the document with a complete example HDF5 file with all the fields populated with plausible data.
After clarification on the points above, I will vote for option 1.
Andy
--
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/49f5dd26-81b8-d61c-7935-82fe79169321%40materials.ox.ac.uk.
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
Daniel Haley
Oct 27, 2020, 2:33:36 PM10/27/20
to atompr...@googlegroups.com
Hi Andy,
> (1) for the coordinate systems, would it be helpful to have diagrams
> for these? Might be tricky in 3D but I can try. I also don't quite
> understand the bit about "If the electrode moves relative to the
> laboratory frame, then the space should be defined such that, in tip
> space, the electrode’s position does not change." - surely the
> electrode *would* move in tip space?
For 1) The idea of the wording was to convey the scenario where, like on
the Stuttgart systems, you have a shuttle with both the tip and
electrode together. In this system, there is a movable arm that moves
the electrode; In such a scenario, the electrode defines the static
component of tip-space. A fixed tip (in lab space) in their system will
move in tip space when the electrode is moved in lab space. However, we
must also accommodate electrode-less setups, so we cannot call it
electrode space. Fixing the tip-space to the sample apex permanently
would require an additional electrode space, which may not be optically
relevant in all cases.
This is still definable by a fixed 3x3 matrix, as the rotation is
invariant, only translational movement is defined. The standard only
calls for a fixed 3x3 matrix between tip and laboratory frames.
Currently only the final snapshot of transformation is recorded in the
LabtoTipSpace group, which holds a 4x4 array. This means that the
transit itself is not recorded, and the geometry is assumed static.
Future revisions can address dynamic setups.
Screenshots are attached to explain the setup somewhat - I think I have
it right in the images - its a freecad file so I can post this if desired.
2) I would agree with your assessment; I think to determine the use case
and performance, demonstration code is required. For the C-Code
currently on gitlab, [1] there is no user-visible change between an mxn
and an nxm system - the vector components are simply transposed.
I have not tested for performance at this point, and think performance
should be addressed in a future revision. Discussing performance
improvements should only be made when a viable "minimum product" is
obtained, and improvements can be quantified.
3) You can auto-generate simulated files using the provided generator,
using the --generate flag. Practical files would require significant
work to provide a more realistic experimental simulator, however we are
free to improve the generator independently of the standard itself.
Hopefully this helps?
[1] https://gitlab.com/atomprobe.tc/apt-hdf5/-/blob/master/common.cpp ,
lines 786-835
On 27.10.20 17:31, London, Andy wrote:
> Hi all,
> I've made some changes directly as encouraged, the main one was adding
an "overview" section that explains the terminology of fields and groups
- people with little HDF5 knowledge would otherwise find later parts of
the document confusing.
>
> Three points to raise:
> (1) for the coordinate systems, would it be helpful to have diagrams
> for these? Might be tricky in 3D but I can try. I also don't quite
> understand the bit about "If the electrode moves relative to the
> laboratory frame, then the space should be defined such that, in tip
> space, the electrode’s position does not change." - surely the
> electrode *would* move in tip space?
the Stuttgart systems, you have a shuttle with both the tip and
electrode together. In this system, there is a movable arm that moves
the electrode; In such a scenario, the electrode defines the static
component of tip-space. A fixed tip (in lab space) in their system will
move in tip space when the electrode is moved in lab space. However, we
must also accommodate electrode-less setups, so we cannot call it
electrode space. Fixing the tip-space to the sample apex permanently
would require an additional electrode space, which may not be optically
relevant in all cases.
This is still definable by a fixed 3x3 matrix, as the rotation is
invariant, only translational movement is defined. The standard only
calls for a fixed 3x3 matrix between tip and laboratory frames.
Currently only the final snapshot of transformation is recorded in the
LabtoTipSpace group, which holds a 4x4 array. This means that the
transit itself is not recorded, and the geometry is assumed static.
Future revisions can address dynamic setups.
Screenshots are attached to explain the setup somewhat - I think I have
it right in the images - its a freecad file so I can post this if desired.
2) I would agree with your assessment; I think to determine the use case
and performance, demonstration code is required. For the C-Code
currently on gitlab, [1] there is no user-visible change between an mxn
and an nxm system - the vector components are simply transposed.
I have not tested for performance at this point, and think performance
should be addressed in a future revision. Discussing performance
improvements should only be made when a viable "minimum product" is
obtained, and improvements can be quantified.
3) You can auto-generate simulated files using the provided generator,
using the --generate flag. Practical files would require significant
work to provide a more realistic experimental simulator, however we are
free to improve the generator independently of the standard itself.
Hopefully this helps?
[1] https://gitlab.com/atomprobe.tc/apt-hdf5/-/blob/master/common.cpp ,
lines 786-835
On 27.10.20 17:31, London, Andy wrote:
> Hi all,
> I've made some changes directly as encouraged, the main one was adding
an "overview" section that explains the terminology of fields and groups
- people with little HDF5 knowledge would otherwise find later parts of
the document confusing.
>
> Three points to raise:
Anna Ceguerra
Oct 27, 2020, 6:06:41 PM10/27/20
to atompr...@googlegroups.com
Vote Option 1 (Disseminate)
I am hoping the community can give feedback on my concern, which is the order of the dimensions (I prefer n x 3 for example, not 3 x n which is in the document).
Regards,
Anna.
From:
atompr...@googlegroups.com <atompr...@googlegroups.com>
Date: Tuesday, 27 October 2020 at 9:20 pm
To: atompr...@googlegroups.com <atompr...@googlegroups.com>
Subject: APT-HDF5 vote
Hi all,
As required by our charter, I am calling for a renewed vote for the
following action:
To disseminate the current draft standard on APT-HDF, for a "Request for
comments" phase from the community, where we invite the wider community
to review the document.
The proposed standard and the associated software links will be provided
as part of the mailout. Dissemination will be via the ISC, likely the
IFES mailing list.
The publicly visible standard document is here:
Option 1: Disseminate the Draft standard
Option 2: Withhold the Draft standard for further revisions.
Please indicate in your response:
"Vote : Option X", where X is 1 or 2
If there are small errors that should be corrected, please directly
modify the document, rather than on-list discussion. If you believe
further revisions are needed, we can discuss this separately to your vote.
I'm told that the document above can be hosted on the IFES website, but
am waiting for the SC to actually upload.
Please do submit your vote as soon as possible.
Thanks,
Daniel
--
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
To view this discussion on the web, visit https://protect-au.mimecast.com/s/7oHKC91WPRTQBq19tEdM-0?domain=groups.google.com.
David Reinhard
Oct 28, 2020, 12:42:08 AM10/28/20
to atompr...@googlegroups.com
Vote: Option 1
David Reinhard
APT Product Manager | CAMECA Instruments, Inc
5470 Nobel Drive | Madison, WI 53711 | USA
David.R...@ametek.com | T: +1 608 467 1234
www.cameca.com
-----Original Message-----
From: atompr...@googlegroups.com <atompr...@googlegroups.com> On Behalf Of Dieter Isheim
Sent: Tuesday, October 27, 2020 8:32 AM
To: atompr...@googlegroups.com
Subject: Re: APT-HDF5 vote
***NOTICE*** This came from an external source. Use caution when replying, clicking links, or opening attachments.
To view this discussion on the web, visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/atomprobe-tc/58E4FC9E-EA89-4166-AD2E-AE82A12C4BA2*40northwestern.edu__;JQ!!HKOSU0g!VdEhA6x2V4cojGgfeK1mxD5pGdzyDK5DSEUOMoISBlQTFR3WA46MJDW1JP-uwbCtih4$ .
David Reinhard
APT Product Manager | CAMECA Instruments, Inc
5470 Nobel Drive | Madison, WI 53711 | USA
David.R...@ametek.com | T: +1 608 467 1234
www.cameca.com
-----Original Message-----
From: atompr...@googlegroups.com <atompr...@googlegroups.com> On Behalf Of Dieter Isheim
Sent: Tuesday, October 27, 2020 8:32 AM
To: atompr...@googlegroups.com
Subject: Re: APT-HDF5 vote
***NOTICE*** This came from an external source. Use caution when replying, clicking links, or opening attachments.
Bertrand Radiguet
Oct 28, 2020, 4:07:55 AM10/28/20
to atompr...@googlegroups.com
Hello all,
I vote "option 1: Disseminate the Draft standard "
best regards,
bertrand
I vote "option 1: Disseminate the Draft standard "
best regards,
bertrand
Oberdorfer, Chris
Oct 28, 2020, 11:39:47 AM10/28/20
to atompr...@googlegroups.com
Hi all,
I join the others and vote for Optin 1 (Disseminate).
Thanks,
Dr. Christian Oberdorfer
College
of Engineering Department
of Materials Science and Engineering
5076A Alpheaus Smith Laboratoy of Physics
(Mail: 2041 College Rd, Columbus , OH 43210)
From: atompr...@googlegroups.com <atompr...@googlegroups.com> on behalf of Daniel Haley <daniel...@materials.ox.ac.uk>
Sent: Tuesday, October 27, 2020 5:20 AM
Sent: Tuesday, October 27, 2020 5:20 AM
Hi all,
As required by our charter, I am calling for a renewed vote for the
following action:
To disseminate the current draft standard on APT-HDF, for a "Request for
comments" phase from the community, where we invite the wider community
to review the document.
The proposed standard and the associated software links will be provided
as part of the mailout. Dissemination will be via the ISC, likely the
IFES mailing list.
The publicly visible standard document is here:
As required by our charter, I am calling for a renewed vote for the
following action:
To disseminate the current draft standard on APT-HDF, for a "Request for
comments" phase from the community, where we invite the wider community
to review the document.
The proposed standard and the associated software links will be provided
as part of the mailout. Dissemination will be via the ISC, likely the
IFES mailing list.
The publicly visible standard document is here:
Option 1: Disseminate the Draft standard
Option 2: Withhold the Draft standard for further revisions.
Please indicate in your response:
"Vote : Option X", where X is 1 or 2
If there are small errors that should be corrected, please directly
modify the document, rather than on-list discussion. If you believe
further revisions are needed, we can discuss this separately to your vote.
I'm told that the document above can be hosted on the IFES website, but
am waiting for the SC to actually upload.
Please do submit your vote as soon as possible.
Thanks,
Daniel
--
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
London, Andy
Oct 30, 2020, 1:29:36 PM10/30/20
to atompr...@googlegroups.com
Hi all,
I did a quick test in matlab just to check that the order didn’t matter for writing real data:
Write 4 by 7.5E7 ions Elapsed time is 28.779395 seconds.
Write 7.5E7 by 4 ions Elapsed time is 26.822038 seconds.
>> hdf5_speed_test
Read 4 by 7.5E7 ions Elapsed time is 1.156697 seconds.
Read 7.5E7 by 4 ions Elapsed time is 1.058228 seconds.
Markus, was your concern about reading chunks of ions? I.e. you want [x1 y1 z1 m1] [x2 …] etc.?
Andy
From: 'Anna Ceguerra' via AtomProbe TC <atompr...@googlegroups.com>
Sent: 27 October 2020 22:07
To: atompr...@googlegroups.com
Subject: Re: APT-HDF5 vote
Vote Option 1 (Disseminate)
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com.
Daniel Haley
Oct 30, 2020, 2:09:04 PM10/30/20
to atompr...@googlegroups.com
Hi Andy,
Thanks very much - its good to have some numbers to go alogn with this.
Can you provide the sample code at all, so we have reference for what is
being tested by those values?
Thanks,
Daniel
On 30/10/2020 17:29, London, Andy wrote:
> Hi all,
>
> I did a quick test in matlab just to check that the order didn’t matter
> for writing real data:
>
> Write 4 by 7.5E7 ions Elapsed time is 28.779395 seconds.
>
> Write 7.5E7 by 4 ions Elapsed time is 26.822038 seconds.
>
>>> hdf5_speed_test
>
> Read 4 by 7.5E7 ions Elapsed time is 1.156697 seconds.
>
> Read 7.5E7 by 4 ions Elapsed time is 1.058228 seconds.
>
>
>
> Markus, was your concern about reading chunks of ions? I.e. you want [x1
> y1 z1 m1] [x2 …] etc.?
>
> Andy
>
>
>
> *From:*'Anna Ceguerra' via AtomProbe TC <atompr...@googlegroups.com>
> *Sent:* 27 October 2020 22:07
> *To:* atompr...@googlegroups.com
> *Subject:* Re: APT-HDF5 vote
> <mailto:atompr...@googlegroups.com> <atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>>
> *Date: *Tuesday, 27 October 2020 at 9:20 pm
> *To: *atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com> <atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>>
> *Subject: *APT-HDF5 vote
> <mailto:atomprobe-tc...@googlegroups.com>.
> <https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com?utm_medium=email&utm_source=footer>.
> <https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.
Thanks very much - its good to have some numbers to go alogn with this.
Can you provide the sample code at all, so we have reference for what is
being tested by those values?
Thanks,
Daniel
On 30/10/2020 17:29, London, Andy wrote:
> Hi all,
>
> I did a quick test in matlab just to check that the order didn’t matter
> for writing real data:
>
> Write 4 by 7.5E7 ions Elapsed time is 28.779395 seconds.
>
> Write 7.5E7 by 4 ions Elapsed time is 26.822038 seconds.
>
>>> hdf5_speed_test
>
> Read 4 by 7.5E7 ions Elapsed time is 1.156697 seconds.
>
> Read 7.5E7 by 4 ions Elapsed time is 1.058228 seconds.
>
>
>
> Markus, was your concern about reading chunks of ions? I.e. you want [x1
> y1 z1 m1] [x2 …] etc.?
>
> Andy
>
>
>
> *Sent:* 27 October 2020 22:07
> *To:* atompr...@googlegroups.com
> *Subject:* Re: APT-HDF5 vote
>
>
>
> Vote Option 1 (Disseminate)
>
>
>
> I am hoping the community can give feedback on my concern, which is the
> order of the dimensions (I prefer n x 3 for example, not 3 x n which is
> in the document).
>
>
>
> Regards,
>
> Anna.
>
>
>
> *From: *atompr...@googlegroups.com
>
>
> Vote Option 1 (Disseminate)
>
>
>
> I am hoping the community can give feedback on my concern, which is the
> order of the dimensions (I prefer n x 3 for example, not 3 x n which is
> in the document).
>
>
>
> Regards,
>
> Anna.
>
>
>
> <mailto:atompr...@googlegroups.com> <atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>>
> *Date: *Tuesday, 27 October 2020 at 9:20 pm
> *To: *atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com> <atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>>
> *Subject: *APT-HDF5 vote
> To view this discussion on the web, visit
> https://protect-au.mimecast.com/s/7oHKC91WPRTQBq19tEdM-0?domain=groups.google.com.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> <mailto:atomprobe-tc...@googlegroups.com>.
> https://protect-au.mimecast.com/s/7oHKC91WPRTQBq19tEdM-0?domain=groups.google.com.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com
> <https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> <mailto:atomprobe-tc...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM
> <https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.
London, Andy
Oct 30, 2020, 3:50:10 PM10/30/20
to atompr...@googlegroups.com
See below, this is using the matlab high-level interface h5read and h5write:
P = readposMatrix(posFile); % from AtomProbeLab
P1 = P(:,1:75E6);
h5create('myfile_test.h5','/DS1',size(P1));
tic
h5write('myfile_test.h5', '/DS1', P1);
toc
P2 = P1';
h5create('myfile_test2.h5','/DS1',size(P2));
tic
h5write('myfile_test2.h5', '/DS1', P2);
toc
tic
d1 = h5read('myfile_test.h5', '/DS1');
toc
tic
d2 = h5read('myfile_test2.h5', '/DS1');
toc
-----Original Message-----
From: atompr...@googlegroups.com <atompr...@googlegroups.com> On Behalf Of Daniel Haley
Sent: 30 October 2020 18:08
To: atompr...@googlegroups.com
Subject: Re: APT-HDF5 vote
> <https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To unsubscribe from this group and stop receiving emails from it, send an email to atomprobe-tc...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/a875b1ff-9bcf-4aa0-1c66-ebd2d40af4f4%40materials.ox.ac.uk.
P = readposMatrix(posFile); % from AtomProbeLab
P1 = P(:,1:75E6);
h5create('myfile_test.h5','/DS1',size(P1));
tic
h5write('myfile_test.h5', '/DS1', P1);
toc
P2 = P1';
h5create('myfile_test2.h5','/DS1',size(P2));
tic
h5write('myfile_test2.h5', '/DS1', P2);
toc
tic
d1 = h5read('myfile_test.h5', '/DS1');
toc
tic
d2 = h5read('myfile_test2.h5', '/DS1');
toc
-----Original Message-----
From: atompr...@googlegroups.com <atompr...@googlegroups.com> On Behalf Of Daniel Haley
Sent: 30 October 2020 18:08
To: atompr...@googlegroups.com
Subject: Re: APT-HDF5 vote
--
You received this message because you are subscribed to the Google Groups "AtomProbe TC" group.
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/a875b1ff-9bcf-4aa0-1c66-ebd2d40af4f4%40materials.ox.ac.uk.
Markus Kühbach
Oct 30, 2020, 5:46:57 PM10/30/20
to atompr...@googlegroups.com
Dear Andy, dear Daniel, and others,
we do not necessarily train people how to think closer about which use cases they want to touch with this preferentially.
Quite frankly, the Matlab test here referred is not convincing to me:
Were these contiguous datasets? Yes, because this is the default behaviour when using the high-level H5 lib in Matlab.
So, then you would typically expect the HDF5 library to write ~ 0.5 - a few GB/s ...
So why does 4 x 75 mio x likely 4 B take in both strategies >20sec? This is flabbergasting, network drive in use?
Also I/O test should ALWAYS be executed multiple times plus caching is an issue
whose effects are admittedly not captured by such simple test.
1.) However, yes, the test shows that for most APT datasets speed effects are minor.
So pick the faster: 2 seconds every time you read and write will add up.2.) Do the same test for a 3D dataset. The numbers will very likely change and this is
because of the nitty gritty details of slice indexing. I have the feeling that there is a misconception of
HDF5 inasmuch as that one does not really have to worry how to shape the arrays.
Yes, different shaping works, so no necessicity in principle to worry.
But for different use cases not with the same speed.
3.) So what are the use cases how we want to read/write array data with our standard?
Think as follows, we have e.g. X, Y, Z coordinates of the recon, let's say take the 75mio dataset referred to.
Use case:Let's pick a specific range of ion sequence IDs.
Then if you store 4 x 75mio shape you will have at least four memory and disk jumps
over larger distances if you want to filter x,y,z for a specific range of ion sequence IDs.
By contrast, for 75mio x 4 interleaved HDF5 would give you a continuous block with on average fewer jumps.
Rest depends on how the file is actually layed out on the disk, a level I dont want to
touch, because to be fair, yes here hardware details matter and benchmarking with such small datasets
rather useless.
4.) Now think about we are going to store 3d data in our format, how should we train people to do it?
Let's take N>=M>=L, take a slice at fixed L: That slice (shape NxM) will be continuous.
But, now take a slice in orthogonal direction (e.g. fixed N, worst MxL): you will get substantially more jumps
for both memory and disk. So we are not talking about premature optimization, then.
So why would we then use a different indexing for 2d data? At this point I am exactly in line
with Anna's concern. Hopefully, we remain consistent.
For these reasons, I have recommended to shape N>=M (>=L) with N,M,L > 0.
By the way this is regardless of whether we use contiguous or chunked layout.
For the second the discussion is even more involved.
Bests,
Markus
----------------------------------------------------------------
Dr.-Ing. Markus KühbachMax-Planck-Institut für Eisenforschung GmbH
Department "Microstructure Physics and Alloy Design"Research group "Theory and Simulation"
Room 649
+49 211 6792 385
m.kue...@mpie.de
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM.
-------------------------------------------------
Max-Planck-Institut für Eisenforschung GmbH
Max-Planck-Straße 1
D-40237 Düsseldorf
Handelsregister B 2533
Amtsgericht Düsseldorf
Geschäftsführung
Prof. Dr. Gerhard Dehm
Prof. Dr. Jörg Neugebauer
Prof. Dr. Dierk Raabe
Dr. Kai de Weldige
Ust.-Id.-Nr.: DE 11 93 58 514
Steuernummer: 105 5891 1000
Please consider that invitations and e-mails of our institute are
only valid if they end with …@mpie.de.
If you are not sure of the validity please contact r...@mpie.de
Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
-------------------------------------------------
Daniel Haley
Oct 31, 2020, 10:19:58 AM10/31/20
to atompr...@googlegroups.com
Hi Markus,
Gb/s is quite high for many systems - here is me writing random data to
disk, using a random (non-blocking) data source, and a file as target.
$ time dd if=/dev/urandom of=benchmark bs=1M count=2048
...
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 32.6801 s, 65.7 MB/s
real 0m32,735s
user 0m0,000s
sys 0m13,578s
As you can see, random data write for me is around 65MB/s - not GB/s.
I'd hazard none of Andy's benchmark are unexpected, and could be IO
bound, rather than bound at the data encoding stage (though more tests
via fopen + random data on Andy's system would be needed).
Ions 7.50E+07
Size/Ion 16
Bytes 1200000000
KB 1171875
MB 1144.4091796875
GB 1.11758708953857
Seconds 28.77
MB/s 39.7778651264338
I'd be interested to see the values for (2) , and if there is a big
difference for (4), why does this not show for Andy's test? Why would
the data not be contiguous?
I am having trouble following your explanation, and a code sample would
greatly help to narrow down your concern here, and allow us to all
reproduce the exact problem.
I've added experimental code to libatomprobe to help provide an
interface to these files, so you can see the sample code I am using
there to do this access.
Thanks,
Dan
> *From: * "London, Andy" <andy....@ukaea.uk>
> *To: * "atompr...@googlegroups.com" <atompr...@googlegroups.com>
> *Sent: * 10/30/2020 6:29 PM
> *Subject: * RE: APT-HDF5 vote
> <mailto:atomprobe-tc...@googlegroups.com>.
> <https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com?utm_medium=email&utm_source=footer>.
> <https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.
>
>
>
> ------------------------------------------------------------------------
> <https://groups.google.com/d/msgid/atomprobe-tc/346570875-916%40xmail1.mpie.de?utm_medium=email&utm_source=footer>.
Gb/s is quite high for many systems - here is me writing random data to
disk, using a random (non-blocking) data source, and a file as target.
$ time dd if=/dev/urandom of=benchmark bs=1M count=2048
...
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 32.6801 s, 65.7 MB/s
real 0m32,735s
user 0m0,000s
sys 0m13,578s
As you can see, random data write for me is around 65MB/s - not GB/s.
I'd hazard none of Andy's benchmark are unexpected, and could be IO
bound, rather than bound at the data encoding stage (though more tests
via fopen + random data on Andy's system would be needed).
Ions 7.50E+07
Size/Ion 16
Bytes 1200000000
KB 1171875
MB 1144.4091796875
GB 1.11758708953857
Seconds 28.77
MB/s 39.7778651264338
I'd be interested to see the values for (2) , and if there is a big
difference for (4), why does this not show for Andy's test? Why would
the data not be contiguous?
I am having trouble following your explanation, and a code sample would
greatly help to narrow down your concern here, and allow us to all
reproduce the exact problem.
I've added experimental code to libatomprobe to help provide an
interface to these files, so you can see the sample code I am using
there to do this access.
Thanks,
Dan
> *To: * "atompr...@googlegroups.com" <atompr...@googlegroups.com>
> *Sent: * 10/30/2020 6:29 PM
> *Subject: * RE: APT-HDF5 vote
>
> Hi all,
>
> I did a quick test in matlab just to check that the order didn’t
> matter for writing real data:
>
> Write 4 by 7.5E7 ions Elapsed time is 28.779395 seconds.
>
> Write 7.5E7 by 4 ions Elapsed time is 26.822038 seconds.
>
> >> hdf5_speed_test
>
> Read 4 by 7.5E7 ions Elapsed time is 1.156697 seconds.
>
> Read 7.5E7 by 4 ions Elapsed time is 1.058228 seconds.
>
>
>
> Markus, was your concern about reading chunks of ions? I.e. you want
> [x1 y1 z1 m1] [x2 …] etc.?
>
> Andy
>
>
>
> Hi all,
>
> I did a quick test in matlab just to check that the order didn’t
> matter for writing real data:
>
> Write 4 by 7.5E7 ions Elapsed time is 28.779395 seconds.
>
> Write 7.5E7 by 4 ions Elapsed time is 26.822038 seconds.
>
> >> hdf5_speed_test
>
> Read 4 by 7.5E7 ions Elapsed time is 1.156697 seconds.
>
> Read 7.5E7 by 4 ions Elapsed time is 1.058228 seconds.
>
>
>
> Markus, was your concern about reading chunks of ions? I.e. you want
> [x1 y1 z1 m1] [x2 …] etc.?
>
> Andy
>
>
>
> *From:*'Anna Ceguerra' via AtomProbe TC <atompr...@googlegroups.com>
> *Sent:* 27 October 2020 22:07
> *To:* atompr...@googlegroups.com
> *Subject:* Re: APT-HDF5 vote
> *Sent:* 27 October 2020 22:07
> *To:* atompr...@googlegroups.com
>
>
>
> Vote Option 1 (Disseminate)
>
>
>
> I am hoping the community can give feedback on my concern, which is
> the order of the dimensions (I prefer n x 3 for example, not 3 x n
> which is in the document).
>
>
>
> Regards,
>
> Anna.
>
>
>
>
>
> Vote Option 1 (Disseminate)
>
>
>
> I am hoping the community can give feedback on my concern, which is
> the order of the dimensions (I prefer n x 3 for example, not 3 x n
> which is in the document).
>
>
>
> Regards,
>
> Anna.
>
>
>
> *From: *atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>
> <atompr...@googlegroups.com <mailto:atompr...@googlegroups.com>>
> *Date: *Tuesday, 27 October 2020 at 9:20 pm
> *To: *atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>
> <atompr...@googlegroups.com <mailto:atompr...@googlegroups.com>>
> *Subject: *APT-HDF5 vote
> <mailto:atompr...@googlegroups.com>
> <atompr...@googlegroups.com <mailto:atompr...@googlegroups.com>>
> *Date: *Tuesday, 27 October 2020 at 9:20 pm
> *To: *atompr...@googlegroups.com
> <mailto:atompr...@googlegroups.com>
> <atompr...@googlegroups.com <mailto:atompr...@googlegroups.com>>
> To view this discussion on the web, visit
> https://protect-au.mimecast.com/s/7oHKC91WPRTQBq19tEdM-0?domain=groups.google.com.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to atomprobe-tc...@googlegroups.com
> <mailto:atomprobe-tc...@googlegroups.com>.
> https://protect-au.mimecast.com/s/7oHKC91WPRTQBq19tEdM-0?domain=groups.google.com.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to atomprobe-tc...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com
> <https://groups.google.com/d/msgid/atomprobe-tc/ME2PR01MB38285017B1791E4D7D086B58B8160%40ME2PR01MB3828.ausprd01.prod.outlook.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to atomprobe-tc...@googlegroups.com
> <mailto:atomprobe-tc...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to atomprobe-tc...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM
> <https://groups.google.com/d/msgid/atomprobe-tc/LNXP265MB105013CA1079BD4C0C0DAAD785150%40LNXP265MB1050.GBRP265.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>.
>
>
>
> ------------------------------------------------------------------------
> -------------------------------------------------
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are
> only valid if they end with …@mpie.de.
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -------------------------------------------------
>
> Max-Planck-Institut für Eisenforschung GmbH
> Max-Planck-Straße 1
> D-40237 Düsseldorf
>
> Handelsregister B 2533
> Amtsgericht Düsseldorf
>
> Geschäftsführung
> Prof. Dr. Gerhard Dehm
> Prof. Dr. Jörg Neugebauer
> Prof. Dr. Dierk Raabe
> Dr. Kai de Weldige
>
> Ust.-Id.-Nr.: DE 11 93 58 514
> Steuernummer: 105 5891 1000
>
>
> Please consider that invitations and e-mails of our institute are
> only valid if they end with …@mpie.de.
> If you are not sure of the validity please contact r...@mpie.de
>
> Bitte beachten Sie, dass Einladungen zu Veranstaltungen und E-Mails
> aus unserem Haus nur mit der Endung …@mpie.de gültig sind.
> In Zweifelsfällen wenden Sie sich bitte an r...@mpie.de
> -------------------------------------------------
>
> --
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> <mailto:atomprobe-tc...@googlegroups.com>.
> You received this message because you are subscribed to the Google
> Groups "AtomProbe TC" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to atomprobe-tc...@googlegroups.com
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/atomprobe-tc/346570875-916%40xmail1.mpie.de
> <https://groups.google.com/d/msgid/atomprobe-tc/346570875-916%40xmail1.mpie.de?utm_medium=email&utm_source=footer>.
London, Andy
Oct 31, 2020, 10:20:28 AM10/31/20
to atompr...@googlegroups.com
Hi Markus,
It was more a check to see if there was a “big” difference, I wasn’t expecting one in this case.
Here are some repeats (results are the same order as before) on my EVO 850 SSD:
Elapsed time is 25.275998 seconds.
Elapsed time is 32.303804 seconds.
Elapsed time is 1.458578 seconds.
Elapsed time is 2.039367 seconds.
Elapsed time is 31.956768 seconds.
Elapsed time is 34.489801 seconds.
Elapsed time is 12.113823 seconds. ?? Not sure what happened here, repeated and got 1.2s for both
Elapsed time is 10.779811 seconds. ??
My mechanical drive seems to have similar or slightly faster write times, but much slower read:
write
Elapsed time is 23.640609 seconds.
Elapsed time is 29.379307 seconds.
read
Elapsed time is 25.644544 seconds.
Elapsed time is 21.093204 seconds.
(read repeat)
Elapsed time is 26.773547 seconds.
Elapsed time is 25.577893 seconds.
I think this is all rather in the detail. I would be willing to blame matlab quirks for not writing at full speed (windows recons I’m only writing at 12% capacity on the ssd?).
It all depends what the use case actually is, and whether it will have a significant impact or not. I would say the write speeds are similar enough, same for read, although 4xn was consistently faster than nx4.
If someone wants to benchmark the C++ implementation – be my guest!
But I do think we should try and get it right first time if we can. It would be annoying to have to be backwards compatible because of some silly mistake.
If the specification changes then we can check using the version to make sure readers can see if the version is supported.
Andy
To view this discussion on the web, visit https://groups.google.com/d/msgid/atomprobe-tc/346570875-916%40xmail1.mpie.de.
Reply all
Reply to author
Forward
0 new messages