miniSEED: progress on a new FDSN standard
Goran Ekstrom
About a year ago (August 6, 2016) I wrote to you concerning the
recognized current need for an updated miniSEED format, and in support
of the ongoing efforts in WGII to develop a new FDSN standard. While
I did not attend the Utrecht workshop on this topic earlier this year,
I understand that it was productive and that much progress has
been made on the detailed definition of a new standard.
I now write again to request that you use the Kobe meeting to
take the next steps toward adoption and implementation of a new
FDSN standard.
My understanding is that a nearly complete, though not final,
specification for a new format - miniSEED3 - exists and has been
discussed by the team of technical developers identified at the
meeting in Utrecht. Input came from a number of groups that
commented on the initial strawman proposed by IRIS, and modifications
were made based on input from the Utrecht meeting and through further
discussions in the technical evaluation team.
I will not delve into, or argue for, specific features of the
proposed format. Most of you are more familiar with its details
than I. I will, however, highlight some important aspects of the
new format, as presented by its developers and proponents:
1. It solves the imminent data-center need to identify and
accommodate larger numbers of networks, stations, and sensors.
[Some modern data sets relevant for the FDSN community cannot be
accommodated with the current miniSEED format. ]
2. Conversion of miniSEED2 to miniSEED3, for archiving or
distribution, is simple and lossless.
3. Networks and data centers that do not need the new features
offered by miniSEED3 may not see any benefit in adopting miniSEED3
and can continue relying on miniSEED2 and their current
infrastructure in their operation.
4. By design, miniSEED3 is not intended to be a low-latency
real-time data format. It primarily addresses issues in data archiving,
data management and data exchange.
Point (1) addresses the critical issue that some data centers have
to deal with now. Having an FDSN standard will strongly encourage
data centers to implement the same standard, facilitating continued
sharing of software and development.
Points (2-3) make it clear that the new format does not impose a
burden on those data centers and networks that do not need miniSEED3.
Point (4) clarifies the limitations of the scope of miniSEED3 and
identifies the separate need for an effort to clarify/develop
standards in the field recording and transmission of data and
to address related issues, such as data latency.
I urge you to use the WGII meeting in Kobe to review and consider the
miniSEED3 format proposal. While doing so, I hope you will not forget
the strategic benefits of having an FDSN standard and also to consider
the risks associated with not adopting a standard. I also hope that your
deliberations will lead to a recommendation that the FDSN Steering
Committee can consider during the second plenary meeting.
My assessment is that the FDSN is ready to, and needs to, move forward
with a solution.
Best regards,
Göran
Reinoud Sleeman
the outcome of the successful FDSN WG2 meeting in the Netherlands, earlier this year, on the future of mini-SEED is
a White Paper which describes the ongoing discussion within WG2 on the motivation and vision for a new format, the
identification of current needs for a new format, a description of several proposals and a proposal for the way forward.
This White Paper, which is attached to this e-mail, will be the basis for further discussion during the FDSN WG2 meeting
in Kobe. At this moment, however, there is no consensus yet about the adoption and implementation of a new FDSN
standard. Feedback from several stakeholders show very strong, different opinions which reflect that the FDSN community
is not ready for a new format at this stage.
In order to get broader support to work towards a new format, the FDSN should first reflect on and take decisions on the
following question:
- What is the scope of FDSN? Does the FDSN envision/support the need to prepare our infrastructure for large N experiments,
and other more complex experiments, often with lower quality sensors, that may go beyond seismic data ?
- Can we solve the identification limitation without major overhaul ?
Introducing an disruptive change of the format has an impact on all stakeholders. Do we accept version incompatibility ?
Because of the different opinions, that we need to respect, and the potential big impact on the current seismological infrastructure
a careful process must be followed. We only can move forward when the motivation for a new format is well understood, accepted and
supported by the FDSN. The white paper proposes a path forward, starting with a requirements list (to be reviewed and approved by WG2)
and the setup of a Technical Working Group to implement draft realisations of the different proposals with the aim to identify
problems in an early stage and evaluate performance. If the FDSN supports the design of a new format it will have to be a long
term sustainable format and this takes time to design it carefully, and even can be more ambitious than the current proposals.
Most likely, SEED was not build in a short time either.
Finally, as soon as the FDSN supports the motivation for a new format it is required to define a WG2 agreed requirements
list for the new format. This must be the starting point for the evaluation of the different proposals. That requirements list
will take away current differences in views and opinions of what the new format should be designed for (e.g. supporting
real-time or not).
Please find attached the White Paper: The Future of mini-SEED.
Best regards,
Reinoud
_______________________________________
From: fdsn-wg2-d...@lists.fdsn.org<mailto:fdsn-wg2-d...@lists.fdsn.org> [fdsn-wg2-d...@lists.fdsn.org] on behalf of Goran Ekstrom [eks...@ldeo.columbia.edu]
Sent: 21 July 2017 19:54
To: FDSN Working Group II
Subject: [fdsn-wg2-data] miniSEED: progress on a new FDSN standard
Best regards,
Göran
----------------------
FDSN Working Group II (http://www.fdsn.org/message-center/topic/fdsn-wg2-data/)
Sent from the FDSN Message Center (http://www.fdsn.org/message-center/)
Update subscription preferences at http://www.fdsn.org/account/profile/
Philipp Kästli
i'd like to add a few comments to the discussion of the next generation waveform format:
a) Much of the variability of the 5 proposals are, imho, the result of different assumptions on the scope of the development: Should we develop for
- FDSN only (mostly measured data from +/- long living stations, organized in defined networks, with a community organized well enough to be able to rely, to some extent, also on conventions)
- Seismology in general (including any type of waveform modelling, and any type of motion sensors, also poorly identified or very mobile ones) This requires additional flexibility for identifying streams from instruments not organized in networks or
- Regular environmental time series in general (leveraging the possibility of software and data exchange with other domains). This requires a better separation of time series and domain metadata and probably an extension of temporal scopes)
ð In order to focus further discussion, I'd like to invite WG2 to take a decision on this.
b) The question of (backward) compatibility
Miniseed2 started from a subset of seed, with seed covering all waveforms, data quality metadata, instrumentation metadata, and event information. In the recent years, separate formats and services for these topics have developed and were adopted as standards, as it proved to be more lightweight and flexible to handle different types of information separately: mseed for waveforms, QuakeML BED for event information, StationXML/InventoryXML for instrumentation metadata, and Mustang/WFcatalog with their respective delivery formats for data quality metadata. So when calling for compatibility of mseed3, we should more aim at the semantically correct distribution of information into these four domains than to maintain decisions inherited from full seed. Otherwise we risk overlaps/collision or gaps with regards to the other domain formats and respective services
ð I'd like to invite WG2 to aim for a future proof affiliation of information to these different domains and the respective formats & services, even at the price of breaking formal compatibility with seed/mseed2.
c) Application scope and requirements list
There seems to be an implicit agreement that the new format should cover different application cases (near-real time transfer, economic storage, efficient seek and retrieval), and that the format should accommodate for this with different representation variations (including variable block size, compression, and variable alignment [storage or time]). However, we define only a file format but none of I transfer protocol, II subformat conversion rules, or III data access strategy.
ð In order to avoid ambiguities or shortcoming in these different applications, I'd like to propose 3 add-ons to the requirements list:
-- forward writeability: You should be able to start writing or transferring a record before having acquired or read all data intended to be added to it. This detaches the transfer protocol from the block structure (I) and reduces the needs for subformat conversion. It disables for data integrity (e.g. crc) in block headers, as those are available only with all data available.
-- content invariance with subformat conversion. This allows to losslessly recode the same information in different subformats. It disallows for features IMplicitly referring to the edges, or extent, of a data block (as this would change such implicit information)
-- explicit alignment strategy. This allows the user to see whether block size is aligned to storage units or time intervals, and whether or not it is subject of change between blocks of the same data source & stream. It requires subformat indicators, and potentially stream ID aliases
Kind regards,
Philipp
Jean-Marie Saurel
I just wanted to add a few comments to some of your points, mainly the
first one.
> a) Much of the variability of the 5 proposals are, imho, the result
> of different assumptions on the scope of the development: Should we
> develop for
>
> - FDSN only (mostly measured data from +/- long living
> stations, organized in defined networks, with a community organized well
> enough to be able to rely, to some extent, also on conventions)
>
> - Seismology in general (including any type of waveform
> modelling, and any type of motion sensors, also poorly identified or
> very mobile ones) This requires additional flexibility for identifying
> streams from instruments not organized in networks or
>
> - Regular environmental time series in general (leveraging the
> possibility of software and data exchange with other domains). This
> requires a better separation of time series and domain metadata and
> probably an extension of temporal scopes)
SEED standard (and therefore miniseed 2.4) already applies to such
general use, by defining channel codes for almost every geophysical or
environmental sensor, which are often in the vicinity of seismic
stations and may be useful to the scientist to interpret the seismic data.
The appendix A at least already list :
- high quality and low quality seismometers (high and low gain, short
period and broadband, geophones);
- accelerometer and gravimeter;
- tiltmeter, creepmeter and strainmeter;
- pressure, humidity, temperature, rainfall, cloud and wind sensors;
- tide and water current sensors;
- electronic test points and magnetometer.
Some of the miniseed 2.4 users might already use those possibilities (we
do at IPGP) and, as you mention, the possibility to use the same data
exchange softwares.
The new standard, if there is one, should at least cover the same range
of instruments and use cases as the actual miniseed 2.4.
> ð In order to focus further discussion, I’d like to invite WG2 to take
> a decision on this.
>
>
>
> b) The question of (backward) compatibility
> Miniseed2 started from a subset of seed, with seed covering all
> waveforms, data quality metadata, instrumentation metadata, and event
> information. In the recent years, separate formats and services for
> these topics have developed and were adopted as standards, as it proved
> to be more lightweight and flexible to handle different types of
> information separately: mseed for waveforms, QuakeML BED for event
> information, StationXML/InventoryXML for instrumentation metadata, and
> Mustang/WFcatalog with their respective delivery formats for data
> quality metadata. So when calling for compatibility of mseed3, we should
> more aim at the semantically correct distribution of information into
> these four domains than to maintain decisions inherited from full seed.
> Otherwise we risk overlaps/collision or gaps with regards to the other
> domain formats and respective services
>
> ð I’d like to invite WG2 to aim for a future proof affiliation of
> information to these different domains and the respective formats &
> services, even at the price of breaking formal compatibility with
> seed/mseed2.
>
Could this affiliation of information could serve as a replacement for
the full SEED volumes ?
Regards.
Jean-Marie SAUREL.
> *From:*fdsn-wg2-d...@lists.fdsn.org
> [mailto:fdsn-wg2-d...@lists.fdsn.org] *On Behalf Of *Reinoud Sleeman
> *Sent:* Mittwoch, 26. Juli 2017 23:23
> *To:* FDSN Working Group II
> *Subject:* [fdsn-wg2-data] miniSEED: progress on a new FDSN standard
> real-timeor not).
--
--------------------------------------
Institut de Physique du Globe de Paris
Observatoires Volcanologiques
1 rue Jussieu
75238 Paris Cédex 05
+33(0)183957437
France