Notes from ICT-ISS meeting relevant to IPET-MDI

[Jeremy]

All - I wanted to make sure that you all had site of the discussion points from the recent ICT-ISS meeting in Geneva. Below are my notes captured from my perspective, with a bias toward information relevant to IPET-MDI. Formal meeting minutes will be available soon. For reference, the meeting wiki page (with attached documentation) is here:  http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=ict_iss-meeting2012#Decisions_related_to_the_work_plan_for_OPAG_ISS

Things marked with !!! are actions or part of the work plan for IPET-MDI in the next CBS session.

--- Informal meeting minutes ---

--- Introduction ---
- Focus on cross cutting issues for OPAG ... Especially structure for next OPAG-ISS
- Mission for meeting is to make recommendations to CBS Management Group.
- Bring out the (contentious) discussion points (only - please!)

--- ET-OI ---
CLIMAT reports ...
Region 1 has _MANY_ 'silent' stations.

Issue with CLIMAT report is that it is only created once per month; typically this is done via database extract, but developing nations do this manually & hence it is error prone.

Issues remain regarding the migration to Table-Driven Codes

[PROPOSAL - not actioned] develop Application Schema for Climate Observation and associated GML schema. Allow transmission of information in XML form as complementary transmission format. XML is far easier to validate than TAC. Could also develop a Rubric to critique XML CLIMAT reports. (and publish online validation tool?)

Issue remains the human error ... No matter what form is used, manual input once per month is subject to human error. (Kelvin Wong)

Need to work with CCl to develop procedures for improving operator capability. A major improvement would be delivered through timely feedback of errors to originating centre (e.g. days or shorter) enabling issues to be resolved in situ. Current feedback timescale is in excess on 1-month.

Also note that commercial tool vendors are not always compliant with the code form. WMO need to work with HMEI to improve compliance [need to assess whether commercial products are correctly encoding content; a 'testbed' for certifying tool compliance (WMO can then issue a 'kite-mark' for compliant tools)]. ET-OI already commented that improvement in tool validation is required, but did not consider (table-driven) XML as a potential solution to ease validation using ubiquitous technology. [bypass binary TDCF and move directly to 'table-driven' XML ... like Aviation community] ... but the main issue remains supporting the manual input of CLIMAT reports (Dave Thomas)

(Discussion occurs regarding incorporation of CLIMAT message monitoring in GISC responsibilities)

Maintenance of VolC1 ...
[QUESTION] is acceptable for 'transitional' VolC1 (derived from WIS DAR Catalogue) to standardise the textual information in the 'Comment' section. Systematic creation of VolC1 from WIS Discovery Metadata records will result in standardised VolC1 records; e.g. All GRIB and all SYNOP records will be described consistently (contrary to current practice).
[ANSWER: yes]

(Dave Thomas) Refer back to the original technical regulation for content. Note that VolC1 has been added to over the years (using the comments).

WMO Core Metadata Profile specification does not currently include formal requirements for specifying metadata required to populate VolC1 for GTS bulletins. Initial development from ApMD will provide Guidance for GRIB and SYNOP.

We note that BUFR and GRIB messages contain a variety of content; creating a standardised textual comment for ALL GRIB  and BUFR messages may not be possible. We need to express the _CONTENT_ of the messages; not be concerned with standardising information for VolC1 based on data format.

We also noted the increasing variety of information being exchanged by other Technical Commissions (outside the operational World Weather Watch); netCDF, GML, etc. for CCl, CAeM, ...

Will WMO Core Metadata Profile specification include these formal requirements in time for retirement of VolC1 in 2015?

Focus on development of WIS Discovery Metadata & WIS DAR Catalogue from which VolC1 can be populated. Don't try to fix existing VolC1.

!!![PRIORITY REQUIREMENT (MDI)] WMO Core Metadata Profile shall formalise the requirements for specifying metadata required to populate VolC1 for GTS bulletins.
(... first create Guidance & validate effectiveness, then migrate to formal requirement in WMO Core Metadata Profile specification)

WIS Monitoring ...
Over next CBS-period develop and implement practical monitoring tools and procedures for user, information & system SLAs & other metrics (e.g. usage) within WIS.

--- ET-CTS ---

--- IPET-DRC ---

--- IPET-MDI ---
Metadata Standard
1) xlinks proposal generated significant debate; recommendation that this activity is prioritised behind the generation of guidance material and other documentation helping the adoption of metadata.
2) debate on specification of temporal extent for transient data products exchanged on WIS; resulted in new post to Google Group.

!!!Recommendation that GISCs support 3-versions of the WMO Core Metadata Profile was accepted & shall be submitted as an amendment to WMO No. 1060 Manual on WIS.

!!!Ongoing education and outreach programme to support the up-take of effective metadata management by Members (DCPCs, NCs).

!!!Urgent requirement to develop Guidance defining the metadata elements required to support population of VolC1 from the WIS DAR Catalogue.

!!!Ongoing requirement to develop Guidance (and associated automated validation tests; "Rubric") in response to emerging inconsistencies in metadata style observed within the WIS DAR Catalogue.

!!!Ongoing capture of requirements for _DISCOVERY_ metadata from (other) WMO Technical Commissions. [priority list?] ... May result in fast-track changes to WMO Core Metadata Profile specification.

!!!Define set of reference keyword thesauri which metadata authors are encouraged to use; externally managed resources such as GEMET or GCMD 5.3.3-2006 (hosted @ NERC Vocab server) and WMO CodeLists that may be published via the proposed WMO Registry.

!!!Development of metadata analysis package that can be deployed @ GISCs or other centres to analyse and report on metadata compliance with formal requirements and guidance. (based on currently methodology deployed at NOAA/NGDC)

???Responsibility for routine monitoring of metadata style & compliance

???Procedure for updating WIS Discovery Metadata records which include additional language translations provided by external agents.

!!!Proposal accepted to release WMO Core Metadata Profile v2 @ CBS 2014 supporting ISO 19115-1:2013, ISO 19115-2:2009, ISO 19157:2013 & ISO 19115-3:2013. Need to establish work activity in collaboration with GISC implementors to develop v2.0 specification. v2 is planned for harmonisation with JCOMM Marine Metadata Profile.

!!!Develop proposals for use of xlinks within WMO Core Metadata Profile; working in collaboration with GISC and/or DCPC to validate proposals.

Data Modelling
- DRC noted that it is their expectation that BUFR Edition 5 should employ the model-driven approach.

- If a Programme has a requirement for exchanging a specific set of data elements in a given way (e.g. synoptic observing requirements) ... then there is benefit to providing Regulatory Status to the Sequence ... e.g. Requiring the use of a specified Sequence (Template) when exchanging data in BUFR in response to that Programme's observing requirement ... Information will always be exchanged including specific elements in a standardised structure.
People can add additional elements; the Template is the minimum requirement. BUFR Section 3 would include the Template plus _ADDITIONAL_ sequences / templates as required.
- Critical precursor to specifying the regulated Template would be the specification of formal requirement by the originating Programme.

- Currently premature to change the Regulation; but we should investigate further.

Note: the update request for participation of other Technical Commissions in the further development of the model-driven approach to refer to ALL Programmes "especially Hydrology and Climatology".

Note: the relationship between the Logical Data Model and the formal programme requirement still needs further explanation; discussion indicates a lack of understanding. Each formal requirement will result in the development of an Application Schema that is built on the 'foundation' provided by the WMO  METCE Logical Data Model (like the Aviation OPMET Application Schema: ICAO MetCC). Physical Encodings can then be derived from each Application Schema in the necessary formats; GML, BUFR etc.

!!!Recommendation to continue development of the Logical Data Model accepted; including inviting participation from all WMO Technical Commissions (noting explicit statement that there is no plan to retrofit existing operational TDCF message types with a Logical Data Model / Application Schema). Task includes developing developing automated process to derive TDCF Templates from Logical Data Model (noting expectation that BUFR Edition 5 should employ the model-driven approach). Further consideration is required regarding proposal to change to Technical Regulation to require the use of Sequences / Templates for exchanging data meeting a formal WMO Programme requirement.
!!!Scope of Logical Data Model development: Aviation OPMET, Climate Observation. (Application Schema for Climate Observation should include a conceptual model for observing stations; which may provide opportunity to update WMO No. 9 VolA as a formal (ISO compliant) gazetteer)

!!!Assess how data compliant with WMO METCE can be communicated via Common Alerting Protocol (CAP)

!!!Develop proposals for referencing CodeLists from Logical Data Model that meet a specific requirement where the TDCF Code-tables offer a broader set of options ... In support of TT-AvXML requirement for ICAO.

!!!In collaboration with OGC & ISO/TC 211, develop proposals for 'pathway' to take thematic standards for Weather, Water & Climate to international standard status. (test case - WaterML2)

Registers ...
Additional method of publishing the information in the TDCF Code-tables ...

Codes need to be published with HTTP URI identifiers in the wmo.int domain so that they can be explicitly referenced from XML formatted data (ICAO requirement to support bilateral exchange of OPMET information via GML)

>> further pilot should occur in the _wmo.int_ domain ... the reference.metoffice.gov.uk identifiers should not become common usage.

Should these HTTP URIs be human-readable? Note development of short-names by IPET-DRC; IPET-DRC have 6 or 7 requirements for these short-names. Possible synergy in activities.

These HTTP URIs do not necessarily resolve to web pages or other resources ... but it is best practice to do so.

Registry is what would resolve the HTTP URI to a meaningful definition.

Choice of RDF payload and HTML presentation consistent with best practice.

ICAO requirement remains a priority ... Should we constrain the register scope to _ONLY_ ICAO requirements? Or do things in a way that supports broader community adoption.

Other requirements are seen from ET-WISC (keyword thesaurus management and publication) and INSPIRE AC-MF Data Specification (parameter and statistical operator) ... Noting that INSPIRE is not a globally applicable requirement. INSPIRE AC-MF have already begun publishing fragments of WMO code-tables in web-accessible form.

WMO currently maintains the authoritative copy of the code tables in a relational database. The web accessible registers can be derived from this. These web resources may be hosted at GISCs for high-availability deployment. However, we note that the WMO database does not include versioning of the code tables and code entities. ... discussion required ...

The register may support reduction in Secretariat effort through automation of generation of code-table documentation. Currently, the code-table _DOCUMENT_ is the authoritative source from which the database and machine readable versions (csv, XML) are created. The documentation _COULD_ be generated from the machine readable information repository.

This is consistent with the WMO Secretariat drive to use automation to reduce effort for maintaining WIS components.

No significant benefit for this approach for IPET-DRC; TDCF formats for World Weather Watch are closely coupled so there the current form of the code tables are adequate. The benefit arises from publishing these codes for use in the broader community.

Whilst no objections were raised regarding the technical proposal, concerns are raised that this activity may divert resources from higher priority initiatives. Also need to be sure that WMO resources are being allocated to requirements that are representative of WMO Member needs.

!!!Summary:
- meet aviation requirement (noting need to deploy an operational Registry to support Amendment 76 of ICAO Annex 3 in mid-2013); publish codes and code-tables in wmo.int domain [need to identify (sub)domain and path within wmo.int domain that can be mapped to external Amazon Elastic IP]
- in a way that is extensible ... allowing us to meet requirements from other communities as they emerge; engage with WMO community to ensure requirements are captured (e.g. as proposed by Ator @ IPET-DRC 2012, use by decoding software components, or deployment for operational systems without Internet connection etc.)
- investigate use of short-names for BUFR descriptors developed by ECMWF within 'human-readable' HTTP URIs
- include publication of WMO CodeLists for metadata within; supporting requirement from ET-WISC for keyword thesaurus management*
- estimate the effort for 100% coverage based on effort to meet aviation requirement
- investigate use of OGC resources through Met-Ocean DWG to help achieve 100% coverage

 * determine which TDCF CodeLists are desirable to reference from WIS Discovery Metadata.

Potential collaborators include: Unidata (Caron), ECMWF (Fucile), NCAR (Burek), UK Met Office (Tandy et al) ... others?

(UK Met Office needs to publish existing software components for web-accessible registers under appropriate open-source license)
(Need further development of the management interface / UI for making changes to the Registers and RegisterItems ... Look at BODC procedures that HTTP POST updates as RDF/XML payload documents; could develop web-form(s) to provide a non-XML UI?)

>> Deploying Change/Defect management tools (e.g. Redmine, as presented by Hiro) is a separate issue; however, a variant of the ISO 19135 process still needs to be implemented to manage change to the Register.

!!!Discussion about proposals for creation of new Common Code-table to happen by correspondence over next 2-weeks between DRC & MDI ... noting requirement from TT-AvXML regarding common parameter definitions to support ICAO

--- Station Identifiers ---

(see IPET-DRC meeting report for more information: section 8.2 WMO station index "universal station identifier")

For TDCF: [Operator identifier]+[station identifier] ... 12 digits.

- Operator identifier; 3-char/10-bit  ... Code table
- Numerical station identifier (10^9 available) ... Numeric
- Character station identifier (for convenience of the operator) (9 character) ... Character

Will these identifiers be registered? ...
WMO No. 9, Volume A

To support legacy TAC station identifiers (where available numbers are constrained by lexical scope) ... see WMO No. 306 Vol I1 Section D 'System of Station Index Numbers': station index number (5-digits, primary key used in Vol A), these are geographically allocated in blocks (block number is first 2-digits of station index number)

- reallocate spare station identifiers upon application, irrespective of geographic location ...
- removal of geographic meaning to station block numbers means that a convenient lookup reference will be required to figure out the location
>> web-accessible register! ... Relates to potential creation of an Application Schema on Climate Observation including definition of observing stations.

--- ET-GDDP ---

--- Structure of OPAG-ISS ---

- Option 2 selected.
- Chair & co-chair have flexibility to select core & associate members based on skills requirement.
- Core / Associate member designation has implication of committed effort from the Expert, supported by the PR.
- Funding for attendance of meeting allocated to those who the Chair /Co-chair designates as (most) necessary to participate in the face to face discussions.
- Task-teams can be established as identified by Chair / Co-chair. Approval of ICT-ISS (or CBS Management Group) required if it is anticipated that the Task-team needs funding (e.g. for face to face meeting).
- Chairs / co-chairs may invite participation of individuals with particular skills that are needed to help Expert Teams to meet their objectives.

!!!Discuss Terms of Reference (the 'colourful' document) for each of the Groups defined in option 2 ... using CBS ISS Chairs Google Group; deadline 1st July
(note - some ToRs are 'shared' ... need to review this more carefully)

--- BUFR Edition 5, GRIB Edition 3 ---
BUFR 5
- driven from Logical Data Model
- uniquely identified Tables; resolving the local-table issue
- Gil's points ...
... but still far off; avoid adverse impact to TDCF Migration
GRIB 3
... lack of strong requirement _NOW_ to publish a new edition of GRIB; again, new edition is 'far off'
Common Parameter tables across GRIB & BUFR.

(note that ET-ADRS recommended use of GRIB & BUFR for operational exchange within World Weather Watch, and recommended the use of netCDF4 / HDF5 for non-operational data exchange as it allows flexibility)

--- Tooling: change/defect management, wiki, version control repository ---

- current situation:
> WIS wiki (tiki-wiki)
> JMA pilot of Redmine
> sourcerepo.com (other options available; sourceforge; GitHub repository) ... Version control repository currently used by Timo (subversion etc.); WMO would be able to commit to providing this capability

- would be nice to migrate WIS wiki to something more stable, with better UI
- WMO secretariat 'should' be getting a content management system; this may replace some of the WIS wiki capability

!!!Set up source code repository (for content published under open-source license) for IPET-MDI ... sourceforge or GitHub or some other 3rd party hosting
???Would like endorsement of CBS to use WMO logo for the repository ...
- Timo suggests using a commercial service under the wmo.int domain
!!!Advise Director (WIS) of cost implications for such an arrangement

- the repository is a core component of WIS; long-term access to assets used to help validate WIS Discovery Metadata records
- require access via web-browser & 'remember me' password management

>> no consensus on endorsement of WMO / CBS regarding the repository, set up something locally for IPET-MDI

!!!Redmine change/defect tracking management tool piloted by JMA will be further developed by JMA; aim to help develop understanding of how such a tool may be used to support management of change to WMO Data Representations and WMO Core Metadata Profile.
!!!Discuss with Eiji & Jitsuko