status of the ioos_qc QARTOD python package?

[John Kerfoot]

Hello,

The IOOS Glider DAC is currently working on the application of QARTOD tests to the real-time (eventually delayed-mode) data sets that are submitted by our data providers.  We were under the impression that the ioos_qc package was developed (and is maintained) exactly for this type of application.

However, upon running the example suite on a few of our T/S data sets, we're seeing that a large portion (>80% or so) of records we know to be good are being flagged as suspect (or worse).  The results are particularly poor for the:

1. flat line

2. rate of change

3. spike test

It also seems that the tests above do not work correctly if there is even a single _FillValue contained in the source data. Does this mean that we need to remove any/all rows that contain a _FillValue prior to running one or more of the tests?

Our plan was to run a gross range and above tests and then to roll them up into an aggregated test result that could then be used by downstream users to decide on how much of the profile is released. However, for the results we're getting, this would result in essentially all of our profiles NOT being deemed valid.

We're wondering the following:

1. Is the ioos_qc package the IOOS endorsed package for applying QARTOD QC?

2. Is the package maintained? The last official release appears to be v2.1 (October 2022) and we have submitted issues/questions (years ago) that have not been addressed or followed up on. Is this package obsolete and should we be using something else or rolling up our own?

3. Who would be the appropriate person/people to contact for specific questions on implementation and results?

4. If there is a person or group from #3, might I suggest a DMAC webinar in which someone could walk the community through some examples using our glider data so that we can determine if we are doing things correctly or not?

5. Are we better off writing our own qc suite specific to our use case?

Thanks in advance.

[Mathew Biddle - NOAA Federal]

Hi John,

Thank you for starting this conversation.

The ioos_qc package has been established as a resource to the IOOS community for implementing quality assurance and quality control for oceanographic datasets.  IOOS supports expanding ioos_qc to meet as many use cases for oceanographic data QA/QC as possible.  We’re actively investigating options to better support the further development of this package and appreciate any recommendations from the community about how this can best be accomplished.  

IOOS’ preference is that the community leverage the work done in ioos_qc and expand it to meet any additional community QC use cases that exist and aren’t already met by other packages.  We’d prefer to avoid the development of new QC packages, if at all possible.  

If there are other packages that you're considering using, please submit an issue in the repository identifying those packages. The idea is we want a community accepted package that we can all work from. If there’s something better out there, we should collaborate.

It is interesting how you're finding a large portion of known to be good records being flagged as bad. Can you please submit an issue to the repository with the details you provide in this message? For reference, here is a resource on writing effective GitHub issues.

As far as the package itself is concerned, it is an open source project which allows (and encourages!) contributions and feedback from the community. Please continue to raise issues in the GitHub repository to clarify any shortcomings you find in the package.  We’ll work to ensure they receive a timely response.

Your comment does raise the question of, how do I contribute to ioos_qc? We started an issue to add a contributing guide to the repo. Thank you for that.  As mentioned above, we’re actively pursuing options to improve overall maintenance of the package, including issue and PR response, and ideally roadmapping additional features and functionality.  

At the end of the day, the ioos_qc package is there as a resource to the IOOS and broader oceanographic community (or anyone who finds it applicable to their work). The authority on ioos_qc is the ioos_qc GitHub repository and that is where the conversation should be directed.

Hope this helps,

IOOS DMAC Team

--
You received this message because you are subscribed to the Google Groups "ioos_tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ioos_tech+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ioos_tech/59636b6e-e1a3-41a7-a917-1f1539646982n%40googlegroups.com.

--

Mathew Biddle, Physical Scientist
NOAA/NOS
US Integrated Ocean Observing System Office
1315 East-West Highway
Silver Spring MD 20910

240-533-9470

ORCiD: 0000-0003-4897-1669

http://www.ioos.noaa.gov/