NCEI netcdf templates for trajectoryProfile

[Emilio Mayorga]

Hi,

I'm involved in a project where we're combining deployments from several Argo-style floats into a single netcdf file using a trajectoryProfile feature type and are trying to follow the current (v2.0) NCEI netcdf templates (https://www.ncei.noaa.gov/data/oceans/ncei/formats/netcdf/v2.0/index.html), for a regridded product rather than the raw data per se. trajectoryProfile seems to be the least fleshed out of the discrete sampling geometry feature types in the NCEI templates. I'd like to see if others have used this feature type while trying to follow the NCEI templates.

** My question is: for trajectoryProfile, does the NCEI "Orthogonal" case correspond to orthogonal depth and incomplete time, while the "Incomplete" case corresponds to incomplete depth and time?

The timeSeriesProfile and trajectoryProfile feature types both have three dimensions: the "instance" dimension for multiple features, and two "element" dimensions (z and profile, following the terms used in the CF convention user manual). For both feature types, NCEI describes templates for orthogonal and incomplete Multidimensional Array representations. The timeSeriesProfile presentation in "Feature Type Templates and Examples" is pretty comprehensive. It defines a pure orthogonal case, where both z and profile ("time") are orthogonal; a fully incomplete case, where both z and profile ("time") are incomplete; and two mixed cases where either z or profile is incomplete and the other is orthogonal. That makes sense, and it's nicely described in a table, under columns Description and General Ocean Data Examples. The accompanying decision tree diagram also fleshes this out (https://www.ncei.noaa.gov/data/oceans/ncei/formats/netcdf/v2.0/decision_tree_high_res.pdf).

For trajectoryProfile, only two cases are presented ("Orthogonal" and "Incomplete") and the table barely elaborates these under Description and General Ocean Data Examples. "Orthogonal" is misleading, because in reality it seems to be orthogonal along z *but* incomplete along time / profile. "Incomplete" is incomplete along both dimensions. In addition, we think the diagram has an error in the "Does each profile in the trajectoryProfile collection share the same vertical coordinates?" decision: when true, this should lead to the orthogonal-along-vertical template, but it leads to the incomplete-along-vertical template instead.

** Has anyone confirmed this error in the trajectoryProfile decision box?

I suppose the fully orthogonal and orthogonal time + incomplete z cases may be less common and are omitted for that reason. They do say "Our templates don't try to cover every possibility. We focused on the templates that we felt would cover most situations involving environmental observations, primarily of the ocean." But the scant documentation, misleading "orthogonal" label and possible error in the decision tree diagram makes things confusing.

Thanks! Cheers,
-Emilio

[Micah Wengren - NOAA Federal]

Hi Emilio, 

I don't have an answer, but I cc'ed the ncei....@noaa.gov email group in case anyone from that list has thoughts.  

There was also discussion within NOAA recently about getting a group of volunteers together to update the NCEI netCDF templates.  I am not sure if that's going forward yet or not.  Resolving questions or issues like this could be part of that effort.

If you were interested in participating, they may want external input as well, potentially.  

Since IOOS leverages the NCEI templates, we would also probably want to invite those from our data management community to weigh in as well to make sure any concerns they might have could be addressed.

Micah

--
You received this message because you are subscribed to the Google Groups "ioos_tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ioos_tech+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ioos_tech/CAPfVOv4yMFP2ymPW7ucm1VYNCVNF5%3D9gfeZC6JiQv8Y_8sOKCQ%40mail.gmail.com.

--

Micah Wengren
DMAC System Architect
U.S. Integrated Ocean Observing System
https://ioos.noaa.gov/
Santa Cruz, CA
+1 240-533-9441 (o)
+1 202-276-3608 (c) 

[Emilio Mayorga]

Hi Micah,

Thanks for the follow up and for forwarding my message to the NCEI netcdf group. I missed that -- even though it's prominent, now that I've looked back at the page.

Cheers,

-Emilio

To view this discussion on the web visit https://groups.google.com/d/msgid/ioos_tech/CAJQ92%2B%3DWDxfrVuLnh26yEu2nevkySOXhBbYvuQ_%3DrM_VVhHebA%40mail.gmail.com.

[Emilio Mayorga]

Thanks, Ken. Great to hear about this plan for an update to the templates. I doubt I'll have the bandwidth to be involved.

In the meantime, it'd be great to get confirmation about the likely error in the trajectoryProfile decision tree diagram I pointed out. I'm pasting my observation and have included a screenshot of the portion of the diagram I'm referring to:

we think the diagram has an error in the "Does each profile in the trajectoryProfile collection share the same vertical coordinates?" decision: when true, this should lead to the orthogonal-along-vertical template, but it leads to the incomplete-along-vertical template instead.

Thanks! Cheers,

-Emilio

On Thu, Mar 14, 2024 at 5:39 AM Kenneth Casey - NOAA Federal <kennet...@noaa.gov> wrote:

Phil Jones is indeed leading an effort, later this FY, to update the netCDF templates.  Please talk directly with him if you would like more information.

Ken

--

--

[NOTE: The opinions expressed in this email are those of the author alone and do not necessarily reflect official NOAA, Department of Commerce, or US government policy.]

Kenneth S. Casey, Ph.D. (he/him)

Deputy Chief, Data Stewardship Division

NOAA National Centers for Environmental Information (NCEI)

1315 East-West Highway

Silver Spring MD 20910 USA
+1-301-713-4849

Connect with NCEI on FaceBook and follow us on Twitter at @NOAANCEI

[Emilio Mayorga]

Hi Phil,

Great! I'm glad to see the timing was good. I'd be happy to connect offline if your team would like to hear more about our use case.

Actually, come to think of it, this project involves some collaboration with Scott Cross at NCEI vis a vis the Global Ocean Currents Database, and eventual archiving of these data down the line.

Thanks,

Emilio

On Fri, Mar 15, 2024, 5:33 AM Philip Jones - NOAA Federal <philip...@noaa.gov> wrote:

Hi Emilio,

Your suggested change is appreciated and is well-timed with the effort happening this year to update the templates. I'll include it for the team to review and reach out to you with any questions.

Thanks,

Phil

--

Philip R. Jones (he/him)

Archive Branch Metadata Lead/Physical Scientist

NOAA National Centers for Environmental Information (NCEI)

Veach-Baley Federal Building

151 Patton Avenue

Asheville, NC 28801-5001 USA

philip...@noaa.gov

Office: +1 828-271-4472

[Micah Wengren - NOAA Federal]

Forwarding Tim's response to ioos_tech as an FYI.

Micah

---------- Forwarded message ---------
From: Tim Boyer - NOAA Federal <tim....@noaa.gov>
Date: Sun, Mar 17, 2024 at 12:42 PM
Subject: Re: [ioos_tech] NCEI netcdf templates for trajectoryProfile
To: Emilio Mayorga <emilio...@gmail.com>
Cc: Philip Jones - NOAA Federal <philip...@noaa.gov>, Kenneth Casey - NOAA Federal <kennet...@noaa.gov>, <ioos...@googlegroups.com>, <ncei....@noaa.gov>

Yes, a revamp and additional support for the netcdf templates is most welcome.  Emilio, are you working with the University of Hawaii ocean currents group along with Scott Cross and Ayesha Genz?  They too have had some difficulties adapting the netcdf templates to shipboard acoustic doppler current profiler (S-ADCP) data - though this is not really Argo-like data.  We also have the World Ocean Database netcdf ragged array format for ocean profile from multiple sources (some Argo-like - Argo profiles not Argo trajectory, but also gliders, research cruises, moored buoys, etc.) .  This form, although it is now our main dissemination form for the World Ocean Database does not have an associated template.  So there is a lot of room for working with various oceanographic communities on netcdf templates that best fit community needs.

Regards,

Tim

[Emilio Mayorga]

Hi Tim (and Phil),

I'd forgotten to follow up. We're not working with U. Hawaii at this time. The project, "Sampling QUantitative Internal-wave Distributions" (http://faculty.washington.edu/girton/squid/), is led by James Girton and is part of the NOPP's Sampling QUantitative Internal-wave Distributions program (https://nopp-giw.ucsd.edu). One of the core goals of the project is to make velocity data from velocity profiling (EM-APEX, ElectroMagnetic APEX) floats more broadly usable and standardized, and one of the mechanisms for doing that will hopefully be via the Global Ocean Currents Database (https://www.ncei.noaa.gov/products/global-ocean-currents-database). Our collaborator there is Scott Cross, but we haven't reconnected about this project in a while.

Great to see that the scope of the netcdf template revamp includes the range of use cases you mention.

Have you or Phil had a chance to check if the issue I pointed out with the trajectoryProfile decision tree is in fact an error? It'd be helpful to have official confirmation. Here's how I described it earlier:

we think the diagram (https://www.ncei.noaa.gov/data/oceans/ncei/formats/netcdf/v2.0/decision_tree_high_res.pdf) has an error in the "Does each profile in the trajectoryProfile collection share the same vertical coordinates?" decision: when true, this should lead to the orthogonal-along-vertical template, but it leads to the incomplete-along-vertical template instead.

Thanks,

-Emilio

On Sun, Mar 17, 2024 at 9:42 AM Tim Boyer - NOAA Federal <tim....@noaa.gov> wrote:

Yes, a revamp and additional support for the netcdf templates is most welcome.  Emilio, are you working with the University of Hawaii ocean currents group along with Scott Cross and Ayesha Genz?  They too have had some difficulties adapting the netcdf templates to shipboard acoustic doppler current profiler (S-ADCP) data - though this is not really Argo-like data.  We also have the World Ocean Database netcdf ragged array format for ocean profile from multiple sources (some Argo-like - Argo profiles not Argo trajectory, but also gliders, research cruises, moored buoys, etc.) .  This form, although it is now our main dissemination form for the World Ocean Database does not have an associated template.  So there is a lot of room for working with various oceanographic communities on netcdf templates that best fit community needs.

Regards,

Tim