MultidimNcFiles after version 2.24

[Italo Gabriele Nicolaus Borrelli]

Hi all!

I was trying to update our ERDDAP to version 2.27 from version 2.21 and came across some issues with accessing our ADCP datasets which are  EDDTableFromMultidimNcFiles. I changed our version a few times and found that 2.25 was where the breaking change is occurring for us. I took a look at the commits and noticed that Chris made some significant changes to improve  EDDTableFromMultidimNcFiles. It would seem then that perhaps our nc files are not exactly structured according to the CF standards per this section of the conventions.

I have since gone through with v2.27 and generated a new section for the datasets.xml. I attempted 3 things with the "Dimensions CSV" with varying levels of success:

• None: Generated a section. cdm_data_type="Other" by default. Managed to access the data.
• time,depth,latitude,longitude:  Failed to generate a section with the error: java.lang.RuntimeException: ERROR in NDimensionalIndex constructor: shape=[0, 0, 0, 0] has a value less than 1
• time,depth: Generated a section. cdm_data_type="Other" by default. Managed to access the data. Latitude and longitude notably not added as variables.

You can access the dataset in question on our website. Admittedly, it's currently not exactly correct, since we used TimeSeries. The reason is that we were switching from EDDTableFromCassandra and did it in a bit of a hurry. It really should be TimeSeriesProfile.

If you look at the attached file, current_index.csv, you'll get the metadata for the dataset linked. updated_index.csv works in v2.27 after doing the following:

• Generated the xml with Dimensions CSV as "time,depth"
• Changed the cdm_data_type to TimeSeriesProfile and added the cdm_***_variables
• Added latitude and longitude variables back

At this point, once again I was getting an error to do with the nc files being bad. I had one last idea and changed latitude and longitude to not be sourced from the nc file, as in:

    <dataVariable>
        <sourceName>=48.33664695</sourceName>
        <destinationName>latitude</destinationName>
        <dataType>double</dataType>
        <addAttributes>
          <att name="ioos_category">Location</att>
          <att name="standard_name">latitude</att>
          <att name="units">degree_north</att>
          <att name="actual_range" type="doubleList">48.3366411 48.3366528</att>
          <att name="long_name">Latitude</att>
        </addAttributes>
      </dataVariable>
      <dataVariable>
        <sourceName>=-123.74975219999999</sourceName>
        <destinationName>longitude</destinationName>
        <dataType>double</dataType>
        <addAttributes>
          <att name="ioos_category">Location</att>
          <att name="standard_name">longitude</att>
          <att name="units">degree_east</att>
          <att name="actual_range" type="doubleList">-123.7499388 -123.7495656</att>
          <att name="long_name">Longitude</att>
        </addAttributes>
      </dataVariable>

This did work as expected! However, the fact that getting the value of latitude and longitude from the NetCDF files makes me think that there may be a mistake in the NetCDF files themselves and they may not follow the CF conventions.

I've attached the output of "ncdump -h" from one of the files. Of note are likely:

netcdf RDIADCP300WH15551_20110720T000000Z_20110720T235958Z-binMapNearest_3beamOn.nc {
  dimensions:
    time = 43200;
    depth = 54;
    latitude = 1;
    longitude = 1;

  variables:
    double time(time=43200);
      :sdn_parameter_name = "time";
      :long_name = "time";
      :axis = "T";
      :calendar = "gregorian";
      :units = "days since 1970-01-01 00:00:00";
      :_ChunkSizes = 43200U; // uint

    double range(depth=54);
      :sdn_parameter_name = "range";
      :long_name = "range from transducer";
      :units = "meters";
      :comment = "Range of measurement bins from the transducer, applies to all parameters except: meanBackscatter and u, v, w, velocityError when bin-mapping is on";
      :_ChunkSizes = 54U; // uint
    double latitude(latitude=1);
      :sdn_parameter_name = "latitude";
      :long_name = "latitude";
      :units = "degree_north";
      :axis = "Y";
      :_ChunkSizes = 1U; // uint
    double longitude(longitude=1);
      :sdn_parameter_name = "longitude";
      :long_name = "longitude";
      :units = "degree_east";
      :axis = "X";
      :_ChunkSizes = 1U; // uint

I don't know the internal representation as I didn't generate these files myself and I don't know how to get that information from ncdump, but if anyone has suggestions how to do that I can confirm if it's CF compliant or not.

Thanks in advance for the input!

[Chris John - NOAA Affiliate]

It'd be helpful if you could open an issue on the GitHub with example files that are causing problems (and dataset definition).

One thing to note, is there was also an update to a new version of netcdf libraries (from 5.5.x to 5.6.0) with that release. I wonder if that's the actual change that caused the issue with parsing your files. You could try changing the version of the netcdf libraries to an older version (like 5.5.3) and then building ERDDAP to see if that is able to parse your files. The netcdf library version is defined in the pom file.

currently:

<netcdfJavaVersion>5.8.0</netcdfJavaVersion>

try:

<netcdfJavaVersion>5.5.3</netcdfJavaVersion>

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

--
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/erddap/6476881e-948b-4c89-9432-33e132b69b1fn%40googlegroups.com.

[Roy Mendelssohn - NOAA Federal]

Thanks. I will take a look at this, I may ask you to post an issue on the GitHub page, but let me look first. Can you send me the complete datasets.xml snippet for that datas. I believe this may be something we have run into before. Also if it isn’t too big can you send me one sample file.

Thanks,

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/6476881e-948b-4c89-9432-33e132b69b1fn%40googlegroups.com.

> <updated_index.csv><current_index.csv><ADCP_headers.txt>

[Roy Mendelssohn - NOAA Federal]

Ah Chris already asked you to open an issue. But please do include the xml snippet so we can see the complete dataset definition you are using, and as I said a sample file can help as it allows us to reproduce the problem and to test possible solutions.

Thanks,

-Roy