ERDDAP dataset not working after ERDDAP update

[Tuomo Saari]

Hi,

We host a load of old datasets that used to work under the older ERDDAP installations. 

With updated ERDDAP (currently version 2.27.0), some datasets won't load, and the error messages are not very helpful. Please find attached the datasets.xml and the netCDF file for just one of the datasets. 

Would anyone have a clue what is going on here? Thank you!

/Tuomo 

P.S. The error message in the log.txt is:

"

n Datasets Failed To Load (in the last major LoadDatasets) = 1
    lc2010_fl6_XBT, (end)
Reasons for failing to load datasets:
lc2010_fl6_XBT: datasets.xml error on line #419
While trying to load datasetID=lc2010_fl6_XBT (after 84 ms)
java.lang.RuntimeException: datasets.xml error on or before line #419:
ERROR in Test.ensureTrue:
No valid data files were found. See log.txt for details.
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:819)
 at gov.noaa.pfel.erddap.LoadDatasets.parseUsingSimpleXmlReader(LoadDatasets.java:672)
 at gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:220)
Caused by: java.lang.RuntimeException:
ERROR in Test.ensureTrue:
No valid data files were found. See log.txt for details.
 at com.cohort.util.Test.error(Test.java:23)
 at com.cohort.util.Test.ensureTrue(Test.java:54)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.<init>(EDDTableFromFiles.java:2103)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromMultidimNcFiles.<init>(EDDTableFromMultidimNcFiles.java:103)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.fromXml(EDDTableFromFiles.java:662)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:781)
 ... 2 more

"

[Roy Mendelssohn - NOAA Federal]

Hi Tuomo:

Sorry you are having problems, I will take a look at the file. Can you please send me the top of your setup.xml so I can see your settings and is line 419 the same line as your file - the error is it is not finding files - perhaps you need to recheck permissions And can you look in your log.txt file (it it in you bigERDDAP directory in the logs subdirectory. Search for the datasetID and there should be more complete errors. Also what version of Java are you using? You did update the Java, if my memory serves (I will have to check that), version 2.27 requires java 21. Also the latest version is 2.28.1, but let’s get things working with what you have.

Sorry to ask for so much information but it is difficult otherwise to debug this.

Thanks,

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/0871438e-bbf5-434b-a427-13a153dde10bn%40googlegroups.com.
> <datasets.xml><lc2010_fl6.nc>

[Tuomo Saari]

Hi Roy,

Thanks for the response! Sorry, I was missing quite much information on my post.

Can you please send me the top of your setup.xml so I can see your settings and is line 419 the same line as your file - the error is it is not finding files - perhaps you need to recheck permissions.

Setup.xml attached. As for the permissions, they have not changed. Also, there are occasions where the same source netCDF file is used for two different datasets, and one of the datasets works. I am certain that this is not a permissions issue. And the line 419 is the end of the dataset config in the datasets.xml.

And can you look in your log.txt file (it in you bigERDDAP directory in the logs subdirectory.  Search for the datasetID and there should be more complete errors. 

The log file and the setup.xml are attached. I found this additional error message: 

"java.lang.RuntimeException: ERROR in NDimensionalIndex constructor: shape=[0, 0] has a value less than 1.
 at com.cohort.array.NDimensionalIndex.<init>(NDimensionalIndex.java:78)
 at gov.noaa.pfel.coastwatch.pointdata.Table.addIndexColumns(Table.java:6526)
 at gov.noaa.pfel.coastwatch.pointdata.TableFromMultidimNcFile.readMultidimNc(TableFromMultidimNcFile.java:357)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromMultidimNcFiles.lowGetSourceDataFromFile(EDDTableFromMultidimNcFiles.java:180)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.getSourceDataFromFile(EDDTableFromFiles.java:4148)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.<init>(EDDTableFromFiles.java:2043)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromMultidimNcFiles.<init>(EDDTableFromMultidimNcFiles.java:103)
 at gov.noaa.pfel.erddap.dataset.EDDTableFromFiles.fromXml(EDDTableFromFiles.java:662)
 at gov.noaa.pfel.erddap.dataset.EDD.fromXml(EDD.java:781)
 at gov.noaa.pfel.erddap.LoadDatasets.parseUsingSimpleXmlReader(LoadDatasets.java:672)
 at gov.noaa.pfel.erddap.LoadDatasets.run(LoadDatasets.java:220)"

Also what version of Java are you using?  

[tsaari-admin@gcoos3:~ _0]$ java --version
openjdk 21.0.8 2025-07-15 LTS
OpenJDK Runtime Environment Temurin-21.0.8+9 (build 21.0.8+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.8+9 (build 21.0.8+9-LTS, mixed mode, sharing)

Thank you for your help!

BR,

Tuomo

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

On Fri, Sep 12, 2025 at 6:02 PM Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:

Hi Tuomo: Sorry you are having problems, I will take a look at the file. Can you please send me the top of your setup. xml so I can see your settings and is line 419 the same line as your file - the error is it is not finding files - perhaps

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

.
> To view this discussion, visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/erddap/0871438e-bbf5-434b-a427-13a153dde10bn*40googlegroups.com__;JQ!!KwNVnqRv!HUbHPXsu7203y4L5CplTWVKd3-bKIAPB8Nyp2fwUuREBmRRsvah-XVWKpN7J9ptVySVU4-RG1o-EVtRRZKouwc_o_Q$.
> <datasets.xml><lc2010_fl6.nc>

[Roy Mendelssohn - NOAA Federal]

You say:

> there are occasions where the same source netCDF file is used for two different datasets

Can you send me examples where this is the case?

Thanks,

-Roy

> <log.txt><setup.xml>

[Roy Mendelssohn - NOAA Federal]

First, thank you for all the info you sent. It makes it much easier to try and find problems. I know there have been some changes in what is required for DSG related file, and I am not an expert on these, this may well be one for Chris. However, I went to:

https://chrisjohnnoaa.github.io/erddap.github.io/docs/server-admin/datasets?_highlight=eddtablefrommultidimncfiles#eddtablefrommultidimncfiles

And when I scroll down I see:

> DimensionsCSV -- GenerateDatasetsXml will ask for a "DimensionsCSV" string. This is a comma-separated-value list of source names of a set of dimensions. GenerateDatasetsXml will only read data variables in sample .nc files which use some or all of those dimensions (and no other dimensions), plus all of the scalar variables in the file, and make the dataset from those data variables. If a dimension is in a group, you must specify its fullName, e.g., "groupName/dimensionName".
> If you specify nothing (an empty string), GenerateDatasetsXml will look for the variables with the most dimensions, on the theory that they will be the most interesting, but there may be times when you will want to make a dataset from some other group of data variables that uses some other group of dimensions.
> If you just specify a dimension name that doesn't exist (e.g., NO_MATCH), ERDDAP™ will just find all of the scalar variables.
> The "DimensionsCSV" string becomes <dimensionsCSV> in the datasets.xml info for the dataset.
>

When I did a search on your datasets.xml I don’t see <dimensionsCSV>.. What I would suggest is going back and for this file use the GenerateDatasetsXml that came with ERDDAP™ 2.27 and regenerate the xml and if you can get that to work, look at the differences and you will know how to change the other files. If I am motivated this weekend I may try this, or maybe early next week, in the meantime you can see if this solves the problem.

I also know that in the newer versions the regex has gotten stricter, so it is possible in the <fileNameRegex>, you might have to escape out the “_” as “\_”. But don't quote me on this as regex has always been a mystery to me.

Let us know if you make any progress.

HTH,

-Roy

[Tuomo Saari]

Thanks Roy,

It seems there is similar post on the ERDDAP Google group: MultidimNcFiles after version 2.24.

Are there any updates to that issue?

I tend to think that my netCDF files (which are old, and I wasn't around when they were generated..) don't conform to the CF standards, and the dimension configurations (inside the source files) are probably off. 

I have been trying to play with different dimensionsCSV strings with no success so far..

It looks like the source data includes variables that are supposed to tell the start and end index of data for each station (start_index, stop_index). Not sure how to deal with that information though.

Thanks!

/Tuomo

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

On Fri, Sep 12, 2025 at 7:49 PM Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:

First, thank you for all the info you sent. It makes it much easier to try and find problems. I know there have been some changes in what is required for DSG related file, and I am not an expert on these, this may well be one for Chris. However,

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

First, thank you for all the info you sent.  It makes it much easier to try and find problems.  I know there have been some changes in what is required for DSG related file,  and I am not an expert on these,  this may well be one for Chris.  However,  I went to:

https://urldefense.com/v3/__https://chrisjohnnoaa.github.io/erddap.github.io/docs/server-admin/datasets?_highlight=eddtablefrommultidimncfiles*eddtablefrommultidimncfiles__;Iw!!KwNVnqRv!Bd7kPGZiPazagfdUiCBjNY-i1bz6fHHmYfCFKLCy981rOr62nx02TlStjXGWAsG2qDwV-eC66LBBJ3Gypthg7NBHAw$

[Roy Mendelssohn - NOAA Federal]

Thanks Tuomo.

I will take a closer look but this may well be one for Chris. My memory among other things is that the chosen type doesn’t agree with how ERDDAP views that data structure. As I said, I don’t have a great knowledge of this, so I may not be of much help, but I will look.

-Roy

[Tuomo Saari]

There is one dataset that uses the same source file, that gets successfully published, but only because it refers to variables that DON't EXIST in the source file (CTD_Depth, CTD_Temperature, CTD_Salinity). As a result, if my logic works, ERDDAP doesn't have to deal with those variables' dimensions, so we don't end up in the same error. Obviously I should remove this datasets as it makes no sense, but it sheds some light to my issue. 

It would appear that in my example, for the lc2010_fl6_XBT dataset, one tries to publish the variables  XBT_Depth and XBT_Temperature (that use dimension XBT_INDEX = 9142). 

And if I interpret correctly, the variables indextype (=2), start_index, stop_index, define where the data for those profile variables are. 

I have no idea how one should configure or redo the source file for this. For now, I think the best I can do is to disable these datasets.

It is a mystery to me how these didn't trigger any errors under the older ERDDAP versions. Also not sure if what was published made sense either. I cannot verify right now because I don't have the old ERDDAP version installed.

BR,

Tuomo

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

[Roy Mendelssohn - NOAA Federal]

Please send that datasets.xml snippet for that file.

Thanks,

-Roy

[Tuomo Saari]

Here...

Thanks!

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

On Mon, Sep 15, 2025 at 3:32 PM 'Roy Mendelssohn - NOAA Federal' via ERDDAP <erd...@googlegroups.com> wrote:

Please send that datasets. xml snippet for that file. Thanks, -Roy > On Sep 15, 2025, at 1: 25 PM, Tuomo Saari <tsaari@ tamu. edu> wrote: > > There is one dataset that uses the same source file, that gets successfully published, but

>
> >> 
> >> 
> >> <log.txt><setup.xml>
> > 
> 
> 

-- 
You received this message because you are subscribed to a topic in the Google Groups "ERDDAP" group.
To unsubscribe from this topic, visit https://urldefense.com/v3/__https://groups.google.com/d/topic/erddap/YO-X6E6SXDs/unsubscribe__;!!KwNVnqRv!DKcZEXaZEuU4eTByJj6hADjUzTzF66_-XhPIlFdMtKVQHnzqwV3U9_mTzm2OOmP30Qt5LOL_A8UBlDoXuw$.
To unsubscribe from this group and all its topics, send an email to erddap+un...@googlegroups.com.
To view this discussion, visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/erddap/385794FE-31F5-4A60-8C46-7F780974BC22*40noaa.gov__;JQ!!KwNVnqRv!DKcZEXaZEuU4eTByJj6hADjUzTzF66_-XhPIlFdMtKVQHnzqwV3U9_mTzm2OOmP30Qt5LOL_A8VkvKa-DQ$.

[Chris John - NOAA Affiliate]

ERDDAP™ uses netcdf-java for reading nc/netCDF files. When we do a versioned release we always update the netcdf-java version. For example for ERDDAP™ 2.27.0 we moved from netcdf-java version 5.7.0 to 5.8.0. Depending on what your previous ERDDAP™ version would be a bigger jump in netcdf-java versions. Theoretically as long as the netcdf-java version did not change the first section ("5" in this case), it should be backwards compatible, but we have seen some cases where netcdf-java updates change how some files are read (usually files that are not full compliant with the spec somehow).

As for the post about MultidimNcFiles after 2.24, I believe that was resolved with an update to the datasets.xml. There was some additional troubleshooting in the issue that was created: https://github.com/ERDDAP/erddap/issues/329

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

To view this discussion, visit https://groups.google.com/d/msgid/erddap/CAMx%3DUCgFBnJDoWLoS4Do96E5p65xZPPV8WtTt-XR2W3QoLaVWQ%40mail.gmail.com.

[Roy Mendelssohn - NOAA Federal]

Hi Tuomo:

I believe the problem is you have the data defined as “profiles” but don’t have the dimensions as required by ERDDAP™ for a profile. I think there are other issues with how the coordinates are defined, if I have time in the next day or two I will see if I can convert the one dataset to something that will work with ERDDAP™.

-Roy

> On Sep 15, 2025, at 1:53 PM, Chris John - NOAA Affiliate <chris...@noaa.gov> wrote:
>
> ERDDAP™ uses netcdf-java for reading nc/netCDF files. When we do a versioned release we always update the netcdf-java version. For example for ERDDAP™ 2.27.0 we moved from netcdf-java version 5.7.0 to 5.8.0. Depending on what your previous ERDDAP™ version would be a bigger jump in netcdf-java versions. Theoretically as long as the netcdf-java version did not change the first section ("5" in this case), it should be backwards compatible, but we have seen some cases where netcdf-java updates change how some files are read (usually files that are not full compliant with the spec somehow).
>
> As for the post about MultidimNcFiles after 2.24, I believe that was resolved with an update to the datasets.xml. There was some additional troubleshooting in the issue that was created: https://github.com/ERDDAP/erddap/issues/329
>
> Christopher John (he/him)
> NOAA Appointed Technical Director of ERDDAP™
> Computer and Information Systems Manager, TSPi
>
>
>
> On Mon, Sep 15, 2025 at 4:41 PM 'Tuomo Saari' via ERDDAP <erd...@googlegroups.com> wrote:
> Here...
> Thanks!
>
> Tuomo Saari | Research Specialist II
> Gulf of America Coastal Ocean Observing System (GCOOS)
> Department of Oceanography | Texas A&M University
> (979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu
>
>
>
> On Mon, Sep 15, 2025 at 3:32 PM 'Roy Mendelssohn - NOAA Federal' via ERDDAP <erd...@googlegroups.com> wrote:
> This Message Is From an External Sender
> This message came from outside your organization.

[Roy Mendelssohn - NOAA Federal]

More on this, please see:

https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#discrete-sampling-geometries

Look at the definition of “profile”. Also look at:

https://chrisjohnnoaa.github.io/erddap.github.io/docs/server-admin/datasets#profile

Part of this is the definitions of the CDM datatypes has changed over time. The reference above has the definitions used by ERDDAP. You can see that profiles is defines as data(i,o) and ERDDAP™ requires certain attributes set in the global attributes.

As I said, I may try to whip up a Python program that will convert your dataset. But also remember I am old and slow, I believe the previous discussion you refer to, if you look at the Github issue, was resolved by rewriting the file to meet ERDDAP™’s requirements.

HTH,

-Roy

[Roy Mendelssohn - NOAA Federal]

Hi Tuomo:

I looked at this some more. Another problem with your file is that it has two profiles that are on different depth scales. While this is a valid netcdf file, strictly speaking it is not a fully CF-compliant file, as Profiles require the data to be on the same “z” grid. You would do best by separating the two profiles into different files, and follow some of the guides on what the files should look like. Another place to look for guidance on the files is the NCEI Templates:

https://www.ncei.noaa.gov/data/oceans/ncei/formats/netcdf/v2.0/index.html

HTH,

-Roy

PS - If you want to have the files in the same file, create a “z” axis that has all the “z” values in the two profiles, and put missing data where it is not defined for each profile. I am not a big fan of that approach, but I do believe it would work.