Weird subsetVariables problem

[Tuomo Saari]

Hi,

I am scratching my head over a problem where some datasets that load on one server won't load on another. The Java, Tomcat, and ERDDAP versions are the same. The error message refers to subsetVariables, even though the list is empty in the XML.

From the status page: 

n Datasets Failed To Load (in the last major LoadDatasets) = 1 EFL7-T180-WH300UP-NS4816-Z130-INS34-REC37_profile, (end) Reasons for failing to load datasets: EFL7-T180-WH300UP-NS4816-Z130-INS34-REC37_profile: <subsetVariables> wasn't set up correctly: "Lat" isn't a valid variable destinationName.

- Java: openjdk 21.0.8 2025-07-15 LTS

- Tomcat: apache-tomcat-10.1.47

- ERDDAP: 2.28.1

Attached is a simplified XML file only attempting to load one problematic dataset. Also the netCDF file and ERDDAP logfile are attached. 

Any ideas are welcome. Perhaps I'm blind to something obvious at this point. 

Thank you!

BR,

Tuomo

[Chris John - NOAA Affiliate]

That error message means the subset variable ("Lat" in this case) is not a valid destination variable name. The best solution is to set the subset variables you want for the dataset, using the destination name of the variables. You can see documentation about that here: https://erddap.github.io/docs/server-admin/datasets?_highlight=subsetvariable#subsetvariables

As for why the behavior changed, most likely the servers are running different versions. There was a change in 2.26 for EDDTableFromNcCFFiles. This was to fix a bug where datasets could not be loaded correctly due to forced subsetVariables. The new approach allows setting the subsetVariables in the dataset configuration and if properly set, should load this dataset.

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

--
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/erddap/ee39cbe4-7b5f-40a5-b250-0b5779e96d9cn%40googlegroups.com.

[Tuomo Saari]

Hi Chris,

Thanks for the reply. 

But my confusion comes from the fact that these two servers have the same :

• versions of Java, Tomcat and ERDDAP (see original post for versions)
• source file and datasets.xml

subsetVariables is defined as: <att name="subsetVariables"></att>

I don't understand where the Lat complaint comes from on this one server. It isn't defined as a subset variable. 

Link to test server where the datasets loads: https://gcoos3.geos.tamu.edu/erddap/index.html

Link to test server where it doesn't: https://ocng-gcoos-tmp.artsci.tamu.edu/erddap/index.html

Thanks!

Best Regards,

/Tuomo

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

On Tue, Oct 14, 2025 at 9:18 AM Chris John - NOAA Affiliate <chris...@noaa.gov> wrote:

That error message means the subset variable ("Lat" in this case) is not a valid destination variable name. The best solution is to set the subset variables you want for the dataset, using the destination name of the variables. You

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

 

ZjQcmQRYFpfptBannerEnd

[Tuomo Saari]

Forgot to add  ... 'Lat' is also not defined as a variable destination name. 

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

[Tuomo Saari]

....another addition. From the log file, one can see that it doesn't list 'Lat' in destinationNames:

getting dataset metadata from /data/erddap/gulfhub/ANCLAJES/CNK37/RECUPERACIONES/EFL7-T180-CNK37/ADCPS/EFL7-T180-WH300UP-NS4816-Z130-INS34-REC37_profile.nc

  ftLastMod first=2022-04-05T17:10:18Z last=2022-04-05T17:10:18Z

destinationNames=station, latitude, longitude, time, depth, bins, u, v, w, ae

<subsetVariables> wasn't set up correctly: "Lat" isn't a valid variable destinationName.

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

[Chris John - NOAA Affiliate]

Differences in the setup.xml could also be impacting behavior (in particular if includeNcCFSubsetVariables is set on one of them, though that is not recommended). Also it looks like the server at  https://ocng-gcoos-tmp.artsci.tamu.edu/erddap/index.html is not running right now.

My guess is on the server that is not working includeSubsetVariables is set to true in setup.xml and I recommend removing that setting or setting it to false. Setting that to true enables to old behavior which was buggy, but some servers relied on.

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi

[Tuomo Saari]

Yeah the server is in and out as I'm trying different things... It is my test server. 

Also, includeNcCFSubsetVariables was not set on either of the setup.xml files. 

I tested setting it to false but still no luck. 

I don't know, I'll probably just try downgrading the Tomcat/ERDDAP combo to older versions until I find something that works.This is too weird.

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

On Tue, Oct 14, 2025 at 12:43 PM Chris John - NOAA Affiliate <chris...@noaa.gov> wrote:

Differences in the setup. xml could also be impacting behavior (in particular if includeNcCFSubsetVariables is set on one of them, though that is not recommended). Also it looks like the server at https: //ocng-gcoos-tmp. artsci. tamu. edu/erddap/index. html

[Tuomo Saari]

Actually, thanks for the tip of comparing the setup.xml files.

On the server where the datasets didn't load, I had these extra lines in the setup.xml file:

<useSharedWatchService>true</useSharedWatchService>

<useSaxParser>true</useSaxParser>

<usePrometheusMetrics>true</usePrometheusMetrics>

I deleted those lines and now the dataset loads!

Thanks again!

BR,

Tuomo

 

Tuomo Saari | Research Specialist II
Gulf of America Coastal Ocean Observing System (GCOOS)

Department of Oceanography | Texas A&M University
(979) 393-2480 | tuomo...@gcoos.org | tsa...@tamu.edu

[Chris John - NOAA Affiliate]

In case it's useful for you or anyone else reading the thread:

useSharedWatchService has been removed as a setting as of 2.28.0, so that should not have an impact.

usePrometheusMetrics defaults to on, so if you removed the line it should have identical behavoir

useSaxParser- this is a new approach to how we parse datasets.xml. There are some differences between the sax parser and the old approach for what's allowed (generally the SAX parser more closely adheres to standard xml), though we did try to minimize these differences. While the SAX parser is optional, in some cases new features are only available with the SAX parser. One common issue when migrating to the SAX parser is needing to declare special characters in the SAX Parser (for example `<!DOCTYPE note [<!ENTITY deg '&#176;'>]>` to be able to use `&deg;` within the file).

Christopher John (he/him)

NOAA Appointed Technical Director of ERDDAP™

Computer and Information Systems Manager, TSPi