EDDTableFromNcFiles returning 0 files when using pathRegex and recursive="true"

[John Mund - NOAA Affiliate]

Hi.
We are trying to create "Strategy-level" datasets that aggregate files from multiple top-level species directories into a single unified dataset but can't get the regex syntax to work.

Our data is organized as: /aftp/data/trace_gases/[species]/[strategy]/[project]/nc/*.nc
Example path: /aftp/data/trace_gases/co2/pfp/surface/nc/co2_mlo_surface-pfp_1_ccgg_event.nc

We want a single dataset for pfp_surface that finds all .nc files in:
trace_gases/*/pfp/surface/nc/

We are using EDDTableFromNcFiles with the following parameters:
fileDir: /aftp/data/trace_gases/
recursive: true
fileNameRegex: .*\.nc
pathRegex: (We have tried multiple variations—see below)

The Issue:
Despite the files being present, ERDDAP consistently reports 0 files found.

Log Output Example:

flask_surface: 0 files found in /aftp/data/trace_gases/
regex=.*\.nc recursive=true pathRegex=.*flask/surface/nc.* time=19ms

Variations Tried for pathRegex:
^[^/]+/pfp/surface/nc/
.*/pfp/surface/nc/.*
.*/pfp/surface/nc(/.*)?
.*pfp/surface/nc.*

We have verified that the tomcat user has full read/execute access to the entire directory
tree and have successfully created datasets of this specific directory without the regex.

ERDDAP Version: 2.29.0 on Linux RHEL 9

Is this configuration possible?  Any suggestions on how to set up our datasets.xml?

[Roy Mendelssohn - NOAA Federal]

Hi John:

Try this:

<fileDir>/aftp/data/trace_gases/</fileDir>
<recursive>true</recursive>
<fileNameRegex>.*\.nc</fileNameRegex>
<pathRegex>.*/pfp/surface/nc/.*</pathRegex>

If that doesn’t work let me know with what is shown in the log file, also I will likely want the full xml snippet as well enough of an example of your data structure so that I can reproduce the problem. It is just hard for me to really see what is going on and experiment otherwise.

But try above and let me know.

-Roy

> --
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/c6a16a50-01a8-42ee-a62c-aa0ca2312e9bn%40googlegroups.com.

[John Mund - NOAA Affiliate]

Hi Roy, 

Thanks for looking into this.

Log output of datasets failing to load:

Datasets Failed To Load (in the last major LoadDatasets) = 6

    flask_surface, in_situ_aircraft, in_situ_surface, in_situ_tower, pfp_aircraft, pfp_surface, (end)

Reasons for failing to load datasets: 

flask_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/flask/surface/nc/.* time=35ms

in_situ_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/aircraft/nc/.* time=8ms

in_situ_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/surface/nc/.* time=7ms

in_situ_tower: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/tower/nc/.* time=11ms

pfp_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/aircraft/nc/.* time=8ms

pfp_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/surface/nc/.* time=7ms

Directory structure:

find /aftp/data/trace_gases/ -maxdepth 4 -type d | head -n 20

/aftp/data/trace_gases/

/aftp/data/trace_gases/CHCl3

/aftp/data/trace_gases/CHCl3/pfp

/aftp/data/trace_gases/CHCl3/pfp/surface

/aftp/data/trace_gases/CHCl3/pfp/surface/txt

/aftp/data/trace_gases/CHCl3/pfp/surface/nc

/aftp/data/trace_gases/CHCl3/pfp/aircraft

/aftp/data/trace_gases/CHCl3/pfp/aircraft/txt

/aftp/data/trace_gases/CHCl3/pfp/aircraft/nc

/aftp/data/trace_gases/H2402

/aftp/data/trace_gases/H2402/pfp

/aftp/data/trace_gases/H2402/pfp/surface

/aftp/data/trace_gases/H2402/pfp/surface/txt

/aftp/data/trace_gases/H2402/pfp/surface/nc

/aftp/data/trace_gases/H2402/pfp/aircraft

/aftp/data/trace_gases/H2402/pfp/aircraft/txt

/aftp/data/trace_gases/H2402/pfp/aircraft/nc

/aftp/data/trace_gases/C3H8

/aftp/data/trace_gases/C3H8/pfp

/aftp/data/trace_gases/C3H8/pfp/surface

1 example path:

ll /aftp/data/trace_gases/co2/pfp/surface/nc/co2_amt_surface-pfp_1_ccgg_event.nc 

-rwxr-xr-x. 1 300 201 288321 Mar  2 11:33 /aftp/data/trace_gases/co2/pfp/surface/nc/co2_amt_surface-pfp_1_ccgg_event.nc

Snippet of dataset xml:

<dataset type="EDDTableFromNcFiles" datasetID="flask_surface" active="true">

<reloadEveryNMinutes>10080</reloadEveryNMinutes>

<updateEveryNMillis>10000</updateEveryNMillis>

<fileDir>/aftp/data/trace_gases/</fileDir>

<fileNameRegex>.*\.nc</fileNameRegex>

<recursive>true</recursive>

<pathRegex>.*/flask/surface/nc/.*</pathRegex>

<metadataFrom>last</metadataFrom>

<standardizeWhat>0</standardizeWhat>

<sortedColumnSourceName>time</sortedColumnSourceName>

<sortFilesBySourceNames>time</sortFilesBySourceNames>

<addAttributes>

<att name="dataset_creation_date">2025-08-15T13:48:06.496334</att>

<att name="dataset_project">surface-flask</att>

<att name="dataset_platform">fixed</att>

<att name="dataset_selection">event</att>

<att name="dataset_selection_tag">event</att>

<att name="dataset_comment">For more information about these data, please see https://gml.noaa.gov/aftp/data/trace_gases/ch4/flask/README_ch4_surface-flask_ccgg.html</att>

<att name="dataset_calibration_scale">CH4_X2004A</att>

<att name="dataset_description">Atmospheric Methane Dry Air Mole Fractions from the NOAA GML Global Greenhouse Gas Reference Network, Carbon Cycle Cooperative Global Air Sampling Network: 1983 - Present</att>

<att name="dataset_warning">Every effort is made to produce the most accurate and precise measurements possible. However, we reserve the right to make corrections to the data based on recalibration of standard gases or for other reasons deemed scientifically justified. We are not responsible for results and conclusions based on use of these data without regard to this warning.</att>

<att name="dataset_fair_use">These data are made freely available to the public and the scientific community in the belief that their wide dissemination will lead to greater understanding and new scientific insights. To ensure that GML receives fair credit for their work please include relevant citation text in publications. We encourage users to contact the data providers, who can provide detailed information about the measurements and scientific insight. In cases where the data are central to a publication, coauthorship for data providers may be appropriate.</att>

<att name="dataset_citation">Lan, X., G. Petron, K. Baugh, A.M. Crotwell, M.J. Crotwell, S. DeVogel, M. Madronich, J. Mauss, T. Mefford, E. Moglia, S. Morris, J.W. Mund, A. Searle, K.W. Thoning, S. Wolter and J. Miller (2025), Atmospheric Methane Dry Air Mole Fractions from the NOAA GML Global Greenhouse Gas Reference Network, Carbon Cycle Cooperative Global Air Sampling Network: 1983 - Present , Version: 2025-08-15, https://doi.org/10.15138/VNCZ-M766</att>

<att name="dataset_usage_description">Please cite the product's citation when using data from this dataset. Relevant literature references for this dataset are listed below for convenience.</att>

<att name="dataset_provider_license">These data were produced by NOAA and are not subject to copyright protection in the United States. NOAA waives any potential copyright and related rights in these data worldwide through the Creative Commons Zero 1.0 Universal Public Domain Dedication (CC0-1.0).</att>

<att name="dataset_reference_total_listed">1</att>

<att name="dataset_reference_1_name">Dlugokencky, E. J., L. P. Steele, P. M. Lang and K. A. Masarie, (1994), The growth rate and distribution of atmospheric methane, Journal of Geophysical Research, 99, D8, 17, doi:10.1029/94JD01245.</att>

<att name="title">Atmospheric trace gas measurements: flask_surface</att>

<att name="institution">NOAA Global Monitoring Laboratory</att>

<att name="summary">Unified aggregate for flask/surface. Promo metadata enables site/species filtering.</att>

<att name="infoUrl">https://gml.noaa.gov/aftp/data/trace_gases/ch4/flask/README_ch4_surface-flask_ccgg.html</att>

<att name="cdm_data_type">TimeSeries</att>

<att name="subsetVariables">site_code, dataset_parameter</att>

<att name="cdm_timeseries_variables">site_code, dataset_parameter</att>

</addAttributes>

<dataVariable>

<sourceName>time</sourceName>

<destinationName>time</destinationName>

<dataType>double</dataType>

<addAttributes>

<att name="ioos_category">Time</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>latitude</sourceName>

<destinationName>latitude</destinationName>

<dataType>double</dataType>

<addAttributes>

<att name="ioos_category">Location</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>longitude</sourceName>

<destinationName>longitude</destinationName>

<dataType>double</dataType>

<addAttributes>

<att name="ioos_category">Location</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>altitude</sourceName>

<destinationName>altitude</destinationName>

<dataType>double</dataType>

<addAttributes>

<att name="ioos_category">Vertical</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>value</sourceName>

<destinationName>value</destinationName>

<dataType>double</dataType>

<addAttributes>

<att name="ioos_category">Measurement</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>unique_sample_location_num</sourceName>

<destinationName>sample_id</destinationName>

<dataType>long</dataType>

<addAttributes>

<att name="ioos_category">Identifier</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>global:site_code</sourceName>

<destinationName>site_code</destinationName>

<dataType>String</dataType>

<addAttributes>

<att name="ioos_category">Identifier</att>

<att name="long_name">Site Code</att>

<att name="cf_role">timeseries_id</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>global:dataset_parameter</sourceName>

<destinationName>dataset_parameter</destinationName>

<dataType>String</dataType>

<addAttributes>

<att name="ioos_category">Identifier</att>

<att name="long_name">Species</att>

</addAttributes>

</dataVariable>

<dataVariable>

<sourceName>global:units</sourceName>

<destinationName>units</destinationName>

<dataType>String</dataType>

<addAttributes>

<att name="ioos_category">Identifier</att>

<att name="long_name">Units</att>

</addAttributes>

</dataVariable>

</dataset>

Let me know if you want more.

Thanks!

John

[Roy Mendelssohn - NOAA Federal]

Hi John:

I misunderstood your structure. This added info helped. Try instead:

<fileDir>/aftp/data/trace_gases/</fileDir>
<fileNameRegex>.*\.nc</fileNameRegex>
<recursive>true</recursive>

<pathRegex>.*/pfp/surface/nc/.*</pathRegex>

I believe that will get the regex correct. As I have said before, I consider regex to be one of the great mysteries of life. But give that a try and let me know. If it fails send just the type of info that you sent in this email, that is what I need.

Thanks,

-Roy

[John Mund - NOAA Affiliate]

Hi Roy,

pfp_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/surface/nc/.* time=6ms

Let me know if anything else would be helpful.

I’d be happy to turn regex over to our AI replacements if they didn’t hallucinate so much.

Thanks!

John

[Roy Mendelssohn - NOAA Federal]

Hi John:

Okay if possible can you update to version 2.30. That is what we use in any of our tests in trying to reproduce the behavior you are seeing and just makes our life easier

However, I am leaning again that you have a permissions problem. You have the user as 300 and the group as 201 can you tell me who or what that user and group are and how these data are being accessed? Is ti over an NFS mount or the like?

Can you run the following and send me the results:

find /aftp/data/trace_gases/ -type f -name "*.nc" | grep -P '.*/pfp/surface/nc/.*’

find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -20

sudo -u tomcat find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -5

One thing you can try if you are willing is set the owner of the entire directory and subdirectory to whatever user is running the tomcat, presumably user “tomcat”, and set the permissions all the way down to 775. If that works you can go back and tighten the permissions as needed.


If all else fails, then if you can send me say one file from /aftp/data/trace_gases/CHCl3/pfp/surface/nc and one file from /aftp/data/trace_gases/CHCl3/pfp/aircraft/nc and one file from /aftp/data/trace_gases/H2402/pfp/surface/nc and one file from /aftp/data/trace_gases/H2402/pfp/aircraft/nc. That will allow me to set up a similar structure on my own machine so I can much more easily see what is happening and try different solutions.

Thanks, and I am sorry this is causing so many problems.

-Roy

[John Mund - NOAA Affiliate]

Hi Roy,

Thanks for spending so much time on this, “Don’t do it this way” is totally an acceptable answer :). 

I don’t think it’s a permissions issue.  Mostly because I have setup variations of dataset configurations that work fine.  IE:

<fileDir>/aftp/data/trace_gases/co2/flask/surface/nc/</fileDir>

<fileNameRegex>.*\.nc</fileNameRegex>

<recursive>true</recursive>

<pathRegex>.*</pathRegex>

I also tried every file as a separate dataset (~5k datasets).

Tomcat runs as user root.  I log in as me then sudu su - tomcat to stop/start the server and create datasets.xml for these tests.

The ftp is an nfs mount the 300/201 users are artifacts of how this server is configured for a future home in the dmz.  It doesn’t have those users mapped, but all the directories and files are readable and enterable by the all user.  

Here are the commands you asked for:

[tomcat@erddap ~]$ find /aftp/data/trace_gases/ -type f -name "*.nc" | grep -P '.*/pfp/surface/nc/.*'|head -5

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wgc_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mko_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_lac_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_bao_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wkt_surface-pfp_97_hats_event.nc

[tomcat@erddap ~]$ find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -20

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wgc_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mko_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_lac_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_bao_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wkt_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mwo_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_lef_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_amt_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mlo_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_tmd_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_crv_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mbo_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_sgp_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_neb_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_nwb_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_bwd_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_lew_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_hfm_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wbi_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_msh_surface-pfp_97_hats_event.nc

Datasets Failed To Load (in the last major LoadDatasets) = 6

    flask_surface, in_situ_aircraft, in_situ_surface, in_situ_tower, pfp_aircraft, pfp_surface, (end)

Reasons for failing to load datasets: 

flask_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/flask/surface/nc/.* time=20ms

in_situ_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/aircraft/nc/.* time=6ms

in_situ_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/surface/nc/.* time=7ms

in_situ_tower: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/tower/nc/.* time=7ms

pfp_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/aircraft/nc/.* time=7ms

pfp_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/surface/nc/.* time=7ms

Unique users (since startup)  

[mund@erddap ~]$ sudo -u tomcat find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -5

[sudo] password for mund: 

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wgc_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_mko_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_lac_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_bao_surface-pfp_97_hats_event.nc

/aftp/data/trace_gases/CHCl3/pfp/surface/nc/CHCl3_wkt_surface-pfp_97_hats_event.nc

I upgraded to ERDDAP, Version 2.30.0 and am still getting this:

Datasets Failed To Load (in the last major LoadDatasets) = 6

    flask_surface, in_situ_aircraft, in_situ_surface, in_situ_tower, pfp_aircraft, pfp_surface, (end)

Reasons for failing to load datasets: 

flask_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/flask/surface/nc/.* time=23ms

in_situ_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/aircraft/nc/.* time=7ms

in_situ_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/surface/nc/.* time=8ms

in_situ_tower: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/in-situ/tower/nc/.* time=7ms

pfp_aircraft: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/aircraft/nc/.* time=7ms

pfp_surface: 0 files found in /aftp/data/trace_gases/

regex=.*\.nc recursive=true pathRegex=.*/pfp/surface/nc/.* time=6ms

See attached for example files.

Cheers,

[Roy Mendelssohn - NOAA Federal]

Thanks. Let me look at this. I still think it may be a permissions problem, but let me set it up on my own instance. The reason I think it may be a permissions problem is that it is possible in a linux system that the file itself has the appropriate permissions, so if you go directly to the file access is okay, but access may be blocked in an intermediate directory and ERDDAP™ has to check those because of the regex. Somewhere in the back of my memory I remember having this problem with an NFS share and it did require that the permissions be set based on users on the ERDDAP™ server, not the NFS server, but don’t quote me on that. I am like AI, I hallucinate a lot.

Since I have to set this up in my own ERDDAP™ it may take me a day or two to get to it, see if I can resolve any problems, and if not see if we can identify the issue. Will get back to you.

-Roy

> Cheers,<H2402_amt_surface-pfp_97_hats_event.nc><CHCl3_mlo_surface-pfp_97_hats_event.nc><CHCl3_acg_aircraft-pfp_97_hats_event.nc>John

>
>> On May 11, 2026, at 21:26, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:
>>
>> Hi John:
>>

>> Okay if possible can you update to version 2.30. That is what we use in any of our tests in trying to reproduce the behavior you are seeing and just makes our life easier
>>
>> However, I am leaning again that you have a permissions problem. You have the user as 300 and the group as 201 can you tell me who or what that user and group are and how these data are being accessed? Is ti over an NFS mount or the like?
>>
>> Can you run the following and send me the results:
>>

>> find /aftp/data/trace_gases/ -type f -name "*.nc" | grep -P '.*/pfp/surface/nc/.*’

>>
>> find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -20
>>

>> sudo -u tomcat find /aftp/data/trace_gases/ -type f -name "*.nc" -path "*/pfp/surface/nc/*" | head -5
>>

>> One thing you can try if you are willing is set the owner of the entire directory and subdirectory to whatever user is running the tomcat, presumably user “tomcat”, and set the permissions all the way down to 775. If that works you can go back and tighten the permissions as needed.
>>
>>
>> If all else fails, then if you can send me say one file from /aftp/data/trace_gases/CHCl3/pfp/surface/nc and one file from /aftp/data/trace_gases/CHCl3/pfp/aircraft/nc and one file from /aftp/data/trace_gases/H2402/pfp/surface/nc and one file from /aftp/data/trace_gases/H2402/pfp/aircraft/nc. That will allow me to set up a similar structure on my own machine so I can much more easily see what is happening and try different solutions.
>>
>> Thanks, and I am sorry this is causing so many problems.
>>
>> -Roy
>>
>>
>>> On May 11, 2026, at 5:20 PM, John Mund - NOAA Affiliate <john...@noaa.gov> wrote:
>>>
>>> Hi Roy,
>>>

>>> pfp_surface: 0 files found in /aftp/data/trace_gases/
>>> regex=.*\.nc recursive=true pathRegex=.*/pfp/surface/nc/.* time=6ms
>>>

>>> Let me know if anything else would be helpful.
>>>
>>> I’d be happy to turn regex over to our AI replacements if they didn’t hallucinate so much.
>>>
>>> Thanks!
>>>
>>> John
>>>
>>>
>>>> On May 11, 2026, at 15:19, Roy Mendelssohn - NOAA Federal <roy.men...@noaa.gov> wrote:
>>>>
>>>> Hi John:
>>>>
>>>> I misunderstood your structure. This added info helped. Try instead:
>>>>

>>>> <fileDir>/aftp/data/trace_gases/</fileDir>

>>>> <fileNameRegex>.*\.nc</fileNameRegex>
>>>> <recursive>true</recursive>

>>>>> <fileDir>/aftp/data/trace_gases/</fileDir>

>>>>> <fileNameRegex>.*\.nc</fileNameRegex>
>>>>> <recursive>true</recursive>

[Divoll, Timothy]

Hello, 

Just chiming in with an idea. I deployed our ERDDAP on a Kubernetes cluster and to get the permissions right, I had to set some environment variables so that tomcat would run as the desired user, which in our case is a service account. The same service count is the user on our NFS volume mount. The two important variables to set are TOMCAT_USER_ID and TOMCAT_GROUP_ID. The value need to be the numeric UID of the desired users and groups. For example, running `id -u myuser` should return the UID for myuser. I was getting permissions errors before I added these TOMCAT mappings to the deployment.

I don't know if this will solve any issues in your case, but it's at least something to try.
--

Timothy Divoll, PhD    Senior Data Scientist Center for Computation and Visualization Brown University | OIT  Pronouns: he/him/his

--
You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.

To view this discussion, visit https://groups.google.com/d/msgid/erddap/ECED0EAE-980E-4C06-A068-DDCE8F0D40F3%40noaa.gov.

[Roy Mendelssohn - NOAA Federal]

Hi Timothy:

Thanks for pitching in. The community has a broad base of knowledge that we can’t possibly be experts in all of the facet related to ERDDAP™ deployment. While i am pretty certain John’s problem has to do with NFS, I want to replicate his setup first (will likely work on that today) without NFS so that I can feel confident that there isn’t a bug in ERDDAP™,

On a related note, if you look at the present ERDDAP™ docs, we (or more properly Chris) are trying to add guides for users and admins to provide guidance in interesting use cases. For example I am writing up with examples on how to have videos in ERDDAP™. Your setup sounds interesting, and if you ever have the time and inclination to write up what you have done and how you do it, that would be great. I encourage that for others also. If you have an interesting setup that could be of use to others, or overcame a problem that we do not have well documented in the docs, please consider writing it up.

Thanks,

-Roy

> On May 13, 2026, at 3:50 AM, Divoll, Timothy <timothy...@brown.edu> wrote:
>
> Hello,
>
> Just chiming in with an idea. I deployed our ERDDAP on a Kubernetes cluster and to get the permissions right, I had to set some environment variables so that tomcat would run as the desired user, which in our case is a service account. The same service count is the user on our NFS volume mount. The two important variables to set are TOMCAT_USER_ID and TOMCAT_GROUP_ID. The value need to be the numeric UID of the desired users and groups. For example, running `id -u myuser` should return the UID for myuser. I was getting permissions errors before I added these TOMCAT mappings to the deployment.
>
> I don't know if this will solve any issues in your case, but it's at least something to try.
> --

> Timothy Divoll, PhD

[John Mund - NOAA Affiliate]

Hi Roy. Don’t spend too much time on this unless you really want to figure it out. The team has decided that they want to skip the regex species aggregation and keep the datasets separate to preserve accurate metadata. Thank you for your help on this!

John

[Roy Mendelssohn - NOAA Federal]

Hi John:

Thanks for providing the files, that allowed me to look more closely at what you have and what you are trying to do. While there still may or may not be permission problems, I believe there is an underlying problem that given your file structure “EDDTableFromNcFiles” is not the correct choice for your data as it doesn’t have the strict grid structure expected. The structure is more of the cdm_data_typee, and that is where you want to begin (for example for strict grids “time”, “latitude”, “longitude”, “altitude” are reserved terms for coordinates). Also I noticed that there are some variables that are in the file that had differing dimensions. The options for these include EDDTableFromMultidimNcFiles and EDDTableFromNcCFFiles.

There are several options for theseV type files, Chris has made a number of updates on the various options based on feedback from users with similar files and he is better person to look at this and suggest the best option. I should add that Chris has been on an extended leave and is just slowly returning to work, I am happy to be able to say that this is because he and Sara are the proud parents of a baby girl. So while we usually try to respond quickly this might be delayed a bit more than usual. I believe the sample data files you provided should be enough for him to see what is going on, I am sure he will contact you if he has questions.

[John Mund - NOAA Affiliate]

Awesome, thanks Roy! I really appreciate you looking into this form me.

[Roy Mendelssohn - NOAA Federal]

I should add that besides my usual typos because I am old and careless, John also emailed me privately because of the time spent on this. This is important to us for two reasons. The first is that we want people to get data into ERDDAP™ the way they want, and several of the improvements in ERDDAP™ have come from user feedback on similar file structures. Those comments, suggestions and requests were really helpful and improved ERDDAP™.

Second. I wanted to make certain that there wasn’t a bug in how ERDDAP™ was interpreting the regex. Finding and removing bugs are high priority.

Finally I would add that once the correct EDD datatype has been suggested, please use GenerateDatasetsXml.sh to start with. The result will likely need to be edited to have all the info wanted (such as some global attributes are being elevated) but it will get you most of the way there with the fewest problems. I have created a lot of ERDDAP™ datasets.xml over the years (we are approaching 20 years) and I still start with GenerateDatasetsXml.sh to add new datasets.

HTH,

-Roy

[Divoll, Timothy]

Hello Roy, 

I'd be happy to write up a description of our configuration for deploying ERDDAP with Kubernetes for the Server Administration section of the ERDDAP docs. We use the official ERDDAP Docker image and add an NFS mount on a server for the data, then deploy the service on our Kubernetes server (different than where the storage mount is).

I'll start an issue at https://github.com/ERDDAP/erddap.github.io and add a markdown file to the server admin section.

Thank you, Tim