Add new dataset & create flag dataset does not show

[callum rollo]

Hi all,

We're having a small problem with our ERDDAP. I think it only started when we upgraded to 2.23, but I could be wrong. When I add a new dataset it does not show on ERDDAP until I restart the Docker image or reboot the server. I add a new dataset to datasets.xml with a datasetID like nrt_SEA056_M62. I then create a flag to tell ERDDAP to load the dataset immediately with:

touch /media/data/erddapData/flag/nrt_SEA056_M62

The file is created then, a second later, removed, presumably by ERDDAP. However, the dataset does not show on our server. I know that the xml is correct. It passes the checker and after rebooting the server, the new dataset appears.

Here's the top lines of the xml. More details at the end of this message.

<dataset type="EDDTableFromMultidimNcFiles" datasetID="nrt_SEA056_M62" active="true">
  <reloadEveryNMinutes>10080</reloadEveryNMinutes>
  <updateEveryNMillis>10000</updateEveryNMillis>
  <fileDir>/Data/nrt/SEA56/M62/timeseries/</fileDir>
  <fileNameRegex>.*</fileNameRegex>
  <recursive>true</recursive>
  <pathRegex>.*</pathRegex>
  <metadataFrom>last</metadataFrom>
  <standardizeWhat>0</standardizeWhat>
  <removeMVRows>true</removeMVRows>
  <sortFilesBySourceNames />
  <fileTableInMemory>true</fileTableInMemory>

How can I make ERDDAP recognise and load new datasets?

Thanks

Callum

Potentially relevant info:

ERDDAP https://erddap.observations.voiceoftheocean.org/erddap

version 2.23

Running on Ubuntu 22.04 with the Axiom docker latest build

pastebin of full dataset xml https://pastebin.com/2GTPDQsF

[Marco Alba]

Hi Callum,

when I add a new datasets I set an hardFlag instead of a flag and the layer is loaded without restarting the docker or the server.

Also, have you checked the log to see what happen when ERDDAP finds the flag file?

Marco

[callum rollo]

Hi Marco,

Thanks for the suggestion. I have tried with hardFlag and the result is the same. Checking the log did give me some ideas that eventually lead to a solution though.

tl;dr: Issue was with docker, not ERDDAP.

Here's the explanation and fix in case anyone has similar problems in future:

datasets.xml was not updating within docker when it was changed on the host machine. When I rebooted the docker container, the datasets.xml synced again and the dataset was processed by ERDDAP and displayed on the server. This is why restarting the container fixed the problem. The reason that datasets.xml stopped syncing was inodes. We use a linux host machine and mount our datasets.xml file to the docker container with:

-v /path/to/our/erddap/directory:/datasets.xml:/usr/local/tomcat/content/erddap/datasets.xml

If you mount a file like this then edit the file on the hosts machine, you can cause the inode to change https://stackoverflow.com/questions/71344789/docker-mount-not-updating-the-files. Check this by calling ls -i on datasets.xml on both host and container and see if they are the same. If not, datasets.xml on the container will stop syncing with datasets.xml on the host.

The solution was to mount the directory instead with

 -v /path/to/our/erddap/directory:/usr/local/tomcat/content/erddap

For docker beginners like me, this kind of thing isn't intuitive!

Callum

--
You received this message because you are subscribed to a topic in the Google Groups "ERDDAP" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/erddap/EIGCjGi3nxg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to erddap+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/7aad9006-dcfa-403a-91f7-ef3da6a094f9n%40googlegroups.com.

[Mathew Biddle - NOAA Federal]

Callum,

I added your findings to the Quick Start Guide for ERDDAP in a Docker Container. See the PR below if I've represented your findings correctly:

https://github.com/ioos/erddap-gold-standard/pull/46

Thanks,

Matt

You received this message because you are subscribed to the Google Groups "ERDDAP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/erddap/CAH67JrqYBva-%2BT5XiDTRzGACy9u%3DuEp7LPO_6dUW%3Dgz%3D2V-6%3Dw%40mail.gmail.com.

--

Mathew Biddle, Physical Scientist
NOAA/NOS
US Integrated Ocean Observing System Office
1315 East-West Highway
Silver Spring MD 20910

240-533-9470

ORCiD: 0000-0003-4897-1669

http://www.ioos.noaa.gov/