Occasional issues updating an ERDDAP dataset when the source file is modified automatically
Antoine Biehler
Hi,
I'm having a small issue with an ERDDAP page that is updated frequently. To give you some context, this is a Griddap subset of the HRDPS meteorological model that combines hourly data covering the past 48 hours and the next 48 hours, and is updated every 6 hours. This data is mainly made available through a connection between ERDDAP and our navigation app Nautilo, which displays the graphs when users create virtual observation stations on the map.
This works perfectly 95% of the time, but every now and then, the ERDDAP dataset stops updating, even though the source NetCDF file has been updated. My hypothesis is that the problem occurs when no queries are sent to the dataset for several days, but I’m not really sure. It could also be because the source file is quite large (about 2 GB), but since it works most of the time, I don’t really understand how that could be the cause of the problem. I’ve tried changing the reloadEveryNMinutes and updateEveryNMillis settings in the dataset’s XML file, but that doesn’t seem to solve the problem. As a workaround, whenever the ERDDAP page stops updating from the source file, we set a flag with the url for that specific dataset (based on the pattern: {erddap_flag_url}?datasetID={dataset_id}&flagKey={flag_key}). This works, but it’s not very practical, since our application needs to be available at all times as a maritime navigation resource.
Do you know if there’s a way to solve this problem using a specific attribut or parameter that I could modify in the dataset’s xml file? Or should I create an external cron job that periodically sets an hardflag for the dataset ?
For information, we use ERDDAP version 2.30.0.
Here is a link to the affected dataset: https://erddap.ogsl.ca/erddap/griddap/ecccHrdpsNautilo.html
Thanks,
--
Antoine
Roy Mendelssohn - NOAA Federal
Thank you for your inquiry, If I had to guess what is happening is that every once in awhile the ERDDAP™ major update doesn’t complete in the given time. I would suggest trying several things.
1. Have the program that updates the data itself set the flag. That way the flag is set with each update of the data at the time of the data update. We use this a lot.
2. Increase the maximum time allotted for a major update in the settings.
3. Move this dataset higher up in the datasets.xml as ERDDAP™ updates in order of occurrence in the file.
4. Look at the update settings for all of your datasets and see if some are updating much more frequently than is needed, If that is the case, changing those will put much less burden on the dataset update.
HTH and let me know how things progress,
-Roy
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/73c65891-a324-420c-bf08-a3268938ee89n%40googlegroups.com.