Occasional issues updating an ERDDAP dataset when the source file is modified automatically
Antoine Biehler
Hi,
I'm having a small issue with an ERDDAP page that is updated frequently. To give you some context, this is a Griddap subset of the HRDPS meteorological model that combines hourly data covering the past 48 hours and the next 48 hours, and is updated every 6 hours. This data is mainly made available through a connection between ERDDAP and our navigation app Nautilo, which displays the graphs when users create virtual observation stations on the map.
This works perfectly 95% of the time, but every now and then, the ERDDAP dataset stops updating, even though the source NetCDF file has been updated. My hypothesis is that the problem occurs when no queries are sent to the dataset for several days, but I’m not really sure. It could also be because the source file is quite large (about 2 GB), but since it works most of the time, I don’t really understand how that could be the cause of the problem. I’ve tried changing the reloadEveryNMinutes and updateEveryNMillis settings in the dataset’s XML file, but that doesn’t seem to solve the problem. As a workaround, whenever the ERDDAP page stops updating from the source file, we set a flag with the url for that specific dataset (based on the pattern: {erddap_flag_url}?datasetID={dataset_id}&flagKey={flag_key}). This works, but it’s not very practical, since our application needs to be available at all times as a maritime navigation resource.
Do you know if there’s a way to solve this problem using a specific attribut or parameter that I could modify in the dataset’s xml file? Or should I create an external cron job that periodically sets an hardflag for the dataset ?
For information, we use ERDDAP version 2.30.0.
Here is a link to the affected dataset: https://erddap.ogsl.ca/erddap/griddap/ecccHrdpsNautilo.html
Thanks,
--
Antoine
Roy Mendelssohn - NOAA Federal
Thank you for your inquiry, If I had to guess what is happening is that every once in awhile the ERDDAP™ major update doesn’t complete in the given time. I would suggest trying several things.
1. Have the program that updates the data itself set the flag. That way the flag is set with each update of the data at the time of the data update. We use this a lot.
2. Increase the maximum time allotted for a major update in the settings.
3. Move this dataset higher up in the datasets.xml as ERDDAP™ updates in order of occurrence in the file.
4. Look at the update settings for all of your datasets and see if some are updating much more frequently than is needed, If that is the case, changing those will put much less burden on the dataset update.
HTH and let me know how things progress,
-Roy
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.
> To view this discussion, visit https://groups.google.com/d/msgid/erddap/73c65891-a324-420c-bf08-a3268938ee89n%40googlegroups.com.
Antoine Biehler
Hi Roy,
Thank you for your suggestions! Sorry for the delay in my response. Since the problem seems to occur fairly randomly, it's difficult to verify the effectiveness of our fixes.
We checked for any unnecessary updates to the datasets and reviewed the order of those updates, but did not find anything that needed to be corrected. So we tried your first suggestion of having the program set a flag, but at first, that didn’t seem to solve the problem. Since the updated dataset is very large, the time between the dataset being fully updated on the server and the flag being set may have been too short. We therefore reviewed our overall pipeline by refining the management of the steps, and so far, the problem has not recurred (fingers crossed). In addition, we started working on a monitoring system that allows us to perform continuous health checks on our datasets. We’re keeping your suggestion to increase the maximum time allowed for a major update in mind in case the problem resurfaces.
Thank you again for your help! We'll be sure to keep you updated if we end up implementing another solution regarding this issue.
--
Antoine
Roy Mendelssohn - NOAA Federal
-Roy
> You received this message because you are subscribed to the Google Groups "ERDDAP" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to erddap+un...@googlegroups.com.