Problems with OPendap and THREDDS access on weekends

[Brown, Murray]

Dearl HYCOM Folks,

 

I know it sounds nuts, but I usually work on UNESCO training lessons on Sunday, using UNIDATA's IDV software.  It seems like I often have trouble getting the OPeNDAP and THREDDS links to open successfully on Sunday.  The attached image is of the error message I get over and over.  Yesterday, everything was fine...today, nothing but this msg.

 

Somebody please put me out of my misery and let me know what i'm doing wrong, or tell me if Sundays are HYCOM's day off.  Thanks for the help.

 

Murray

 

 

Murray Brown
1710 Washington St., Apt. B
New Smyrna Beach, Florida 32168
United States of America
386.428.6367
http://oceanexpert.org/viewMemberRecord.php?&memberID=242

[Michael McDonald]

Murray,

The OPeNDAP URL you were using was cutoff from view in the provided
screenshot. Can you please reply with this URL. Thanks.

Regarding the training lessons, is there an entire classroom of
students hammering the HYCOM.org THREDDS server at the same time every
Sunday? If so, how many students/users are making queries?

>  error4hycom.jpg
> 59KViewDownload

[Brown, Murray]

Good Morning Michael,

The OPeNDAP URL I usually use in IDV is
http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9 and the THREDDS URL is
http://tds.hycom.org/thredds/catalog.xml.

I am working alone here as a UNESCO volunteer, with no students "hammering"
anything. The intended courses will be taught in August in Mexico and
September in Belgium. I'm simply trying to update my lessons in good time
for early review by colleagues.

Murray

[Michael McDonald]

Murray,

> The OPeNDAP URL I usually use in IDV is
> http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
> and the THREDDS URL is
> http://tds.hycom.org/thredds/catalog.xml.

Good. Both of these URLs are correct and in working order.

> I am working alone here as a UNESCO volunteer, with no students "hammering"
> anything.  The intended courses will be taught in August in Mexico and
> September in Belgium.  I'm simply trying to update my lessons in good time
> for early review by colleagues.

But you see the direction I was leaning in with my logic. You "will
eventually" be in a situation where there "will be" a group of
students sitting behind powerful desktops, simultaneously querying the
HYCOM.org THREDDS server. I'm assuming they will be following your
"Direct Connect" Exercise here,
(http://marinedataliteracy.org/ops/circ_mod_direct.htm). I cannot
guarantee that a sudden burst of traffic will or will not overload our
servers. Your "Download" method of using NetCDFSubset to obtain a
pre-canned dataset would be a wise backup option if users start
experiencing timeouts using IDV
(http://marinedataliteracy.org/ops/circ_mods.htm)

Regarding the timeouts you were experiencing every Sunday, I too was
able to get this error (once), not on a Sunday. When this happened I
emailed unidata directly (via the 'Support Form' button the error box)
to inquire unidata what this "Read timed out" value is set to in IDV.
The THREDDS server *could* have been returning data properly, but IDV
simply gave up waiting. I'll follow up if/when I find out from unidata
what this timeout value is set to and if it can be overridden. I don't
think that Sunday mornings are any different from any other morning
(access/server-wise). The post-processing and publishing of the
GLBa0.08/expt_90.9 data *was* taking place daily between
START@8PM(EST) and FINISH@9AM(EST), roughly 13 hours. However, the
actual publishing (writing) of data files only occurs at the very end
of post-processing, usually between ~ 6AM EST and 9AM EST). This was
occurring daily and I now see how this might have been giving you and
others intermittent errors with access to data. It essentially
overwrites the last 10~15 data files with the best hindcast data and
new forecast files. Immediately following this the tomcat server
usually gets restarted with the new catalog info (we have a mixture of
legacy static catalogs and dynamic "scan" catalogs). I'll look into
moving this "publishing" period to off peak hours. It's just a matter
of starting the conversion process earlier in the day.


FYI, There are several possible pitfalls that users might encounter
when using the IDV tool that I noticed with just a few minutes of
usage:

1. By default, IDV will load ALL time-series values (Time tab, "Use
default" is checked) in an aggregated dataset (e.g.,
http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9) unless
otherwise specified. I'm glad that you instruct users to only select
the days of data that they need, otherwise students should be prepared
to wait a long while for all the days/frames to download.

2. Have you tried using the rectilinear version of the
GLBa0.08/expt_90.9 data in IDV for your analysis
(http://tds.hycom.org/thredds/catalog/datasets/global/GLBa0.08_rect/data/catalog.xml)?
This might help with views in the higher and lower latitudes. It's not
an aggregated dataset, but available via a similar catalog access
method which IDV supports.

The "datasets" catalog is a mirror of what's available via our FTP
server (ftp.hycom.org),
http://tds.hycom.org/thredds/datasets.xml

This is a non-aggregated single-file access method for all of the
datasets we have. It's essentially direct OPeNDAP access to all the
individual netCDF files. This is usually a much quicker and more
reliable method of access.

[Brown, Murray]

Good Morning Michael,

1. Both URL's (THREDDS and OPeNDAP) just timed out on me this morning,
using IDV. I agree with you this may be a key concept.

2. I can group the students, as needed, to explore both URL's and the
subset method to avoid overload. As you can imagine, when they come from
Togo or Pago Pago, they are customarily extremely conservative when
downloading large resources.

3. It seems to me that there was some housekeeping overnight, as the folder
titles seem different. I hope this helps. Your "latest 10 days" is an
excellent idea. We are principally interested in only the current analyses
and sometimes the forecast grids, never that I can think of with legacy
grids. I'd even suggest that you add a "latest" as they do with many other
global systems to mirror the most recent product, with a re-usable name.

4. What is your definition of rectilinear?

5. Yes, http://tds.hycom.org/thredds/catalog.xml could be a good route for
single-file selection, but in fact even as I'm writing this your server
seems to have died again, and none of the links (FTP included) associated
with the integrated resource seems to be working.

Murray



----- Original Message -----
From: "Michael McDonald" <michael....@hycom.org>
To: "Brown, Murray" <m.bro...@gmail.com>
Cc: "HYCOM.org Forum" <fo...@hycom.org>
Sent: Thursday, May 24, 2012 6:02 PM
Subject: Re: Problems with OPendap and THREDDS access on weekends