Re: hycom download problem

Michael McDonald

unread,

Apr 21, 2012, 3:54:52 PM4/21/12

to forum, George R. Halliwell

George,

First possible reason, the GOMl0.04/expt_31.0 dataset has not
completed converting to netCDF yet. There is data at the beginning and
data at the end, but e.g., there is all of 2011 that is missing. The
post-processing server is catching up on creating this data.

Second likely reason, the direct opendap access method is more
responsive w/o thredds aggregation. Which leads me to think that all
of the hourly GOM data (currently at over 15,000 files) aggregated
into a single OPeNDAP dataset is bogging down our thredds server.
Although very slow, I have not gotten a timeout accessing this data
using ferret, it just takes a while as you have also experienced.

e.g. try this non-aggregated thredds access method.

http://tds.hycom.org/thredds/catalog/datasets/GOMl0.04/expt_31.0/data/catalog.html

--
Michael McDonald
HYCOM.org Administrator
http://hycom.org

On Sat, Apr 21, 2012 at 1:48 PM, George R. Halliwell
<george.h...@noaa.gov> wrote:
> Michael:
>
> I'm having problems with downloading hycom fields from the 1/25 GoM model.
> When I execute the program to download the data, I get the 'opening file'
> line, and then nothing else happens:
>
> user@host: ~/hycom_opendap/gom_hycom_realtime>./get_hycom_opendap
> Opening file -->http://tds.hycom.org/thredds/dodsC/GOMl0.04/expt_31.0
>
> The session remains frozen for hours. In contrast, this program has no
> problem downloading fields from the global hycom, expt 90.9. Given that
> fact, are there any problems on your side that could be causing this?
>
> Thanks in advance,
> George H.

George R. Halliwell

unread,

Apr 23, 2012, 12:16:09 AM4/23/12

to Michael McDonald, forum, ashwanth srinivasan

Hellp Michael:

The non-aggregated access method does not work with our hycom_opendap
software - see the errors below.

The aggregated access that works with global hycom but not gom expt_31.0
essentially freezes permanently. I left one attempt running all night
and it still remained frozen. I was attempting to download April 2012
files, so that should not be a problem.

Ashwanth, can you download gom hycom expt_31.0 files using your version
of get_hycom _opendap?

Thanks for your help - I will keep trying.

George H.

ghalliwell@phodmod:~/hycom_opendap/gom_hycom_realtime>./get_hycom_opendap
Opening file -->
http://tds.hycom.org/thredds/catalog/datasets/GOMl0.04/expt_31.0/data
syntax error
context: <htm^><head><title>Apache Tomcat/6.0.33 - Error
report</title><style></style>
</head><body><h1>HTTP Status 400 - Bad request: The "path"
[datasets/GOMl0.04/expt_31.0/data.dds] field must end in
".html".</h1><HR size="1" noshade="noshade"><p><b>type</b>
Status report</p><p><b>message</b> <u>Bad request: The "path"
[datasets/GOMl0.04/expt_31.0/data.dds] field must end in
".html".</u></p><p><b>description</b> <u>The request sent by
the client was syntactically incorrect (Bad request: The
"path" [datasets/GOMl0.04/expt_31.0/data.dds] field must end
in ".html".).</u></p><HR size="1" noshade="noshade"><h3>Apache
Tomcat/6.0.33</h3></body></html
error - from NetCDF library call
NetCDF: DAP server error

--
Dr. George R. Halliwell, Jr.

NOAA/AOML/PhOD Telephone: (305) 361-4346
4301 Rickenbacker Causeway Fax: (305) 361-4392
Miami, FL 33149-1026 Mobile: (305) 972-3975

Web Page: http://www.aoml.noaa.gov/phod/people/ghalliwell.html
(available soon)

The contents of this message are mine personally and do not
necessarily reflect any position of the U.S. Government or of
the National Oceanic and Atmospheric Administration (NOAA).

Michael McDonald

unread,

Apr 23, 2012, 12:22:35 PM4/23/12

to George.H...@noaa.gov, forum

George,

> The non-aggregated access method does not work with our hycom_opendap
> software - see the errors below.

The OPeNDAP URL you specified
(http://tds.hycom.org/thredds/catalog/datasets/GOMl0.04/expt_31.0/data)
is not valid. OPeNDAP cannot do directory listings. The data URL is
either an aggregation/concatenation (i.e., the
http://tds.hycom.org/thredds/dodsC/GOMl0.04/expt_31.0 URL, which as
you stated previously is timing out for you), or it points to
*individual* netCDF files, which in essence become the OPeNDAP URLs.
The URL looks like you would be able to do a wget on it, but it is in
fact an OPeNDAP data URL.

e.g., See the THREDDS "Access" methods page here,
http://bit.ly/Ialxe9
OR
http://tds.hycom.org/thredds/catalog/datasets/GOMl0.04/expt_31.0/data/catalog.html?dataset=datasets/GOMl0.04/expt_31.0/data/archv.2012_118_00_3z.nc

##
Catalog http://tds.hycom.org/thredds/catalog/datasets/GOMl0.04/expt_31.0/data/catalog.html
Dataset: data/archv.2012_118_00_3z.nc
Access:
OPENDAP: http://tds.hycom.org/thredds/dodsC/datasets/GOMl0.04/expt_31.0/data/archv.2012_118_00_3z.nc
FTP: ftp://ftp.hycom.org/datasets/GOMl0.04/expt_31.0/data/archv.2012_118_00_3z.nc
##

So, for all of April you would need to write code to cycle through all
of the days/files (@00z or any requested hourly value you wanted)

e.g., here is some test code to see if you can pull out just the ascii
'Date' variable from each of the April data files remotely using:
bash, seq, and curl.

My day of the year calendar is not perfect (i.e., no 31st day in
April) but others might find helpful, http://hycom.org/tools/calendar)

#!/bin/bash
for doy in `seq -w 91 121`; do curl
http://tds.hycom.org/thredds/dodsC/datasets/GOMl0.04/expt_31.0/data/archv.2012_${doy}_00_3z.nc.ascii?Date;
done

Note how I simply added on the ".ascii?Date" part to the end of the
OPeNDAP URL to test its responsiveness. THREDDS supports this mode in
addition to binary (dods) mode.

> I was attempting to download April 2012 files, so
> that should not be a problem.

It possibly could be a problem if the request was for the last three
days of the month (of April) and the program did not take into account
missing data files...

Reply all

Reply to author

Forward