Problems downloading HYCOM data with getdata

[sally.wood]

Hi all, 

is anyone trying to download HYCOM data using getdata at the moment? I am having problems with the download constantly crashing (every half hour or so). I have downloaded this same data in the past with getdata and not had any problems. I have posted on the HYCOM forum but they don't think there's a problem at their end (see https://groups.google.com/a/hycom.org/forum/#!topic/forum/rfde8IEafnY).

I am trying to download the global 1/12 degree reanalysis data (surface layer u, v, SST and salinity) for 1999 from http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1. Here are the various error messages I've been getting (in the output file) after each crash - they are not the same each time: 

 Error in the netCDF file: http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1

 NetCDF: Malformed or inaccessible DAP DDS

 Error in the netCDF file: http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1

 NetCDF: Malformed or inaccessible DAP DATADDS

 Error in the netCDF file: http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1

 NetCDF: I/O failure

syntax error, unexpected WORD_STRING, expecting WORD_WORD

context: Error { code = 500; message = "java.io.IOException: java.net.SocketTimeoutException: Read timed out"^;};

syntax error, unexpected WORD_STRING, expecting WORD_WORD

context: Error { code = 500; message = "java.io.IOException: opendap.dap.DAP2Exception: ucar.nc2.util.net.HTTPException: java.net.SocketTimeoutException: Read timed out"^;};

After each crash, I re-start the download from the last date I have, the download starts again and the same thing happens within about half an hour. Any ideas?

Cheers,

Sally

[Qamar Schuyler]

Hi Sally,

I am having the same problem you are. I get the following errors. Sometimes it doesn't even go half an hour! I'll post in the same forum...maybe if enough of us do so they will get it!

Qamar

Error in the netCDF file: http://tds.hycom.org/thredds/dodsC/GLBa0.08/reanalysis

 NetCDF: file not found

curl error details: 

 Error in the netCDF file: http://tds.hycom.org/thredds/dodsC/GLBa0.08/reanalysis

 NetCDF: I/O failure

[sally.wood]

Having this problem again today, get about 7-9 nest files, it crashes, I restart the download and get another 7 files before it crashes. I get the same problem using both http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2011 and http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1 (i.e. the link to all the data).

Not expecting a solution, just logging it here for info

 Curl error details...

 Error in the netCDF file: 

 http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2011

 NetCDF: I/O failure

[sally.wood]

Hi, this latest error seems to have been a problem with getdata unnecessarily opening an extra netcdf file for every nest file it creates (and leaving it open), so that the number of open connections to the HYCOM server increases as the code loops over each nest file. While I guess this has always been the case with getdata, something along the line, external to CMS, must have been changed recently which limits the number of open connections to the HYCOM server. This has now been fixed in mod_getdata.f90.

This doesn't, however, account for earlier error messages, which were likely an issue at the HYCOM server end.

Sally

[Claire PARIS]

Hi Sally - thank you for debugging  this, it is is an important one!

Claire

  

Claire B. PARIS-LIMOUZY

claire...@gmail.com

--
You received this message because you are subscribed to the Google Groups "connectivity-modeling-system-club" group.
To unsubscribe from this group and stop receiving emails from it, send an email to connectivity-modeling-...@googlegroups.com.
To post to this group, send email to connectivity-mod...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[sally.wood]

I can't really take the credit for this - it was mainly the work of our High Performance Computing support guys.

There is still the issue of getdata querying the .dds/.das for each run through of the nest files loop (in SUBROUTINE readdata), but I've left this for now as I don't understand the code well enough so worry I will mess things up elsewhere in the script (i.e. readdata is called from other subroutines not just SUBROUTINE make_nestfiles_opendap.

Cheers,

Sally

  

Claire B. PARIS-LIMOUZY

claire...@gmail.com

To unsubscribe from this group and stop receiving emails from it, send an email to connectivity-modeling-system-club+unsubscribe@googlegroups.com.
To post to this group, send email to connectivity-modeling-system-cl...@googlegroups.com.

[salwood82]

Note I've opened an issue on this in the github, as I still have problems with getdata crashing regularly:

https://github.com/beatrixparis/connectivity-modeling-system/issues/17

Here is where we have got to in figuring it out:

getdata first opens a network connection and makes 5 queries; 1 to get the data on depth, lat, time and tau, 1 to get the data on lon (why is this done separately, for offset grids?), 2 for dds (why 2?) and 1 for das.

For each nest file downloaded it then opens a network connection for each variable to download (e.g. 1 for uvel and 1 for vvel), one after the other, and makes again 5 queries for each; 1 for the variable, 1 for depth, lat, time and tau (this seems unnecessary), 1 for das and 2 for dds (which as Micheal suggests may be unnecessary).

So while it only ever has 1 network connection open at once, it is making a lot of (unnecessary?) queries = a lot of bandwidth required, which slows the download down and I think might also be causing it to crash often:

Note I'm now writing a Matlab script to download the HYCOM data, which can then be formatted for use in CMS using getdata. I'll post here when it's done. If anyone else has an alternative script for downloading HYCOM data please share.

Cheers,

Sally

[Ana Vaz]

Hi Sally,

Thank you so much for looking into this - it does look like there are too many unnecessary queries - I also have the issue of crashing when using it, and I will be soon looking into it as well. 

I have an older version of a code to create nests files using loaddap to download nests (make_input_CMS_GOM_2010), and a newer one to download wind using the matlab built in functions. Combining both can give an updated alternative method to create nest files using Matlab.

But I believe if we update the get data it should work much faster than the matlab code will.

Let me know if anyone gets any questions on these. 

Cheers,

Ana

--

-----------------------------------------------------------------
Ana Carolina Vaz

Assistant Scientist

Rosenstiel School of Marine & Atmospheric Science

University of Miami

4600 Rickenbacker Causeway, Miami, 33149

http://www.rsmas.miami.edu/users/avaz/

------------------------------------------------------------------

[Rebecca Ross]

Hi all,

I usually use a wget script to download hycom files, adapted from one shared by Michael MacDonald on the hycom forum a while ago. You can try that in the meantime if you like?

1. Copy the code below into a text file on your linux machine and save it where you want to download to.

2. Edit for your dates/experiment/bounding box

3. Run the script using bash script_filename.txt (while in the folder you saved it to)

4. I usually use getdata to reformat the downloaded files into CMS optimised file structures.

Note I haven't used it in a while so there may be some adaptations needed, and I'm not sure whether it will overcome the crashing issue or not, but I am pretty sure it will make several reattempts on its own. Give it a bash. ;) (pun intended)

All the best,

Bex

--------------

#!/bin/bash

 

WGET='/usr/bin/wget'

 

YEAR='2008'

MONTH='09'

DAY='19'

StartSeq='0'

EndSeq='229'

 

NCSS='http://ncss.hycom.org/thredds/ncss/grid'

MODEL='GLBa0.08'

EXPT='expt_90.6'

 

VARS="var=u,v"

Subset='spatial=bb'

NORTH='north=70'

SOUTH='south=40'

EAST='east=0'

WEST='west=-30'

 

for PlusDay in `seq $StartSeq $EndSeq`; do

 

  MyTime=`date -d "$YEAR-$MONTH-$DAY +$PlusDay days" +%Y-%m-%dT%H:%M:%SZ`

  TimeStart="time_start=$MyTime"

  TimeEnd="time_end=$MyTime"

 

  OutFile=$MODEL"_"$EXPT"_`echo $MyTime | cut -d 'T' -f 1`T00Z.nc"

 

  URL="$NCSS/$MODEL/$EXPT?$VARS&$SPATIAL&$NORTH&$SOUTH&$EAST&$WEST&$TimeStart& $TimeEnd"

 

  if [ -s $OutFile ]; then

          echo "[warning] File $OutFile exists (skipping)"

  else

          wget -O $OutFile "$URL"

  fi

done

-----------

--