Trouble with FMRC_best access

115 views
Skip to first unread message

p.mac...@gmail.com

unread,
Nov 24, 2019, 1:06:38 PM11/24/19
to HYCOM.org Forum
Recently I have been having trouble with an automated job I run to get an extraction of the daily HYCOM forecast using FMRC.  It has worked nearly perfectly for about a year, but lately I have been getting errors that indicate the server is not responding.  This happened again this morning.  I am rerunning right now by hand and it appears to be working correctly, but of course I would like it to run automatically. I run the job by cron at 12:10 AM PST every morning.  Any guidance you have is appreciated.  For reference, below is a standalone python program that I use to test the way I do the FMRC request.

Thanks,

Parker

""""
Code to test getting hycom files using the new FMRC_best file.
"""

import os
import netCDF4 as nc

from urllib.request import urlretrieve
from urllib.error import URLError
from socket import timeout
import time
from datetime import datetime, timedelta

testing = True

# specify output file
fn_out = 'test.nc'
# get rid of the old version, if it exists
try:
    os.remove(fn_out)
except OSError:
    pass # assume error was because the file did not exist

# specify time limits
# get today's  date string in LO format
dstr00 = datetime.now().strftime('%Y.%m.%d')
print('Working on ' + dstr00)
# get time limits for forecast
dt00 = datetime.strptime(dstr00, '%Y.%m.%d')
dt0 = dt00 - timedelta(days=2)
if testing == True:
    dt1 = dt00 - timedelta(days=1)
else:
    dt1 = dt00 + timedelta(days=5)
# put them in ncss format
dstr0 = dt0.strftime('%Y-%m-%d-T00:00:00Z')
dstr1 = dt1.strftime('%Y-%m-%d-T00:00:00Z')
print('- dt0 = ' + dstr0)
print('- dt1 = ' + dstr1)

# specify spatial limits
#aa = [-129, -121, 39, 51]
# # find indices of a sub region
# aa = hfun.get_extraction_limits()
north = 51
south = 39
west = 231 #-129 + 360
east = 238 #-122 + 360

if testing == True:
    var_list = 'surf_el'
    #var_list = 'surf_el,salinity'
else:
    var_list = 'surf_el,water_temp,salinity,water_u,water_v'

# create the request url
    '?var='+var_list +
    '&north='+str(north)+'&south='+str(south)+'&west='+str(west)+'&east='+str(east) +
    '&time_start='+dstr0+'&time_end='+dstr1+'&timeStride=8' +
    '&addLatLon=true&accept=netcdf4')

# get the data and save as a netcdf file
counter = 1
got_file = False
while (counter <= 3) and (got_file == False):
    print('Attempting to get data, counter = ' + str(counter))
    tt0 = time.time()
    try:
        (a,b) = urlretrieve(url,fn_out)
        # a is the output file name
        # b is a message you can see with b.as_string()
    except URLError as e:
        if hasattr(e, 'reason'):
            print(' *We failed to reach a server.')
            print(' -Reason: ', e.reason)
        elif hasattr(e, 'code'):
            print(' *The server couldn\'t fulfill the request.')
            print(' -Error code: ', e.code)
    except timeout:
        print(' *Socket timed out')
    else:
        got_file = True
        print(' Worked fine')
    print(' -took %0.1f seconds' % (time.time() - tt0))
    counter += 1

# check results
ds = nc.Dataset('test.nc')
print('\nVariables:')
for vn in ds.variables:
    print('- '+vn)
print('Times:')
htime = ds['time'][:]
# get time info for the forecast
t = ds['time'][:]
tu = ds['time'].units
# e.g. 'hours since 2018-11-20 12:00:00.000 UTC'
# Warning: Brittle code below!
ymd = tu.split()[2]
hmss = tu.split()[3]
hms = hmss.split('.')[0]
hycom_dt0 = datetime.strptime(ymd + ' ' + hms, '%Y-%m-%d %H:%M:%S')
# make a list of datetimes that are in the forecast
hycom_dt_list = [] # datetimes
hycom_iit_list = [] # indices
iit = 0
for th in t: # assume t is sorted
    this_dt = hycom_dt0 + timedelta(th/24)
    hycom_dt_list.append(this_dt)
    print('- '+str(this_dt))
    hycom_iit_list.append(iit)
    iit += 1
ds.close()

Michael McDonald

unread,
Nov 26, 2019, 1:05:45 PM11/26/19
to Parker MacCready, HYCOM.org Forum
Parker,
We briefly discussed this similar issue in another thread here.

this URL you are using will need to be removed (in the near future) as the dataset name "FMRC" conflicts with other FMRC model runs (also named FMRC), so the automated cron updates have not been running in the background (this is an internal component of THREDDS). We are also testing out the addition of StickySession in our Apache proxy load balancer that might help with queries *sticking* to one server per request, rather than getting bounced around to other servers in the pool with (potentially) different metadata.

e.g.,
Please browse here to get the (hopefully) better FMRC OPENDAP URL

There is a newer link "GLBy0.08_930_FMRC/" under the "FMRC (Forecast Model Run Collection)" dataset heading, which is a globally unique FMRC that should now be updating in the background properly. The legacy "FMRC/", "FMRC_ice/", and "FMRC_sur" will need to be replaced by the uniquely named "GLBy0.08_930_FMRC/", "GLBy0.08_930_FMRC_ice/", and "GLBy0.08_930_FMRC_sur/".

e.g., see the new NCSS URL from here,

the new base URL for NCSS scripted queries becomes,



after trial and error, we discovered that the path "GLBy0.08/expt_93.0/FMRC" and the name "GLBy0.08_930_FMRC" both needed to be globally unique, which they were not in the legacy URL you were using.







Recently I have been having trouble with an automated job I run to get an extraction of the daily HYCOM forecast using FMRC.  It has worked nearly perfectly for about a year, but lately I have been getting errors that indicate the server is not responding.  This happened again this morning.  I am rerunning right now by hand and it appears to be working correctly, but of course I would like it to run automatically. I run the job by cron at 12:10 AM PST every morning.  Any guidance you have is appreciated.  For reference, below is a standalone python program that I use to test the way I do the FMRC request.

Thanks,

""""



--
Michael McDonald
HYCOM.org Administrator

Ivica Janeković

unread,
Nov 28, 2019, 9:50:49 PM11/28/19
to HYCOM.org Forum
Dear all,
is there some delay with the model stream?
As I see from the TDS the last model run was 2019-11-22, or getting the last time step via ncks:

ncks -O -v time -d time,-1 https://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0/FMRC/GLBy0.08_930_FMRC_best.ncd -o /tmp/hycom_test.nc

gives date of 2019-12-02 12

Thanks for taking care of it!

Cheers
Ivica

Michael McDonald

unread,
Nov 29, 2019, 10:00:06 AM11/29/19
to Ivica Janeković, HYCOM.org Forum
No. The model runs are on track. Please reclarify your issue or double-check your code. 



--
--
You received this message because you are a member of HYCOM.org
To ask a question, send an email to fo...@hycom.org

Ivica Janeković

unread,
Nov 29, 2019, 8:59:16 PM11/29/19
to HYCOM.org Forum, ivic...@gmail.com
On the link you just sent


I can see that the last model output day is 2019-11-28 and the tau time is from 23rd which seems not correct (too old).

not sure if I am doing something wrong?

the same holds for other links with best time series:

or

last model output is 2019-12-03 @ 12
and should be 2019-12-05 @12 or even 6th


Cheers
I

Michael McDonald

unread,
Nov 29, 2019, 9:15:27 PM11/29/19
to Ivica Janeković, HYCOM.org Forum
Sorry. We (Florida State University, hosting for hycom.org) and the NAVY DSRC that runs the GOFS ocean model is on holiday for the U.S. Thanksgiving break. There is something that has been causing issues with the data transfer that will need to get sorted out after the break concludes on Monday. 

Only the 24hr forecast bundle for today's run (2019-11-28) was transferred and unpacked online. 

As of this email, this is this latest run (2019-11-28)



Martin Santivañez

unread,
Dec 1, 2019, 7:49:04 PM12/1/19
to HYCOM.org Forum
Dear all,
I'm trying to do the same as Parker but in R and doesn't work. Here's the code:

############################
año='2019'
mes='11'
día='30'
fecha2=paste(año,mes,día,sep='-')

fecha2,
"&addLatLon=true&accept=netcdf4",sep='')

savefile=tempfile(, fileext = ".nc4")
require(curl)
curl_download(file, savefile)

######################

I obtain the next error string:

###############

Error in curl_download(file, savefile) : HTTP error 500.

#################

Could you help me to solve this, please?

Best regards,
Martin

Michael McDonald

unread,
Dec 6, 2019, 2:50:48 PM12/6/19
to Martin Santivañez, HYCOM.org Forum
Has this issue resolved itself? Just after our Nexsan upgrade, there was a period of about 5-6 days where operations were taking *twice* as long as they usually took, but now the storm has subsided and file operations are performing normally again. 



--
--
You received this message because you are a member of HYCOM.org
To ask a question, send an email to fo...@hycom.org

Parker MacCready

unread,
Dec 6, 2019, 6:03:36 PM12/6/19
to Michael McDonald, Martin Santivañez, HYCOM.org Forum
Michael,

My FMRC extractions are working fine.  My old version was working fine, but I just updated the URL per your instructions, and it worked the same.

Thanks for your help, as always.

Parker

---
To unsubscribe from this topic, visit https://groups.google.com/a/hycom.org/d/topic/forum/cHsDJjWB9Dk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to forum+un...@hycom.org.


--
Parker MacCready, Professor
University of Washington, School of Oceanography
Email: pm...@uw.edu
URL: faculty.washington.edu/pmacc
Reply all
Reply to author
Forward
0 new messages