Availanility of GLBy0.08_930_FMRC outputs

[Sarah Barbier]

Dear HYCOM Group ,

I have scheduled in my crontab a task to download GLBy0.08_930_FMRC water_u and water_v forecast outputs through the ncss service (wget).

To assign an hour of execution to my task, I have checked on ftp.hycom.org the deposit hour of the hycom_glby_930_YYYYMMDD12_txxx_uv3z.nc files which is approximately between 21:00UTC and 22:00UTC.

So I decided to schedule the execution at 22:30 UTC .

But when my task run the download, my output file is empty.

Therefore, is there a lag between the deposit hour of the forecasts on ftp.hycom.org and the download from the ncss service (wget) ?

Then, at what time do you suggest me to run my job ?

Thank you in advance for your answers.

Best Regards,

BARBIER Sarah

Meteo France Engineer

Martinique, France

[Michael McDonald]

Yes. The files getting written to disk do not trigger a THREDDS
catalog/FMRC update until all servers/workers have been restarted.
This can take a few hours to complete after data updates are written
as we do a rolling pool upgrade and try to not interrupt ongoing
OPENDAP operations. I would add a 2hr delay to your cron job.

> --
> --
> You received this message because you are a member of HYCOM.org
> To ask a question, send an email to fo...@hycom.org
>


--
Michael McDonald
HYCOM.org Administrator

[Sarah Barbier]

Micheal,

Thank you for your answer.

After I have posted my question, I read the information in the conversations header that THREDDS server are restarting between 2.00 and 3.00am .

So I re-scheduled my job yesterday at 4.00 am the next day (this morning) (to download the HYCOM run of yesterday 120000UTC then ) but my file is still empty.

I fear that even at 00.30 am I do not have the hycom run.

Than you for your return.

BARBIER Sarah

Meteo France Engineer

[Michael McDonald]

Please share the exact/specific URL that you are attempting to
download from and the methods (for reproducibility) used to properly
diagnose and debug the issue you are having.

[Sarah Barbier]

Please find below the cron task my bash script that is running from this cron task. I was unable to attach the bash script itself.

Everayday at 4.00 am, the code downloads, given a spatial domain, a 7-day water_u water_v hycom forecast of the previous run (day-1)

00 04 * * * /home/dirag_app/SARGASSES/bin/get_surface_current_hycom_data.sh > /dev/null 2>&1

#!/bin/bash
#__Author__                         : ECMPF/DEV
# __Date__                          : 08/02/2022
# __Mise_a_jour__                   : 07/04/2022
# __Version__                       : 1.1
                

# Données Modele
DATAHOME='/data/data_SARGASSES/hycom/'
DATASET='GLBy0.08'
EXPT='93.0'
EXP='930'
FORMAT='netcdf4' #ou netcdf4

# Domaine spatiale
lat_north=25
lat_south=-5
lon_east=350
lon_west=280
depth='0.0'
verticale_step=$( echo $depth | wc -l)

# Domaine temporelle
year_start=`date +%Y`
month_start=`date +%m`
day_start=`date --date '1 day ago' +%d`
hour='12:00:00Z'

year_end=`date --date '6 day' +%Y`
month_end=`date --date '6 day' +%m`
day_end=`date --date '6 day' +%d`

start_date=${year_start}${month_start}${day_start}
end_date=${year_end}${month_end}${day_end}

#Le modele fournit des sorties toutes les 3h mais on souhaite toute les 24h
time_step=8

FORECAST_RUN=${year_start}-${month_start}-${day_start}T${hour}
END_RUN=${year_end}-${month_end}-${day_end}T


echo ' '
echo ' '
echo 'Parametre de requete '
echo '----------------------------'
echo 'lat_north      = '$lat_north
echo 'lat_sud        = '$lat_south
echo 'lon_east       = '$lon_east
echo 'lon_west       = '$lon_west
echo 'depth          = '$depth
echo 'verticale_step = '$verticale_step
echo 'year_start     = '$year_start
echo 'month_start    = '$month_start
echo 'day_start      = '$day_start
echo 'hour           = '$hour
echo 'year_end       = '$year_end
echo 'month_end      = '$month_end
echo 'day_end        = '$end_date
echo 'time_step      = '$time_step
echo 'FORECAST_RUN   = '$FORECAST_RUN
echo 'END_RUN        = '$END_RUN
echo ' '
echo ' '



REQUETE="https://ncss.hycom.org/thredds/ncss/${DATASET}/expt_${EXPT}/FMRC/runs/${DATASET}_${EXP}_FMRC_RUN_${FORECAST_RUN}?var=water_u&var=water_v&north=${lat_north}&west=${lon_west}&east=${lon_east}&south=${lat_south}&disableProjSubset=on&horizStride=${verticale_step}&time_start=${year_start}-${month_start}-${day_start}T12%3A00%3A00Z&time_end=${END_RUN}12%3A00%3A00Z&timeStride=${time_step}&vertCoord=${depth}&addLatLon=true&accept=${FORMAT}"

cd ${DATAHOME}
wget -O hycom_${DATASET}_${EXP}_${start_date}120000_${end_date}120000.nc $REQUETE
echo ' '

exit

[Michael McDonald]

Thanks for sharing this code. Please also share your full terminal
output when you run this command manually from the command line.