Reposting a problem/solution from the ioos tech/Unidata THREDDS list: Forecast Model Collections

[RobC]

AOOS did bring its server online and with the help of Unidata's John
Caron got one of those forecast model run collections going. If
anyone hits a similar pitfall, this might be helpful. Problem and
solution were posted to the Unidata THREDDS list. We also found a
couple curious things with TDS that we will continue to forward to
Unidata.

Our server can be found here: http://137.229.40.88/thredds/catalog.html.
It is by no means in operational shape, if you plan to hook into it
operationally, please contact me first so I can let you know when
changes are going to happen. There is a LOT of work yet to be done.
[The collection is not functioning as we are currently having a
separate RAID issue on which the forecasts are contained, bummer.]

There is a complied list of servers that can be obtained from this
site at present:
http://coast-enviro.er.usgs.gov/thredds/ioos_catalog_top.html (the
catalog file is also available in the Files area of this group)

So, for the tech side of this message, to get a forecast collection
going, there is a special dimension called 'runtime' that knits
together all the model forecasts. I thought initially that I had to
tell it to use the time dimension (which was incorrect). When the TDS
aggregates the dataset, it pulls together all the runs into a new
'runtime' dimension. So, essentially time becomes 2D. The runtime
(initial time for the model runs) and time (runs along each individual
forecast).

<datasetFmrc name="Collections" path="fmrc/ROMS">
<metadata inherited="true">
<dataFormat>NetCDF</dataFormat>
<documentation type="summary">JPL ROMS</documentation>
<serviceName>all_services</serviceName>
</metadata>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/
ncml-2.2">
<aggregation dimName="runtime" type="forecastModelRunCollection"
recheckEvery="120 min">
<scan location="/dp2/space/data/forecasts/ROMS/PWS/2009/raw/"
suffix=".nc" dateFormatMark="pws_fcst_#yyyyMMddHH" />
</aggregation>
<attribute name="Conventions" type="String" value="CF-1.0"/>
</netcdf>
</datasetFmrc>

Still have a long way to ramp up on this.

If an aggregation fails for whatever reason, the log file to look at
is:
$TOMCAT_HOME/content/thredds/logs/models.log

You will see stuff like this:
[31/Jul/2009:22:01:10] ERROR thredds.catalog.InvDatasetFmrc : Error
making catalog for fmrc/ROMS
java.lang.IllegalArgumentException: no grids

[James T. Potemra]

Hi Rob (and others):

I'll just add two issues that came up with our TDS. First, I've found that
the forecast model best time series works really well if, for some reason,
you don't get output for a certain output period, but not so well if you
get
a short or otherwise corrupt file. So, for example, we get model output
each day. If we miss a day because the model forcing fields don't come in,
the TDS "best time series" handles the missing day. However, if for some
reason we get output for the day, but the run stops mid-way (so we get
half of
a day's output), the TDS does not work correctly. The symptom of this
is if you do an ascii dump on forecast time, the very first record goes back
to the "time since" number. Again, for an example, our forecast times
should be like -72 -64 -56....., but when there are problems the first time
is -8760 -64 -56... I hope this makes sense. Our solution has just been to
check the file size, and if inconsistent, we save the output to a separate
directory (and it's not picked up by TDS).

Second, I've not been able to overcome the problem of model fields on
different grids. This is a client-side problem, but I was thinking there
might be a way to address it on the server side. In our ocean model, the
velocity components (u,v,w) and scalar fields (t, s, sea level) are on
different
co-ordinates. TDS serves it, and Matlab can read it without problem.
GrADS however cannot read this data set. I was hoping to fake GrADS
by mapping different coordinates onto lat/lon/altitude.

Anyway, just two issues that might be of common interest.

Jim

[David Stuebe]

Hi Jim

From your email it sounds as though you have found a clever solution
for dealing with staggered grid model data.
If this is what you mean (See link as an example:
http://www.gfdl.noaa.gov/~vb/gridstd/gridstdse2.html#x4-130002.6) let
us start a separate thread for this discussion as that would be
extremely helpful across a range of forums.

David