Anyone using the Unidata IDD to get NCEP and OBS data pushed to them?

[rsignell]

Gang,

Is anyone doing ocean forecasts using the Unidata IDD to have NOAA and
NAVY model results and realtime data pushed to them?

At the Ocean Observing Initiative Cyberinfrastructure Kick-Off meeting
earlier this month, I was talking to Ben Domenico from Unidata about
how academics in my experience do regional modeling: obtain the data
needed for forcing, boundary conditions and assimilation using a
collection of Unix shell, perl or python scripts that get data via ftp
or via OpenDAP.

Nothing wrong with this for hindcasting, but Ben was surprised to hear
that we typically use the same approach for forecast modeling. Why
don't we use the Unidata Internet Data Distribution (IDD) software, he
was wondering, which pushes down the latest NCEP and other model
products as well as realtime obs as soon as it becomes available. As
Ben described it, you subscribe to what you want, can add your own obs
to the stream, and others can pick it up from you -- a peer-to-peer
system.

It looks like a bunch of academic groups are using the IDD for met
modeling, but I've never heard of it being used for ocean modeling.
Is anyone out there using it?

For more info:
http://www.unidata.ucar.edu/software/idd/

-Rich

[Ruoying He]

Well, first time hearing about this. will definitely check it out. As
always, Thanks much, Rich!

--
Ruoying He Coastal Ocean Observing and Modeling Group
Tel: 919-513-0249 Marine, Earth & Atmospheric Sciences
Fax: 919-515-7802 North Carolina State University
www4.ncsu.edu/~rhe Raleigh, NC 27695, U.S.A

[Steve Cousins]

Hi Rich,

Is this the same as LDM? Doing a quick search, I see that IDD uses LDM.
We have an LDM feed set up to get some data for a partner in China who
doesn't have a consistent network link. We cache the data for 30 days
for him. I wasn't aware of NCEP using this though. It's worth knowing
about (or being reminded) but OpenDAP access works pretty well for us.

Steve

[rsignell]

Steve,

Yes, IDD uses the LDM. I think the main advantage over OPeNDAP is
that you get the data as soon as it becomes available. You don't have
to keep polling the OpenDAP site to see if the data is there yet so
you can run your forecast. But I don't have any experience with how
easy it is to set up and use. I was hoping for someone to chime in
and say that they use it to download the NAM forecasts and data for
assimilation or something and give us a little summary of how well it
works for them...

-Rich

[Rob Cermak]

Indeed we are using the LDM and are hooked up to the IDD. Not
everything is on the IDD, however a lot of OBS data streams over the
IDD. Of particular interest might be the surface METARs, buoy data
and water level/hydrometeorological data (SHEF format). In most
cases, the data is RAW and yet to go through QA/QC. When NDBC was
taken out by the hurricane (with their website), there was a minor
interruption to the buoy data on the IDD.

Due to the density of model data, only some of what is available from
the NCEP FTP(now HTTP) area is sent over the IDD via the CONDUIT or
HRS feeds. Usually, the more modern grids are kept and the older ones
retired. Being on the fringe of Internet2 in Alaska, bandwidth is not
the greatest and we still get drop outs during peak transmission times
(and it could be a I/O bandwidth issue on the poor server too). We
have to check to see if all the fields come through and if necessary
grab one or more of the fields/files from the NCEP server or
motherlode.

If you run a LDM, then you can also seek permission to get a LDM feed
of QA/QC'd surface and other data from the MADIS group.

See: http://madis.noaa.gov/

Cheers,
Rob

[Richard Signell]

Rob,

Are there tools from Unidata that allow you to turn the stream of data
from the IDD/LDM into files that can be served via the THREDDS Data
Server, making your own mini-Motherlode?
Or do you have your own custom conversion programs and scripts that
set things up for the models you run?

And overall, do you think the IDD/LDM is something that other IOOS
modeling groups should be looking into, or old push technology whose
time is nearly done, replaced by systems that pull data and model
results via OPeNDAP?

Thanks,
Rich

--
Dr. Richard P. Signell (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

[Rob Cermak]

Either with the LDM or the THREDDS data server there is a pqact.conf
(product queue action) configuration file that gets one started on
creating their own mini-motherlode. The configuration file makes
heavy use of a Unidata tool group called decoders. See:
http://www.unidata.ucar.edu/software/decoders/

[rsignell]

Folks,

I just wanted to report back and say that with Unidata's help, we got
the IDD/LDM feed set up at WHOI to receive the NAM 12 km (CONUS) and
GFS 0.5 degree (GLOBAL) met model products, and it's been working
great so far. The data just goes right onto our THREDDS server, so we
can use the same OPeNDAP scripts we were using before, just pointing
to our own THREDDS server rather than NOMADS. (Well, we also had to
change the scripts to reflect the fact that the variable names are
different -- they must do some variable remapping between the GRIB
variables and what they present on NOMADS.) John Warner's
experimental COAWST ROMS-SWAN coupled forecast model for the East
Coast hasn't dropped any forecasts since we set this up!

-Rich

On Sep 24, 11:05 am, Rob Cermak <rob.cer...@gmail.com> wrote:
> Either with the LDM or the THREDDS data server there is a pqact.conf
> (product queue action) configuration file that gets one started on
> creating their own mini-motherlode.  The configuration file makes
> heavy use of a Unidata tool group called decoders.  See:http://www.unidata.ucar.edu/software/decoders/
>

> On Thu, Sep 24, 2009 at 2:00 AM, Richard Signell <rsign...@gmail.com> wrote:
>
> > Rob,
>
> > Are there tools from Unidata that allow you to turn the stream of data
> > from the IDD/LDM into files that can be served via the THREDDS Data
> > Server, making your own mini-Motherlode?
> > Or do you have your own custom conversion programs and scripts that
> > set things up for the models you run?
>
> > And overall, do you think the IDD/LDM is something that other IOOS
> > modeling groups should be looking into, or old push technology whose
> > time is nearly done,  replaced by systems that pull data and model
> > results via OPeNDAP?
>
> > Thanks,
> > Rich
>

> > On Thu, Sep 24, 2009 at 1:15 AM, Rob Cermak <rob.cer...@gmail.com> wrote:
>
> >> Indeed we are using the LDM and are hooked up to the IDD.  Not
> >> everything is on the IDD, however a lot of OBS data streams over the
> >> IDD.  Of particular interest might be the surface METARs, buoy data
> >> and water level/hydrometeorological data (SHEF format).  In most
> >> cases, the data is RAW and yet to go through QA/QC.   When NDBC was
> >> taken out by the hurricane (with their website), there was a minor
> >> interruption to the buoy data on the IDD.
>
> >> Due to the density of model data, only some of what is available from
> >> the NCEP FTP(now HTTP) area is sent over the IDD via the CONDUIT or
> >> HRS feeds.  Usually, the more modern grids are kept and the older ones
> >> retired.  Being on the fringe of Internet2 in Alaska, bandwidth is not
> >> the greatest and we still get drop outs during peak transmission times
> >> (and it could be a I/O bandwidth issue on the poor server too).  We
> >> have to check to see if all the fields come through and if necessary
> >> grab one or more of the fields/files from the NCEP server or
> >> motherlode.
>
> >> If you run a LDM, then you can also seek permission to get a LDM feed
> >> of QA/QC'd surface and other data from the MADIS group.
>
> >> See:http://madis.noaa.gov/
>
> >> Cheers,
> >> Rob
>