Jython and NetCDF Java

[skb]

As far as I can tell, my task is to create NcML versions of the
metadata parts of my present NetCDF products, as well as to fortify
the original metadata sections with additional information to make the
files more widely accessible and usable. I've just downloaded and
tried the <STRONG>toolsUI</STRONG> GUI for the NetCDF Java libraries.
As the interface is fairly well designed, it didn't take long to point-
and-click my way to a <TT>test.ncml</TT> equivalent of a <TT>test.nc</
TT> file. It immediately occurred to me, however, that I really don't
want to be firing up a GUI all the time to create NcML files. Going
over the alternatives, I immediately rejected learning how to program
in Java in order to access the NetCDF Java libraries. So this old
horse looked for an easier trick, and possibly found one. There's a
Java version of the Python scripting language that's called Jython
which allows you to use Java libraries from within Python scripts.
I'm already familiar with the elegance and ease of Python, so I'm
going to spend a bit of time figuring out how to combined Jython and
the NetCDF Java libraries in a useful way that will let me
automatically created NcML files.

My attempts at this are being documented on the fly on my <A
HREF="http://stommel.tamu.edu/~baum/ncml.html">NcML</A> summary page.

BTW, If anyone's found another alternative for the automatic creation
of NcML files I'm all ears.

[Richard Signell]

Steve,

As you mention, we have been using NcML to correct and/or provide
metadata for NetCDF (or GRIB) files so that they can be accessed via a
THREDDS Data Server as CF-compliant data via Web Services like
OpenDAP.

Since we typically only have to create a few lines of NcML one time to
serve up a certain type of files (e.g. SCCOOS ROMS results, NERACOOS
POM results), we've just been generating these files manually (e.g. in
an XML editor).

I do have a few scripts in Matlab that do things like interrogate a
collection of NetCDF files on a remote web site and generate NcML that
aggregates that data, but I as yet haven't felt the strong need for a
"NcML toolbox".

Do you really have a strong need for automated NcML production?
If so, I'd definitely be interested in hearing more about what the issues are.

-Rich

--
Dr. Richard P. Signell (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598

[RobC]

This is very cool. Following your footsteps. It is unfortunate that
jython is a separate installation from your regular python
installation. The stable release is 2.2.1 while the regular python
version is 2.6 and better now :( Looks like it might be a pain to
bring matplotlib and other useful modules into jython...

The other frustrating thing is doing discovery through the ucar.* tree
to find all the needed components. This might help.
http://www.unidata.ucar.edu/software/netcdf-java/v2.2.20/javadocAll/overview-summary.html

This looks like the part you need...
http://www.unidata.ucar.edu/software/netcdf-java/v2.2.20/javadocAll/ucar/nc2/ncml/NcMLWriter.html

$ jython
*sys-package-mgr*: processing new jar, '/usr/local/lib/jython/
netcdfAll-4.0.jar'
Jython 2.2.1 on java1.6.0_07
Type "copyright", "credits" or "license" for more information.

>>> import ucar
>>> dir(ucar)
['__name__', 'atd', 'grib', 'grid', 'ma2', 'nc2', 'unidata', 'units']

>>> import thredds
>>> dir(thredds)
['__name__', 'catalog', 'cataloggen', 'crawlabledataset',
'filesystem', 'util', 'wcs']

Maybe slightly off topic...

Lastly, I wonder if there is a CF tester inside? Unidata is working
on a libcf utility that has that ability. http://www.unidata.ucar.edu/software/libcf/.
Might be possible to roll your own python module around it to then do
autodetect of CF elements and generate NcML.

A boilerplate python module wrapper around Unidata's UDUNITS2 package
can be found here:
https://db.aoos.org/wiki/index.php/AOOS_Python#UDUNITS2

Of course now that made a bit easier via jython and ucar.units... but
the wrapper is more useful in my bigger pile of tools in python.
Sigh...

[RobC]

Related question, if NcML is added or modified, do you have to restart
the TDS?

Dynamic creation of NcML could be useful say if a data provider wanted
to submit data (in a somewhat automated fashion) to a data aggregation
center (DAC) that is new to the DAC's TDS server.

[RobC]

A boilerplate NcML writer via jython. You don't have to stop at
NetCDF. This library reads much more. Thanks again for pointing this
tool out. Additional documentation will be generated in the IOOS
Technical google group (http://groups.google.com/group/ioos_tech).

#!/usr/bin/env jython

import ucar

wrf = ucar.nc2.dataset.NetcdfDataset.openDataset('wrfout-
d01-20090404_00z.nc')

# Open a file to write the NcML
f = open('wrfout-d01-20090404_00z.xml', 'w')

wrf.writeNcML(f,None)

---

$ cat wrfout-d01-20090404_00z.xml
<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
location="file:wrfout-d01-20090404_00z.nc">
<dimension name="time" length="49" />
<dimension name="nob" length="19" />
<dimension name="ydim" length="78" />
<dimension name="xdim" length="141" />
<attribute name="title" value="4 km Prince William Sound Regional
WRF Model Output from AEFF" />
....
<variable name="TMP_2_HTGL" shape="time ydim xdim" type="float">
<attribute name="long_name"
value="air_temperature_at_2m_above_ground" />
<attribute name="standard_name"
value="air_temperature_at_2m_above_ground" />
<attribute name="units" value="degC" />
</variable>
....
<variable name="time" shape="time" type="double">
<attribute name="long_name" value="forecast_reference_time" />
<attribute name="standard_name" value="forecast_reference_time" />
<attribute name="units" value="days since 0001-01-01 00:00:0.0" />
<attribute name="timezone" value="GMT" />
<attribute name="_CoordinateAxisType" value="Time" />
</variable>
<variable name="lat" shape="ydim xdim" type="float">
<attribute name="long_name" value="latitude" />
<attribute name="standard_name" value="latitude" />
<attribute name="units" value="degrees_north" />
<attribute name="_CoordinateAxisType" value="Lat" />
</variable>
<variable name="lon" shape="ydim xdim" type="float">
<attribute name="long_name" value="longitude" />
<attribute name="standard_name" value="longitude" />
<attribute name="units" value="degrees_east" />
<attribute name="_CoordinateAxisType" value="Lon" />
</variable>
<variable name="PRES_SFC" shape="time ydim xdim" type="float">
<attribute name="long_name" value="surface_air_pressure" />
<attribute name="standard_name" value="surface_air_pressure" />
<attribute name="units" value="mbar" />
<attribute name="_CoordinateAxisType" value="Pressure" />
</variable>
</netcdf>

[Richard Signell]

Rob,

> Related question, if NcML is added or modified, do you have to restart
> the TDS?

Yes, if static TDS catalogs change, then the TDS must be reloaded.

>
> Dynamic creation of NcML could be useful say if a data provider wanted
> to submit data (in a somewhat automated fashion) to a data aggregation
> center (DAC) that is new to the DAC's TDS server.

The TDS also creates dynamic catalogs when you use features like
<datasetScan> or <datasetFmrc>. Then things change in these virtual
datasets you don't have to reload. You could add additional NcML
functionality to these functions or create new ones, but this would of
course be done in the TDS source code.

-Rich


If you just modify

[Richard Signell]

Rob & Steve,

I just want to make sure I understand what you guys are thinking here.

Jython is java that acts like python, right? Sort of a software
emulation of python in Java?

So the idea with a Jython interface to NetCDF-Java is that you could
can utilize NetCDF-Java functionality using the python language
instead of writing programs in Java or Matlab, right?

So is this a scenario in which you would use such a tool?

- You regularly want to access NCOM global data from
<http://edac-dap.northerngulfinstitute.org/dods-bin/nph-dods/NCOM/region1>
as an aggregation instead of a "pile of files".

- You write a python script using Jython that accesses the OpenDAP
URLS, extracts the time information, and then creates the NcML that
aggregates this data using only the time values that constitute a
continuous time series of data.


-Rich

[RobC]

Yup. It is just adding another tool to the toolbox beyond using
Matlab and writing java applications. NcML and more...

From the NcML side of things, at least two purposes that come to mind
are (1) initial creation of NcML for datasets we want to toss at the
TDS; (2) future use for adaptive modeling. Adaptive modeling could
include hi-res runs of ocean/atmospheric models for oil spill/search
and rescue response situations where the grid isn't guaranteed to be
the same location. To do this, it looks like part of the procedure is
a restart of the TDS to add new grids. Once a grid is in place,
adding new model runs to existing grids does not require a restart.

---- Text from the Jython site about regular python modues ----
4.1 What parts of the Python library are supported?

The good news is that Jython now supports a large majority of the
standard Python library. The bad news is that this has moved so
rapidly, it's hard to keep the documentation up to date.

Built-in modules (e.g. those that are written in C for CPython) are a
different story. These would have to be ported to Java, or implemented
with a JNI bridge in order to be used by Jython. Some built-in modules
have been ported to JPython, most notably cStringIO, cPickle, struct,
and binascii. It is unlikely that JNI modules will be included in
Jython proper though.

If you want to use a standard Python module, just try importing it. If
that works, you're probably all set. You can also do a dir() on the
modules to check the list of functions it implements.

> >http://www.unidata.ucar.edu/software/netcdf-java/v2.2.20/javadocAll/o...

>
> > This looks like the part you need...

> >  http://www.unidata.ucar.edu/software/netcdf-java/v2.2.20/javadocAll/u...