Reading gridded MDV files

[greg.b...@noaa.gov]

Hello all,

I'm posting to see if I can get some assistance in reading some gridded MDV files I found from the Mesoscale group at UCAR:

http://www2.mmm.ucar.edu/imagearchive/WSI/mdv/

After playing around with PyART for a bit, I was able to get the file read without PyART crashing or anything by using pyart.io.read_mdv_grid().  The grid object that gets returned has five attributes which are shown below in my attached output.  The attributes axes and fields contain data (as seen below), but the field attribute does not have any keys associated with it.  After adding print statements to a couple of places in the code a bit with this, I found that there should be "Reflectivity" key, but it's not appearing because filemetadata.get_field_name() is returning None (see below) and so the loop below never actually loads any data.  I don't know much about the MDV format, so I'm posting this in hope that maybe someone else has read these files successfully or has some information that can help me get them read.  

I've attached a sample MDV file to this post (230000.mdv).  I've also included information about the latest commit I'm using (I cloned from Github).  I'm also on OS X Yosemite.

Thanks!

Greg Blumberg

OU/CIMMS

Output from read_mdv_grid():

<pyart.core.grid.Grid object at 0x10a2b9190>

In [1]: data.

data.add_field  data.axes       data.fields     data.metadata   data.write      

In [1]: data.fields

Out[1]: {}

In [2]: data.metadata

Out[2]: 

{'Conventions': 'CF/Radial instrument_parameters',

 'comment': '',

 'history': '',

 'institution': '',

 'instrument_name': '20030508.2330.MASTER15',

 'references': '',

 'source': '',

 'title': '',

 'version': '1.3'}

In [3]: data.axes

Out[3]: 

{'alt': {'data': array([ 500.]),

  'long_name': 'Altitude at grid origin',

  'standard_name': 'altitude',

  'units': 'm'},

 'lat': {'data': array([ 52.99998856]),

  'long_name': 'Latitude at grid origin',

  'standard_name': 'latitude',

  'units': 'degree_N',

  'valid_max': 90.0,

  'valid_min': -90.0},

 'lon': {'data': array([-95.00001526]),

  'long_name': 'Longitude at grid origin',

  'standard_name': 'longitude',

  'units': 'degree_E',

  'valid_max': 180.0,

  'valid_min': -180.0},

 'time': {'calendar': 'gregorian',

  'data': array([ 450.]),

  'long_name': 'Time in seconds since volume start',

  'standard_name': 'time',

  'units': 'seconds since 2003-05-08T23:22:30Z'},

 'time_end': {'calendar': 'gregorian',

  'data': array([ 900.]),

  'long_name': 'Time in seconds of volume end',

  'standard_name': 'time',

  'units': 'seconds since 2003-05-08T23:22:30Z'},

 'time_start': {'calendar': 'gregorian',

  'data': array([ 0.]),

  'long_name': 'Time in seconds of volume start',

  'standard_name': 'time',

  'units': 'seconds since 2003-05-08T23:22:30Z'},

 'x_disp': {'axis': 'X',

  'data': array([-129990.44799805, -129971.32753767, -129952.20707729, ...,

          -60047.80394211,  -60028.68348174,  -60009.56302136]),

  'long_name': 'X-coordinate in Cartesian system',

  'units': 'degree_E'},

 'y_disp': {'axis': 'Y',

  'data': array([ 20008.99124146,  20026.95530094,  20044.91936043, ...,

          52955.07634059,  52973.04040007,  52991.00445956]),

  'long_name': 'Y-coordinate in Cartesian system',

  'units': 'degree_N'},

 'z_disp': {'axis': 'Z',

  'data': array([ 500.]),

  'long_name': 'Z-coordinate in Cartesian system',

  'positive': 'up',

  'units': 'm'}}

In [4]: 

From mdv_grid.py

395     # fields

396     fields = {}

397     mdv_fields = mdv._make_fields_list()

398     for mdv_field in set(mdv_fields):

399         field_name = filemetadata.get_field_name(mdv_field)

400         if field_name is None:

401             continue

Commit information:

commit 6021043383a02761c8ed2d40a86c71839ef4cdb8

Merge: a65dfa8 0e76c02

Author: Jonathan J. Helmus <jjhe...@gmail.com>

Date:   Tue Jun 2 13:37:19 2015 -0500

    Merge pull request #309 from jjhelmus/python3_nexrad

    

    PY3: Python 3 support in nexrad_level2 and nexrad_level3

[Scott Collis]

Hey Anderson,

Are you able to help Greg?

-sent from a mobile device-

--
You received this message because you are subscribed to the Google Groups "Py-ART Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyart-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<230000.mdv>

[Jonathan Helmus]

Greg,

    The reason you are not reading in any field data from the file is that the name of the field in the MDV file does not match those recognized by Py-ART.  Specifically the field name 'refl' is specified in the file and we are expecting reflectivity fields to have the name 'DBZ' or 'DBZ_F' which is how they appear in radar MDV files produced by TITAN and similar programs.  In Py-ART you can ignore these default field mappings and use the field names in the file by adding the keyword parameter file_field_names=True to the read_grid_mdv function call [1]. 

    Unfortunately, if you try to use this on the sample MDV you provides and other from the link you will get another error, most likely NotImplementedError: unsupported compression mode.  In looking at the header information in these files and the source code in the readMDV.tar.gz tarball these "MDV" files do not conform to the specifications in the MDV FORMAT Interface Control Documentation and are using structures which are significantly different than those in that document.  From my understanding of the development of the MDV format there were many early version which seem to be incompatible, these files seem to be an earlier version which unfortunately Py-ART cannot read.

Cheers,

    - Jonathan Helmus

[1] http://arm-doe.github.io/pyart-docs-travis/user_reference/generated/pyart.io.read_grid_mdv.html

[Anderson Gama]

I just found this site describing an older version of MDV that seems to agree with the file Greg provided:
http://rap.ucar.edu/projects/IHL/RalHtml/protocols/mdv_file/

section 7 describes the encoding, it is a RLE, it is simple but I don't think there is a decompression tool available.

[Jonathan Helmus]

Great find Anderson!  Sigmet files also use RLE to compress data, I'll see if I can write up something than can decode these files.

    - Jonathan Helmus

[Jonathan Helmus]

Greg,

    I created a branch that I think can read these files, would you mind testing out the change and see if the results seem reasonable?  Branch is at https://github.com/jjhelmus/pyart/tree/mdv_grid_rle.  The sample file can be read using "grid = pyart.io.read_grid_mdv('230000.mdv', file_field_names=True)"

Cheers,

    - Jonathan Helmus

[Greg Blumberg - NOAA Affiliate]

Jonathan, Anderson, and Scott,

Thank you very much for all of your responses and help.  You've made this PhD student really, really happy today.  :) Using Jonathan's fork, I was able to get the files read in and the grid object now has data in it.  The data looks spot on (see attached) when I compare it to some radar graphics I have online.  I was able to get it plotted using Basemap and np.meshgrid(), but there are a few nuances I encountered when plotting this data using PyART:

1.) The lat, lon arrays that get returned in the axes attribute of the grid object look to be off by a few orders of magnitude.  Dividing them by 1000 makes them more reasonable.

2.) Using http://nbviewer.ipython.org/github/scollis/notebooks/blob/master/Gates%20to%20grid%20testing.ipynb, I tried plotting the data and the map background.  The reflectivity structures do pop up on the screen, but the display.plot_basemap() function doesn't seem to be working for this dataset.  I even tried this after resetting the lat, lon arrays, but it didn't help.

By the way guys, thank you again for the help.  This was my first time using PyART, and it's a really clean and great piece of software.  I'm quite impressed and hope I can find a way to contribute back to the project someday.

Best,

Greg

 

--
You received this message because you are subscribed to a topic in the Google Groups "Py-ART Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyart-users/n9hD38Yefys/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyart-users...@googlegroups.com.

[Anderson Gama]

Always happy to help Greg

The grid object is thought to contain data from a simple grid with axes in meter from the reference point (alt, lat, lon), not with axes in latlon degrees. As mdv allows that we are reading it to the grid, but other pyart routines may not work properly with this kind of grid axes, that is probably the origin of the problems you had. There is a open issue in GitHub discussing possible changes to the grid structure (https://github.com/ARM-DOE/pyart/issues/285).

About the wrong scale, that is a real bug, thank you for spotting that.

Jonathan this is a simple fix, lines 338 to 341 (https://github.com/ARM-DOE/pyart/blob/master/pyart/io/mdv_grid.py#L338-L341) shoudn't have the *1000 case "mdv.field_headers[0]["proj_type"] == mdv_common.PROJ_LATLON"

Cheers
Anderson

[Greg Blumberg]

Hi Anderson,

You're welcome.  I'm glad I can help!

I actually am in favor of having both types of units (i.e. lat/lon and x,y).  I'm sure there are those who prefer the x,y points, but 've found myself wrestling with many of the UCAR Motherlode datasets, which often only have the dimensions x,y.  I may be making this more difficult than it needs to be, but without the lat/lon grid, I can't easily resize or zoom in on the data using Basemap without getting the proper map projection settings first so I can back out the lat/lon grid.

Greg

[Jonathan Helmus]

Greg,

    Glad to hear your positive comments on Py-ART!  I made the change to the latitude and longitude scaling as suggested by Anderson in my fork and will be merging it into the master Py-ART branch shortly [1].  As Anderson mentioned the Grid display classes and function in Py-ART are designed around a Cartesian grid and will not work with latitude/longitude grids.  We are interested adding better support for grids of all kinds in Py-ART in the future, I just know I don't have the time for this at the moment.  For the time being using basemap directly as you have been doing is probably the best best.

Cheers,

    - Jonathan Helmus

[1] https://github.com/ARM-DOE/pyart/pull/313

[Federico otero]

Hello all, as same as Greg Im having trouble to open the mdv files. When I try to use the script proposed in this conversation I have an error reading de mdv files. This error is

"UnboundLocalError: local variable 'az_deg' referenced before assignment"

I've attached a sample MDV file to this post (230000.mdv).  

Can anyone help my?

Thanks!

Federico

[Anderson Gama]

Hello Federico

can you inform us the pyart version (it is in variable: pyart.version.full_version)

and also the complete trace back, any information about the file and line this error occurred would be useful

Cheers

Anderson

--

[Anderson Gama]

Federico

you have an old version of pyart, that didn't support Cartesian mdv (like the one you provided). Please update you pyart to a more recent version. I recommend release 1.4 (that happened in the 9th Juni).

If for some reason you want an older version, you need at least the 21th May

also, don't use read_mdv, but rather read_grid_mdv. We still can't automatically separate polar from cartesian MDV.

try that and come back to us if you have any problems

2015-06-20 20:30 GMT+02:00 Federico otero <federico...@gmail.com>:

Hi Anderson thak you for the quick answer, my pyart version is 1.4.0.dev+4dbc6c8. This is what Iget when I try to open the file:

import pyart import matplotlib.pyplot as plt pradar = pyart.io.read_mdv(filename)

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-14-1fc106edf2f9> in <module>()
      3 
      4 # plot quickly
----> 5 pradar = pyart.io.read_mdv(filename)

/usr/local/lib/python2.7/dist-packages/pyart/io/mdv.pyc in read_mdv(filename, field_names, additional_metadata, file_field_names, exclude_fields)
     80                                 file_field_names, exclude_fields)
     81 
---> 82     mdvfile = MdvFile(filename)
     83 
     84     # value attributes

/usr/local/lib/python2.7/dist-packages/pyart/io/mdv.pyc in __init__(self, filename, debug, read_fields)
    362         if debug:
    363             print "Calculating Radar coordinates"
--> 364         az_deg, range_km, el_deg = self._calc_geometry()
    365         self.az_deg = np.array(az_deg, dtype='float32')
    366         self.range_km = np.array(range_km, dtype='float32')

/usr/local/lib/python2.7/dist-packages/pyart/io/mdv.pyc in _calc_geometry(self)
    844             el_deg = self.vlevel_headers[0]['level'][0:nsweeps]
    845 
--> 846         return az_deg, range_km, el_deg
    847 
    848     def _make_time_dict(self):

[Jonathan Helmus]

Federico,

    Just as a data point, I was able to read the example MDV file you attached using Py-ART version 1.4.0 which can be installed from my binstar channel using conda [1] as well as the the latest version of Py-ART build from the source code on GitHub [2].  I used the following two lines of code to read the file:

import pyart
grid = pyart.io.read_grid_mdv('163428.mdv')

Cheers,

    - Jonathan Helmus

[1] https://binstar.org/jjhelmus/pyart
[2] https://github.com/ARM-DOE/pyart/blob/master/INSTALL.rst

You received this message because you are subscribed to the Google Groups "Py-ART Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pyart-users...@googlegroups.com.

[Jonathan Helmus]

Federico,

    I was able to plot the reflectivity field in the example MDV file using the following:

import pyart
import matplotlib.pyplot as plt

import pyart
grid = pyart.io.read_grid_mdv('163428.mdv')

display = pyart.graph.GridMapDisplay(grid)
display.plot_basemap()
display.plot_grid('reflectivity', level=0, vmin=-30, vmax=70)
plt.savefig('figure.png')


You can change the level number to plot the various z levels of the plot.  Note that you will need to have basemap installed for this script to work.

Cheers,

    - Jonathan Helmus

On 06/22/2015 10:34 AM, Federico otero wrote:

I can open the file as you propose, but I cant plot any grap! can you help me Jonathan?

Thanks

Federico

...