Quality control algorithms for radar reflectivity data.

[Jure Žbontar]

Hello everyone,

I am using Py-ART to read NEXRAD Level II data downloaded from AWS. Since the

reflectivity data contain a significant amount of nonprecipitating echoes, I will

have to perform some form of quality control. I'm looking for automated

approaches that remove as much noise as possible including anomalous

propagation, ground clutter, biological targets, and sun strobes. Although the

literature contains many algorithms for identifying these nonprecipitating

echoes (see below), I was not able to find a free (GPL or BSD licensed)

implementation of any quality control algorithm. I have the following

questions:

1. Does Py-ART contain any quality control algorithms for reflectivity data? I

found a method for removing small objects (pyart.correct.despeckle), but this

will not identify anomalous propagation, for example.

2. If not Py-ART, do you know any other open-source libraries that implement

quality control algorithms? Warning Decision Support System

(http://wdssii.org/) is one, but is only available for research purposes.

Wradlib (http://wradlib.org/) is another, but it only contains a basic clutter

identification algorithm.

3. Would anybody be interested in helping me implement (and possibly integrate

into Py-ART) a quality control algorithm? 

Best,

Jure

--

I'm including a list of relevant literature:

- Richard A Fulton, Jay P Breidenbach, Dong-Jun Seo, Dennis A Miller, and Timothy O’Bannon.

The wsr-88d rainfall algorithm. Weather and Forecasting, 13(2):377–395, 1998.

- Mircea Grecu and Witold F Krajewski. An efficient methodology for detection of anomalous

propagation echoes in radar reflectivity data using neural networks. Journal of Atmospheric

and Oceanic Technology, 17(2):121–129, 2000.

- C Kessinger, S Ellis, and J Van Andel. The radar echo classifier: A fuzzy logic algorithm for

the wsr-88d. In Preprints-CD, 3rd Conference on Artificial Applications to the Environmental

Science, 2003.

- Witold F Krajewski and Bertrand Vignal. Evaluation of anomalous propagation echo detection

in wsr-88d data: A large sample case study. Journal of Atmospheric and Oceanic Technology,

18(5):807–814, 2001.

- Valliappa Lakshmanan, Kurt Hondl, Gregory Stumpf, and Travis Smith. Quality control of weather

radar data using texture features and a neural network. In Preprints, 31st Radar Conference,

pages 522–525. Citeseer, 2003.

- Valliappa Lakshmanan, Angela Fritz, Travis Smith, Kurt Hondl, and Gregory Stumpf. An

automated technique to quality control radar reflectivity data. Journal of applied meteorology

and climatology, 46(3):288–305, 2007.

- Valliappa Lakshmanan, Christopher Karstens, John Krause, and Lin Tang. Quality control of

weather radar data using polarimetric variables. Journal of Atmospheric and Oceanic Tech-

nology, 31(6):1234–1249, 2014.

- Valliappa Lakshmanan, Travis Smith, Gregory Stumpf, and Kurt Hondl. The warning decision

support system-integrated information. Weather and Forecasting, 22(3):596–612, 2007.

- Valliappa Lakshmanan, Jian Zhang, Kurt Hondl, and Carrie Langston. A statistical approach

to mitigating persistent clutter in radar reflectivity data. IEEE Journal of Selected Topics in

Applied Earth Observations and Remote Sensing, 5(2):652–662, 2012.

- Valliappa Lakshmanan, Jian Zhang, and Kenneth Howard. A technique to censor biological

echoes in radar reflectivity data. Journal of Applied Meteorology and Climatology, 49(3):453–

462, 2010.

[Jonathan Helmus]

Jure,

    The only other function that I Py-ART has that would fall into "quality control" would be the "texture_of_complex_phase" function which uses wradlib's phase texture function. 

You may also want to check out the list of Py-ART extension and related software [2] where you may find some open source quality control algorithms in these projects. 

    That said, Py-ART is currently lacking in terms of quality control algorithms and we would be vary interested in helping to incorporate some of these into Py-ART.   I do not have much time to implement these routines myself but I'd be happy to help review code from others for incorporation into Py-ART.

Cheers,

    - Jonathan Helmus

[1] http://arm-doe.github.io/pyart-docs-travis/user_reference/generated/pyart.bridge.texture_of_complex_phase.html
[2] https://github.com/ARM-DOE/pyart#extensions-and-related-software

--
You received this message because you are subscribed to the Google Groups "Py-ART Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyart-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jure Žbontar]

Hi all,

I implemented the quality-control algorithm by Steiner and Smith. The code is available on github (https://github.com/jzbontar/steiner_smith). I thought this would be useful to some of you here. Note that this is fresh code and there might still be some differences between our implementation and the algorithm described in the paper. Please see the Notes section of the readme file in the github repository.

On a related note, I have a question for you, Jonathan. What is the reason behind the `_interpolate_scan` function (https://github.com/ARM-DOE/pyart/blob/master/pyart/io/nexrad_archive.py#L285)? I had some problems related to it while implementing the qc algorithm.

Cheers,

Jure

[Jonathan Helmus]

Jure,

    Thanks for sharing your implementation of the Steiner and Smith algorithm, I'm sure many on this list will be interested in the work.

    As for the _interpolate_scan function in Py-ART's nexrad_archive.py module.  Some Level 2 files contain a mixture of sweeps (also called scans) where the spacing between gates varies between 250 meters and 1000 meters.  The data model Py-ART uses requires a uniform gate spacing for all sweeps, so it is necessary to convert all of these sweeps to a single spacing.  The _interpolate_scan function performs a interpolation (either linear or nearest neighbor) which take a sweep with 1000 meter gates and converts it into a sweep with 250 meter spacing. 

    When a file which contains mixed gate spacing sweeps is detected a UserWarning is issued when interpolation is performed.  The type of interpolation can be specified using the linear_interp argument and interpolation can be avoided completely by using the scans argument to specify sweeps which all have the same gate spacing.

Cheers,

    - Jonathan Helmus

[Jure Žbontar]

Hi Jonathan,

thanks for the explanation, it makes sense now.

Maybe I should explain why the interpolation bothers me so much. In their quality control algorithm, Steiner and Smith compute a feature called SPINchange. The SPINchange features counts, for each 11 x 21 window in polar coordinates, how many times the value of reflectivity changes (from gate to gate in the radial direction) by more than 2 dBZ. The SPINchange feature obviously depends on whether interpolation (nearest neighbour or linear) was used on the reflectivity field. This makes it a bit harder to fit a good threshold value for this feature.

Best,

Jure

[Greg Pearson]

Hi Jure and everyone.

I've been playing around with pyART and decluttering of weather radar data and came across your implementation of the Steiner/Smith algorithm Jure (thanks for making this available).

One thing that puzzles me is that you seem to have fixed sweeps in your implementation (0 & 2). As I understand the algorithm, for each pixel in the base sweep, you need to find the highest sweep that contains a reflectivity above a certain threshold (you actually do the same for the neighbours of the base pixel). So this 'top sweep' is likely to vary.

Thanks Jure and everyone, and apologies if this is the wrong place to post such a question.

Kind regards,

Greg

[Robert Jackson]

Jure:

This is not an algorithm that we have implemented in Py-ART, so your best bet would be to contact Jure directly for this question. Try raising an issue on the github for this algorithm: https://github.com/jzbontar/steiner_smith

Cheers,

Bobby

[Luciana Cunha]

Hi Jure,

Have you made progress on that? I am also looking into quality control the data from AWS.

Thank you so much!

Luciana