Nexrad Level III files from NOAAPORT

[Chad Bahrmann]

I have been working with NEXRAD Level III data files retrieved from 

ftp://tgftp.nws.noaa.gov/SL.us008001/DF.of/DC.radar

with code written using the Py-ART with beautiful results.

I tried implementing the same code using the PSU NOAAPORT NIDS product feed and I am receiving the following error:

 'Level3 product with code %i is not supported' % (code))

NotImplementedError: Level3 product with code 3338 is not supported

after quite a bit of searching and asking our data guru it appears this data may be compressed using bz2 compression for this product.

Does anyone have any suggestions?  I would rather use the data feed vs ftp'ing the files for routine image generation.

Thanks,

Chad

[Ryan May]

FYI, MetPy should be handling these just fine: https://github.com/metpy/MetPy/blob/master/metpy/io/nexrad.py

Feel free to lift and adapt for PyART the code for handling the compression.

Ryan

[Chad B]

Thank you for the comments Ryan.  I did find that MetPy is supporting, as you so eloquently stated, most NIDS compression formats found in the wild.  I think this would be a great addition to Py-ART as well and thank everyone for these wonderful tools.   

[Jonathan Helmus]

Chad, Ryan,

Py-ART should be able to decompress compressed bzip2 or zlib NEXRAD Level III files on the fly if the filename is provided to the pyart.io.read or pyart.io.read_nexrad_level3 function.  If a file-like object is passed to the read function it is assumed that the file has already been decompressed which for bzip2 compressed can be done using the bz2 module in the standard library [1]. 

If you are running into Level III files that Py-ART cannot read please send me a sample of one of them. I should be able to figure out what has changes and add support for them.

Cheers,

    - Jonathan Helmus

[1] https://docs.python.org/3.5/library/bz2.html

--
You received this message because you are subscribed to the Google Groups "Py-ART Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyart-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan May]

Jonathan,

Does that include NIDS files which can be composed of multiple zlib-compressed frames? I had to write code that calls zlib.decompress() repeatedly.

Ryan

You received this message because you are subscribed to a topic in the Google Groups "Py-ART Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyart-users/5PqUw4xsV7I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyart-users...@googlegroups.com.

[Chad B]

Hi Jonathan,

Thank you for the reply.   Test file is attached.

The simple code I was trying to use was:

import pyart

filename='N1X_20160606_0004'

radar = pyart.io.read_nexrad_level3(filename)

Error:

Traceback (most recent call last):

  File "me.py", line 3, in <module>

    radar = pyart.io.read_nexrad_level3(filename)

  File "/home/meteo/cpb11/anaconda2/lib/python2.7/site-packages/pyart/io/nexradl3_read.py", line 78, in read_nexrad_level3

    nfile = NEXRADLevel3File(prepare_for_read(filename))

  File "/home/meteo/cpb11/anaconda2/lib/python2.7/site-packages/pyart/io/nexrad_level3.py", line 135, in __init__

    'Level3 product with code %i is not supported' % (code))

NotImplementedError: Level3 product with code 3338 is not supported

[Jonathan Helmus]

Chad,

    The file you attached seems to have a 11-byte header before the "normal" NEXRAD Level 3 text header.  If you skip over this header Py-ART can read the file, for example:

import pyart
f = open('./N1X_20160606_0004', 'rb')
f.read(11)
radar = pyart.io.read(f)

I'll look into this a bit more and try to add something to Py-ART so that these files can be read directly without this hack.  Until then the above should work.

Ryan,

    The extra 11 bytes are b'\x01\r\r\n982 \r\r\n'.  Have you seen this before? Is it some type of NOAAPORT record separator?

Cheers,

    - Jonathan Helmus

[Chad B]

I was told by our data guru here the following:

Those bytes are part of the WMO transmission protocol, e.g.:

001  \r  \r  \n   3   4   8      \r  \r  \n 

SOH, CR-CR-LF, sequence number + space, CR-CR-LF

There's also a CR-CR-LF, ETX at the end of the file which may be
spurious, but the code may not notice that if the beginning's okay.

[Jonathan Helmus]

Exactly what I was seeing, thanks for the confirmation.  Since it seems official, I added the proper logic so that Py-ART can read these files directly and will automatically detect the file type so pyart.io.read will work.  Changes were made in PR 540 [1] and have been merged into the master branch if you care to update.

Thanks for bringing this issue up Chad!

Cheers,

    - Jonathan Helmus

[1] https://github.com/ARM-DOE/pyart/pull/540