Reading NEXRAD Level III data obtained from NOAA got "Unknown or unsupported file format"

[Andy Zhang]

Hello everyone, 

I have downloaded NEXRAD Level III data from NOAA (1). The downloaded file is attached here (2). However, when I trying to open them using pyart, I got the following errors with these scripts:

1.

> radar = pyart.io.read(filename='C:/Users/zhang/Documents/NEXRAD/A1/NWS_NEXRAD_NXL3_KPBZ_20150101000000_20150101235959.tar.gz')

> TypeError: Unknown or unsupported file format: UNKNOWN

2. 

> radar = pyart.io.read(filename='C:/Users/zhang/Documents/NEXRAD/A1/NWS_NEXRAD_NXL3_KPBZ_20150101000000_20150101235959.tar')

>TypeError: Unknown or unsupported file format: UNKNOWN

3. 

> radar = pyart.io.read(filename='C:/Users/zhang/Documents/NEXRAD/A1/NWS_NEXRAD_NXL3_KPBZ_20150101000000_20150101235959/KPBZ_NXUS61_GSMPBZ_201501010004')

>TypeError: Unknown or unsupported file format: UNKNOWN 

 

 

Am I wrong in somewhere? I am a new student to PyART and radar data, looking forward to your suggestions! 

1. https://www.ncei.noaa.gov/access/search/data-search/weather-radar-level-iii?dataTypes=DVL&startDate=2015-01-01T00:00:00&endDate=2015-01-02T00:00:59&stations=KPBZ

2. https://drive.google.com/drive/folders/1kGNLS5tc7jp7uVInm4HgAL6iTNK35ymK?usp=sharing

[Scott Collis]

You need to ungzip or pipe through python gzip library 

-sent from a mobile device-

On Dec 15, 2020, at 11:54 PM, Andy Zhang <andyz...@gmail.com> wrote:

Hello everyone, 

--
You received this message because you are subscribed to the Google Groups "Py-ART Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyart-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyart-users/52f34418-d9cd-4280-bed2-30cc20da9bfen%40googlegroups.com.

[Andy Zhang]

Thank you Scott! I implemented the ungzip function in my code, but the new problem is that there is an error "ValueError: embedded null byte". Could you please share me some insights on this error? Many thanks!

> Code:

import matplotlib.pyplot as plt

import pyart

import gzip 

import tarfile

# open the file

filename = './Data/NEXRAD/A1/NWS_NEXRAD_NXL3_KPBZ_20150101000000_20150101235959.tar.gz'

if filename.endswith("tar.gz"):

    tar = tarfile.open(filename, "r:gz")

    for member in tar.getmembers():

        f = tar.extractfile(member)

        if f is not None:

            content = f.read()

            radar = pyart.io.read(content, file_field_names=True)

            print(radar.info('compact'))

> Error message:

Traceback (most recent call last):

  File "/media/andy/b4a51c70-19cd-420f-91e4-c7adf2274c39/Proposal/scripts/nexrad_data_pyart.py", line 31, in <module>

    radar = pyart.io.read(content, file_field_names=True)

  File "/home/andy/miniconda3/envs/pyart_env/lib/python3.7/site-packages/pyart/io/auto_read.py", line 74, in read

    filetype = determine_filetype(filename)

  File "/home/andy/miniconda3/envs/pyart_env/lib/python3.7/site-packages/pyart/io/auto_read.py", line 182, in determine_filetype

    f = open(filename, 'rb')

ValueError: embedded null byte

Again, the original file is here (https://drive.google.com/drive/folders/1kGNLS5tc7jp7uVInm4HgAL6iTNK35ymK?usp=sharing)

Thanks,

Zhuoran

[Collis, Scott M.]

Are those level 3 files?

To view this discussion on the web visit https://groups.google.com/d/msgid/pyart-users/2e2cdc27-1253-4044-805c-29eb2ff762ben%40googlegroups.com.

[Andy Zhang]

Yes, they are level 3 files.

[Collis, Scott M.]

Yeah.. my experience with Level 3 is limited.. if you are more interested in raw fields use level2

 

Have you tried taking one file and manually un-tar-gzipping and reading the files?

 

Zach, others, what is the state of a our level3 reading? Is it an Aux-io function?

To view this discussion on the web visit https://groups.google.com/d/msgid/pyart-users/2238496e-4c93-4d18-b056-aa39c9cb9c9en%40googlegroups.com.

[Jason Webster]

After unzipping the level 3, the individual files can be read with the following

pyart.io.read_nexrad_level3(‘KPBZ_SDUS51_N0QPBZ_201501010006')

To view this discussion on the web visit https://groups.google.com/d/msgid/pyart-users/09CE5356-4802-4AE3-A0D7-7722C90A97F1%40gmail.com.

[Collis, Scott M.]

So problem is not with Py-ART 😊

Maybe pipe from pythons gunzip -> tar

 

I have some code I did with level2 I will see if I can find…

[Jason Webster]

I suggest using the following

    from sh import gunzip

    import tarfile

Use gunzip to expand and tarfile to extract out the level 3 files. 

If you want to be efficient, you can use tarfile to extract only the level 3 files you want

I’ll try to put together a complete example later.

[Collis, Scott M.]

Here is my (very very old) code

https://github.com/scollis/houston/blob/master/scripts/stage_has.py

 

I use a tempfile.. This also manages ftping from NCEI

 

Note: I don’t use this anymore.. All my data comes from S3 now 😃

 

 

From: Jason Webster <jrwe...@atmosci.com>

Date: Wednesday, December 16, 2020 at 10:39 AM
To: <pyart...@googlegroups.com>
Cc: <scolli...@gmail.com>, <andyz...@gmail.com>

[Collis, Scott M.]

Thanks Jason!