GFS Grib2 data

[Dan H]

Hi

I am trying to use Iris to subset GFS (NCEP) data. I have downloaded a full single timestep from http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod and stored the file locally. 

Using this script:

import iris

def callback_grib_missingmeta(cube, grib_wrapper, filename):

    try:

        cube.attributes.update({'table2Version':grib_wrapper.table2Version,

                                'indicatorOfParameter':grib_wrapper.indicatorOfParameter,

                                'grib.edition':grib_wrapper.grib.edition})

    except AttributeError:

        print 'Cannot find GRIB key parameterCategory'

filename='/tmp/gfs/test.grb2'

iris.FUTURE.strict_grib_load = True 

cubes = iris.load(filename, callback=callback_grib_missingmeta)

print (cubes)

, I am able to open the file test.grb2, but for 43 of the first 44 variables I get the error message Cannot find GRIB key parameterCategory, and on printing cubes these are shown as "0: unknown / (unknown)                 (latitude: 721; longitude: 1440)" 

from field 45 onwards, through to 155 everything seems fine, including knowledge of different levels, eg 110: relative_humidity / (%)             (pressure: 26; latitude: 721; longitude: 1440)

I am used to opening GFS grib data in GrADS, and having to supply supplimentary files to describe the grib data.  Is there something I am missing, or a way of giving IRIS the NCEP data codes http://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/?

Thanks for any help you can provide 

Dan                                                                                                                                                                                                 

  gitude

[Andrew Dawson]

Hi Dan

This is quite a multi-faceted issue. If I understand you correctly, your main problem is the missing metadata for some fields. This is likely due to some fields in the GFS output not having translations to CF metadata that iris can use. If you can find out what fields these are we can probably get them added via the metarelate project. It looks like what you are trying to do with the callback is exactly this, add some metadata to work out what the 'unknown' fields actually are. However, there are a couple of problems with the code you posted, I think to do with confusion between GRIB editions 1 and 2.

Firstly, you are using iris.FUTURE.strict_grib_load=True which is a good thing, but it does mean that the object provided as the second argument to your callback is not a GribWrapper object anymore, which in turn means you can't directly access GRIB keys as attributes. Instead you need to know which section each coded key you want is in and access it via the relevant index of the sections dictionary. For example, to get the edition number, which is stored in the key 'editionNumber' in section 0:

edition = grib_wrapper.sections[0]['editionNumber']

Secondly, I think you have the incorrect key name for table version, did you want 'tablesVersion' (also in section 0)? Thirdly, with GRIB 2 I don't think there is directly an 'indicatorOfParameter' key, instead there are a set of 3 keys to determine a parameter: 'discipline', 'parameterCategory', 'parameterNumber'. Taking all this into account you could re-write your callback as

def callback(cube, grib_message, filename):
    cube.attributes.update({
        'tablesVersion': grib_message.sections[1]['tablesVersion'],
        'editionNumber': grib_message.sections[0]['editionNumber'],
        'discipline': grib_message.sections[0]['discipline'],
        'parameterCategory': grib_message.sections[4]['parameterCategory'],
        'parameterNumber': grib_message.sections[4]['parameterNumber']})

This should put the required extra attributes on the cube produced with the load call. You can then write back with this information and we'll see what we can do about fixing the missing translations.

Hope that helps.

Andrew

[Dan H]

Hi Andrew, thanks for the reply.

Yes your change to the script does give me the required attributes where the field is unknown..

For example, cube[0] has this information:

     Attributes:                                                                                                                                                                                                                                                        

          discipline: 0                                                                                                                                                                                                                                                 

          editionNumber: 2                                                                                                                                                                                                                                              

          parameterCategory: 2                                                                                                                                                                                                                                          

          parameterNumber: 224                                                                                                                                                                                                                                          

          tablesVersion: 2    

Is this the information I need to send for the missing translations?

Dan

Dan     Attributes:                                                                                                                                                                                                                                                        

2      

On Thursday, 21 January 2016 23:12:30 UTC, Dan H wrote:

[Andrew Dawson]

Can you post the url of the actual file you are attempting to read. As far as I can tell parameter number 224 might be defined by local tables, and we'd need to know more about the file before we can figure out how to translate it properly.

[Dan H]

Andrew, I am trying to open a full NCEP GFS 0.25deg timestep.

I just re-tried the script with the latest available data for timestep T+3, downloaded from the following URL: http://www.ftp.ncep.noaa.gov/data/nccf/com/gfs/prod/gfs.2016012312/gfs.t12z.pgrb2.0p25.f003  

Dan

On Thursday, 21 January 2016 23:12:30 UTC, Dan H wrote:

[marqh]

Hello Dan

Andrew is correct in pointing out that a GRIB2 parameter
0-2-224
(disc-pCat-pNum)

is a local definition, not defined by the WMO:
http://codes.wmo.int/grib2/codeflag/4.2/0-2-224
returns a 404

Where GRIB2 parameter codes are WMO codes, we maintain a collection of translations which you are very welcome  to contribute to.  It is this collection which correctly translates the relative humidity codes into the name and unit of measure.

You can contribute translations for any GRIB2 codes which resolve within
http://codes.wmo.int/grib2/codeflag/4.2/
to
http://www.metarelate.net/metOcean
providing CF names and units

If there are locally defined codes, such as 0-2-224 from NCEP then we don't have a good translation mechanism for these at present;
you are likely to need a customised callback for such cases

mark

[Dan H]

Ok thanks for the reply Mark, I'll take a look..

On Thursday, 21 January 2016 23:12:30 UTC, Dan H wrote: