Syntax for opening and reading a BUFR file

[Mark Dempsey]

I'm hoping someone might be able to give me an idea of how to open and read a BUFR file with nctoolbox.  I get as far as
ds = ncdataset('5900.20030601.surface');
then I type
bctd = ds.netcdf.getVariables()

and get

bctd =
 
[
   Sequence {
     String time;
       :units = "minutes since 2003-06-01 00:00:00Z";
       :long_name = "time of observation";
       :_CoordinateAxisType = "Time";
     byte WMO block number;
       :missing_value = 127B; // byte
       :BUFR:TableB_descriptor = "0-1-1";
       :BUFR:bitWidth = 7; // int
     short WMO station number;
       :missing_value = 1023S; // short
       :standard_name = "station_WMO_id";
       :BUFR:TableB_descriptor = "0-1-2";
       :BUFR:bitWidth = 10; // int
     String Type of station;
       :units = "CodeTable 0-2-1";
       :BUFR:CodeTable = "Type of station (0-2-1)";
       :BUFR:TableB_descriptor = "0-2-1";
       :BUFR:bitWidth = 2; // int
     short Year;
       :units = "Year";
       :missing_value = 4095S; // short
       :BUFR:TableB_descriptor = "0-4-1";
       :BUFR:bitWidth = 12; // int
     byte Month;
       :units = "Month";
       :missing_value = 15B; // byte
       :BUFR:TableB_descriptor = "0-4-2";
       :BUFR:bitWidth = 4; // int
     byte Day;
       :units = "Day";
       :missing_value = 63B; // byte
       :BUFR:TableB_descriptor = "0-4-3";
       :BUFR:bitWidth = 6; // int
     byte Hour;
       :units = "Hour";
       :missing_value = 31B; // byte
       :BUFR:TableB_descriptor = "0-4-4";
       :BUFR:bitWidth = 5; // int
     byte Minute;
       :units = "Minute";
       :missing_value = 63B; // byte
       :BUFR:TableB_descriptor = "0-4-5";
       :BUFR:bitWidth = 6; // int
     float Latitude (coarse accuracy);
       :units = "degrees_north";
       :_CoordinateAxisType = "Lat";
       :BUFR:TableB_descriptor = "0-5-2";
       :BUFR:bitWidth = 15; // int
     float Longitude (coarse accuracy);
       :units = "degrees_east";
       :_CoordinateAxisType = "Lon";
       :BUFR:TableB_descriptor = "0-6-2";
       :BUFR:bitWidth = 16; // int
     float Height of station;
       :units = "m";
       :_CoordinateAxisType = "Height";
       :BUFR:TableB_descriptor = "0-7-1";
       :BUFR:bitWidth = 15; // int
     char Short station or site name(5);
       :standard_name = "station_id";
       :BUFR:TableB_descriptor = "0-1-18";
       :BUFR:bitWidth = 40; // int
     String Time significance;
       :units = "CodeTable 0-8-21";
       :BUFR:CodeTable = "Time significance (0-8-21)";
       :BUFR:TableB_descriptor = "0-8-21";
       :BUFR:bitWidth = 5; // int
     float Time period or displacement;
       :units = "Second";
       :BUFR:TableB_descriptor = "0-4-26";
       :BUFR:bitWidth = 13; // int
     float Wind speed;
       :units = "m s-1";
       :BUFR:TableB_descriptor = "0-11-2";
       :BUFR:bitWidth = 12; // int
     short Wind direction;
       :units = "Degree true";
       :missing_value = 511S; // short
       :BUFR:TableB_descriptor = "0-11-1";
       :BUFR:bitWidth = 9; // int
     float Pressure;
       :units = "Pa";
       :BUFR:TableB_descriptor = "0-10-4";
       :BUFR:bitWidth = 14; // int
       :_CoordinateAxisType = "Pressure";
     float Pressure reduced to mean sea level;
       :units = "Pa";
       :BUFR:TableB_descriptor = "0-10-51";
       :BUFR:bitWidth = 14; // int
     float Temperature/dry-bulb temperature;
       :units = "K";
       :BUFR:TableB_descriptor = "0-12-1";
       :BUFR:bitWidth = 12; // int
     byte Relative humidity;
       :units = "%";
       :missing_value = 127B; // byte
       :BUFR:TableB_descriptor = "0-13-3";
       :BUFR:bitWidth = 7; // int
     float Rainfall/water equivalent of snow (averaged rate);
       :units = "kg m-2 s-1";
       :BUFR:TableB_descriptor = "0-13-14";
       :BUFR:bitWidth = 12; // int
   } obs(*);

]
Then I'm not sure what to do next. If anyone has any thoughts I would be grateful.
Kind regards,
Mark

[Brian Schlining]

Hi Mark,

Take a look at the file 'nctoolbox/demo/demo10m' for an example of working with structures. Basically, you have to drop down to the Netcdf-Java API. I know that's less than ideal, but at least it works. (We haven't really done much work in nctoolbox to deal with structures… sorry) Anyway you can do something like the following to extract what you need:

ds = ncdataset('5900.20030601.surface');

% Grab the root structure

obs = n.netcdf.findVariable('obs') 

% Extract a variable of interest from the structure

windSpeed = obs.findVariable('Wind speed').read().copyToNDJavaArray();

% windSpeed is float (single precision); make it Matlab-friendly double

ws = double(windSpeed);

Hope that helps!!

Cheers

--
Brian Schlining

--
You received this message because you are subscribed to the Google Groups "nctoolbox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nctoolbox+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mark Dempsey]

Hi Brian,
Wow! I cannot thank you enough for your help in all of this. That is so remarkably easy. I had looked at several other ways before this method and had only some success. The BUFR format has been oddly elusive when it comes to decoding.
Thank you so much again!
Mark

[Brian Schlining]

Glad to help!

[Mark J. Dempsey]

Hi Brian,

I hope you are well. 

I've one more question that has proven irksome. I am attempting to open another BUFR file. This time there are internal structures. I am running the following matlab script:

ds = ncdataset('5900.20030601.rass');

obs = ds.netcdf.findVariable('obs')

t = obs.getVariables().get(0)

time = t.read().copyToNDJavaArray();

 

stID = obs.getVariables().get(2)

stationID = stID.read().copyToNDJavaArray();

 

yr = obs.getVariables().get(4)

year = yr.read().copyToNDJavaArray();

 

mn = obs.getVariables().get(5)

month = mn.read().copyToNDJavaArray();

 

dy = obs.getVariables().get(6)

day = dy.read().copyToNDJavaArray()

 

hr = obs.getVariables().get(7)

hour = hr.read().copyToNDJavaArray()

 

min = obs.getVariables().get(8)

minute = min.read().copyToNDJavaArray()

 

lat = obs.getVariables().get(9)

latitude = lat.read().copyToNDJavaArray()

 

lon = obs.getVariables().get(10)

longitude = lon.read().copyToNDJavaArray()

 

height = obs.getVariables().get(11)

hagl = height.read().copyToNDJavaArray()

 

sn = obs.getVariables().get(12)

shortName = sn.read().copyToNDJavaArray()

 

% descending into the structure

 

seq1 = ds.netcdf.findVariable('obs.seq1')

 

VTs = seq1.getVariables().get(1)

This all runs fine, but now I would like to access the Virtual Temperature variable and when I call the 

VT = VTs.read().copyToNDJavaArray()

I get the following error

Java exception occurred:

java.lang.ClassCastException: ucar.ma2.ArrayObject$D1 cannot be cast to ucar.ma2.ArrayStructure

at ucar.nc2.Variable.reallyRead(Variable.java:854)

at ucar.nc2.Variable._read(Variable.java:831)

at ucar.nc2.Variable.read(Variable.java:709)

at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:518)

at ucar.nc2.dataset.VariableDS._read(VariableDS.java:502)

at ucar.nc2.Variable.read(Variable.java:709)

I wonder if you might be able to clarify for me. I'm unclear about what the next step is that I am to take. I will include the BUFR file.

You've helped tremendously already. Thank you.

kind regards, 

Mark

[Brian Schlining]

Hi All,

I'm stumped on how to get data out of the nested sequence data in the attached BUFR file using NetCDF-Java. I can read all variables in the top-level Sequence (obs), but I can't get them out of the nested Sequence (seq1). I've tried all the method calls I can think of. Any suggestions or help would be greatly appreciated? Here's some pseudo code I've used to get the variable and try to read it.:

netcdf = //  Open the BUFR file

seq1 = netcdf.findVariable("obs.seq1);

// --- METHOD 1L Try using the recommended structure iterator

i = seq1.getStructureIterator();

while (i.hasNext()) {

    v = i.next();

    d = v.getArray();

}

//  METHOD 1 OUTPUT

2013-05-14 14:55:24,154 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member time

2013-05-14 14:55:24,154 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member WMO block number

2013-05-14 14:55:24,154 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member WMO station number

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Type of station

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Year

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Month

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Day

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Hour

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Minute

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Latitude (coarse accuracy)

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Longitude (coarse accuracy)

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Height of station

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Short station or site name

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Type of measuring equipment used

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Time significance

2013-05-14 14:55:24,155 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member Time period or displacement

2013-05-14 14:55:24,156 [main] WARN  ucar.nc2.dataset.StructureDS  - StructureDataDS.convert Cant find member seq1

Java exception occurred:

java.lang.NullPointerException

at ucar.nc2.dataset.StructureDS.convert(StructureDS.java:445)

at ucar.nc2.dataset.SequenceDS$StructureDataConverter.next(SequenceDS.java:78)

// --- METHOD 2: Try calling read

vt = netcdf.findVariable('obs.seq1.Virtual temperature);

vt.read();

// METHOD 2 OUTPUT

java.lang.ClassCastException: ucar.ma2.ArrayObject$D1 cannot be cast to ucar.ma2.ArrayStructure

at ucar.nc2.Variable.reallyRead(Variable.java:854)

at ucar.nc2.Variable._read(Variable.java:831)

at ucar.nc2.Variable.read(Variable.java:709)

at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:518)

at ucar.nc2.dataset.VariableDS._read(VariableDS.java:502)

at ucar.nc2.Variable.read(Variable.java:709)

The CDL is:

netcdf 5900.20030601.rass {

 types:

  enum Type of station { 'Automatic' = 0, 'Manned' = 1, 'Hybrid: both manned and automatic' = 2, 'Missing value' = 3};

  enum Type of measuring equipment used { 'Pressure Instrument associated with wind measuring equipment' = 0, 'Optical theodolite' = 1, 'Radio theodolite' = 2, 'Radar' = 3, 'VLF-Omega' = 4, 'Loran C' = 5, 'Wind profiler' = 6, 'Satellite navigation' = 7, 'Radio-acoustic Sounding System (RASS)' = 8, 'Sodar' = 9, 'Pressure instrument associated with wind measuring equipment but pressure element failed during ascent' = 14, 'Missing value' = 15};

  enum Time significance { 'Time series' = 1, 'Time averaged (see Note 1)' = 2, 'Accumulated' = 3, 'Forecast' = 4, 'Forecast time series' = 5, 'Forecast time averaged' = 6, 'Forecast accumulated' = 7, 'Ensemble mean (see Note 2)' = 8, 'Ensemble mean time series' = 9, 'Ensemble mean time averaged' = 10, 'Ensemble mean accumulated' = 11, 'Ensemble mean forecast' = 12, 'Ensemble mean forecast time series' = 13, 'Ensemble mean forecast time averaged' = 14, 'Ensemble mean forecast accumulated' = 15, 'Start of phenomenon' = 17, 'Analysis' = 16, 'Start of orbit' = 19, 'Radiosonde launch time' = 18, 'Time of ascending node' = 21, 'End of orbit' = 20, 'Monitoring period' = 23, 'Time of occurrence of wind shift' = 22, 'Nominal reporting time' = 25, 'Agreed time limit for report reception' = 24, 'First guess' = 27, 'Time of last known position' = 26, 'End of scan or time of ending' = 29, 'Start of scan' = 28, 'Missing value' = 31, 'Time of occurrence' = 30};

 variables:

     String Type of measuring equipment used;

       :units = "CodeTable 0-2-3";

       :BUFR:CodeTable = "Type of measuring equipment used (0-2-3)";

       :BUFR:TableB_descriptor = "0-2-3";

       :BUFR:bitWidth = 4; // int

     String Time significance;

       :units = "CodeTable 0-8-21";

       :BUFR:CodeTable = "Time significance (0-8-21)";

       :BUFR:TableB_descriptor = "0-8-21";

       :BUFR:bitWidth = 5; // int

     float Time period or displacement;

       :units = "Second";

       :BUFR:TableB_descriptor = "0-4-26";

       :BUFR:bitWidth = 13; // int

     Sequence {

       short Height above station;

         :units = "m";

         :missing_value = 32767S; // short

         :_CoordinateAxisType = "Height";

         :BUFR:TableB_descriptor = "0-7-6";

         :BUFR:bitWidth = 15; // int

       float Virtual temperature;

         :units = "K";

         :BUFR:TableB_descriptor = "0-12-7";

         :BUFR:bitWidth = 12; // int

       byte Wind profiler quality control test results;

         :units = "FlagTable 0-25-34";

         :missing_value = 15B; // byte

         :BUFR:TableB_descriptor = "0-25-34";

         :BUFR:bitWidth = 4; // int

     } seq1;

   } obs(*);

 :history = "Direct read of BUFR data by CDM";

 :location = "5900.20030601.rass";

 :BUFR:edition = 3; // int

 :BUFR:categoryName = "Vertical soundings (other than satellite)";

 :BUFR:category = 2; // int

 :BUFR:subCategory = 0; // int

 :BUFR:localSubCategory = -1; // int

 :BUFR:centerName = "59.0 (The NOAA Forecast Systems Laboratory, Boulder, CO, USA)";

 :BUFR:center = 59; // int

 :BUFR:subCenter = 0; // int

 :BUFR:table = 0; // int

 :BUFR:tableVersion = 3; // int

 :BUFR:localTableVersion = 3; // int

 :WMO Header = "";

 :Conventions = "BUFR/CDM";

 :_CoordSysBuilder = "ucar.nc2.dataset.conv.DefaultConvention";

}

-- 
Brian Schlining

[Brian Schlining]

Hi Mark,

Sorry in the delay in getting back to you. Your email got hung up in the Google Groups Spam filter. 

[snip ...] 

I've one more question that has proven irksome. I am attempting to open another BUFR file. This time there are internal structures. I am running the following matlab script:

ds = ncdataset('5900.20030601.rass');

obs = ds.netcdf.findVariable('obs')

t = obs.getVariables().get(0)

time = t.read().copyToNDJavaArray();

 

[snip …]

sn = obs.getVariables().get(12)

shortName = sn.read().copyToNDJavaArray()

 

% descending into the structure

 

seq1 = ds.netcdf.findVariable('obs.seq1')

 

VTs = seq1.getVariables().get(1)

This all runs fine, but now I would like to access the Virtual Temperature variable and when I call the 

VT = VTs.read().copyToNDJavaArray()

I get the following error

Java exception occurred:

java.lang.ClassCastException: ucar.ma2.ArrayObject$D1 cannot be cast to ucar.ma2.ArrayStructure

at ucar.nc2.Variable.reallyRead(Variable.java:854)

at ucar.nc2.Variable._read(Variable.java:831)

at ucar.nc2.Variable.read(Variable.java:709)

at ucar.nc2.dataset.VariableDS.reallyRead(VariableDS.java:518)

at ucar.nc2.dataset.VariableDS._read(VariableDS.java:502)

at ucar.nc2.Variable.read(Variable.java:709)

I wonder if you might be able to clarify for me. I'm unclear about what the next step is that I am to take. I will include the BUFR file.

OK, I'm stumped on this one. I walked through nearly every method call on VTs that I could find and dug into the NetCDF-Java docs I could find on Sequence/Structure data. I was unable to get any data out of the seq1 object. This could either be my ignorance of structures, but it could also be an issue with the netcdf-java library. I've posted the question to the NetCDF-Java mailing list.

Once I have an answer, I'll translate it to Matlab for you. (I'd like to know how to do it too.)

p.s. If you're ever curious to see what methods can be called on an object in Matlab you can use the methods function. For example, here's a listing of methods that can be called on the netcdf object you used above:

    methods(ds.netcdf)

With that list in hand you can try each out like so:

   % Use as: object.method

   ds.netcdf.getDetailInfo

   ds.netcdf.getIosp

   % etc

Cheers

-- 

Brian Schlining 

[Mark J. Dempsey]

Hi Brian,

Good to hear from you. 

I wonder if this helps:

The information is for a wind profiler RASS set up which is obtaining information from three beams. There would be three sets of numbers for about 15 levels into the atmosphere and for about 12 stations or so. I'm not sure if the final array is three dimensions or three two dimensional arrays. But this is the nature of the structure, if I'm not mistaken.

Anyway thanks for your interest in this. I've been working on deciphering these BUFR files in matlab for some time now and I feel like I'm getting closer with the help of people like yourself. Thank you.

kind regards,

Mark

--
You received this message because you are subscribed to a topic in the Google Groups "nctoolbox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nctoolbox/B4SY36BE-i4/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to nctoolbox+...@googlegroups.com.

[user343553]

Hi Mark!

Did you find a solution for your problem? I ran into the same structure (nested sequences) in my data set. Did you find any solution?

Thanks!

[Mark J. Dempsey]

Hi Am,

Unfortunately I did not have any luck. Instead I ended up using the BUFRDC software to obtain an ascii version and then wrote a script to translate the results into a netCDF file. If you decide to go down this route, let me know and I would be happy to share my matlab scripts. Are you also looking at profiler data?

kind regards,

Mark

To unsubscribe from this topic, visit https://groups.google.com/d/topic/nctoolbox/B4SY36BE-i4/unsubscribe.

[Niranjan Kumar]

Hello, Mark and Brian

I read all the previous comment regarding  BUFR data issue. I am now in the same problem as I need to read the data from *.bfr file which I attached. I am using the following code and i am getting the errors (commented) given below. If you cold able to solve the problem now, kindly let me know

Thank you

clear all;close all;fclose all;clc

 file='MSG3-SEVI-MSGCLAP-0000-0000-20150101004500.000000000Z-20150101005935-1187380.bfr'

ds = ncgeodataset(file);

% ds = 

% 

%   ncgeodataset with properties:

% 

%      location: 'MSG3-SEVI-MSGCLAP-0000-0000-20150101004500.000000000Z-20150101005935-1187380.bfr'

%        netcdf: [1x1 ucar.nc2.dataset.NetcdfDataset]

%     variables: {'obs'}

ds.data('obs')

% Error using ncdataset/readdata (line 467)

% Java exception occurred:

% java.lang.NullPointerException

% 

% at ucar.ma2.ArrayStructure.copyBytes(ArrayStructure.java:484)

% 

% at ucar.ma2.ArrayStructure.extractMemberArray(ArrayStructure.java:430)

% 

% at ucar.ma2.ArrayStructureMA.factoryMA(ArrayStructureMA.java:226)

% 

% at ucar.nc2.dataset.StructureDS.convert(StructureDS.java:305)

% 

% at ucar.nc2.dataset.StructureDS.convert(StructureDS.java:338)

% 

% at ucar.nc2.dataset.SequenceDS.read(SequenceDS.java:111)

% 

% 

% Error in ncdataset/data (line 165)

%                     d = obj.readdata(variable);

% 

% Error in ncgeodataset/subsref (line 630)

%                             B = builtin('subsref',obj,g);