RDR files and the RT-STPS bug

[Adam]

Hi

The RDR files we generate here at our SCISYS X/L band station at SMHI,
Norrköping, all have orbit number equals 1, both in the filename and
in the attributes in the hdf5 file. This is apparently a bug in the
NASA/DRL RT-STPS software. This software is used to go from the raw
CADU to the level 0 RDR files.

Do you know more about this bug and when it will be fixed?

In the mean time we elaborated a little bit with h5py to make a small
program to fix this afterwards. I use pyorbital from pytroll to get
the orbit number from the observation times, and I get the observation
times (of the granules) by scanning the hdf5 file. It is quite easy.
Below is the hdf5 code to fix the file. It not difficult (thanks to
Martin reminding me that you don't have to copy the entire file
content, you can just change the relevant attributes in the h5py file
object).

But despite the fact that the file looks exactly as the origin (except
of course for the changed orbit number) it is not anymore liked by
CSPP. I used hdfview and h5dump to look at the file content. This is
the error message from CSPP:

INFO:adl_geo_ref:GEO file
GIMGO_npp_d20120405_t0049586_e0051227_b00001_c20120408194840733101_cspp_dev.h5
belongs to /data/proj6/safutv/cspp_test2/work/
SVI04_npp_d20120405_t0049586_e0051227_b00001_c20120411074624762365_cspp_dev.h5
Traceback (most recent call last):
File "/data/proj/safutv/cspp_test2/CSPP3.1/viirs/sdr/
adl_viirs_sdr.py", line 613, in <module>
sys.exit(main())
File "/data/proj/safutv/cspp_test2/CSPP3.1/viirs/sdr/
adl_viirs_sdr.py", line 609, in main
return viirs_sdr( work_dir, args.filenames )
File "/data/proj/safutv/cspp_test2/CSPP3.1/viirs/sdr/
adl_viirs_sdr.py", line 542, in viirs_sdr
add_geo_attribute_to_h5(work_dir, granules_to_process)
File "/data/proj/safutv/cspp_test2/CSPP3.1/viirs/sdr/
adl_viirs_sdr.py", line 445, in add_geo_attribute_to_h5
to_convert = list(get_created_blobs_list(work_dir,
granules_to_process, skim_dir(work_dir,
N_Collection_Short_Name=short_name)))
File "/data/proj/safutv/cspp_test2/CSPP3.1/viirs/sdr/
adl_viirs_sdr.py", line 423, in get_created_blobs_list
pa = it['BlobPath']
KeyError: 'BlobPath'
OOPS: VIIRS SDR did not complete without errors


Anyone having good ideas?

Best regards
Adam

============================

def fix_nppfile4orbitnumber(orbits, npp_file, outdir):
"""Fix the NPP VIIRS RDR/SDR files with respect to oribit number.
The
RDR/SDR file is read and the correct orbit number is inserted, and
the
filename is fixed as well."""
import shutil

# Determine the new output filename:
start_orbnum = orbits['start']
npp_filename = os.path.basename(npp_file)
new_filename = (npp_filename.split('_b')[0] +
'_b%.5d' % (start_orbnum) +
npp_filename[npp_filename.find('_c')::])
outfile = os.path.join(outdir, new_filename)

if os.path.exists(outfile):
print "File exists! %s" % os.path.basename(outfile)
return outfile

shutil.copy(npp_file, outfile)


# Start browsing file and change the orbit number:
out = h5py.File(outfile)

for group in out['/Data_Products'].keys():
for dset in out['/Data_Products'][group]:
# Attributes:
subsubg_attrs = out['/Data_Products'][group][dset].attrs
for key, val in subsubg_attrs.items():
if key in ['AggregateBeginningOrbitNumber',
'N_Beginning_Orbit_Number']:
start_orbnum = orbits[group+dset+'Beginning']
#print "Change orbit number from 1", start_orbnum
val = np.array([[start_orbnum]]).astype('uint64')

out['/Data_Products'][group][dset].attrs[key] =
val

if key in ['AggregateEndingOrbitNumber']:
end_orbnum = orbits[group+dset+'Ending']
#print "Change orbit number from 1", end_orbnum
val = np.array([[end_orbnum]]).astype('uint64')

out['/Data_Products'][group][dset].attrs[key] =
val

return outfile

[Herve ROQUET]

Dear Adam,

we have sent an e-mail to DRL about this problem with RT-STPS. They told
us that they will fix it, but without commitment for a date.

Best regards,

Hervé

[Adam Dybbroe]

Okay, thanks Herve.

Sometimes it is hard to plan your work when you have no date expecting
such things to be solved.
Considering that it only requires a few lines of code to fix it, and
very little work, and in addition it goes fast with h5py, I thought our
post fix was worth pursuing. But I got unexpected problems it seems.
Maybe wee need to wait. Too bad! :-(

-Adam

--
Adam Dybbroe,
Satellite Remote Sensing Scientist,
Numerical models and Remote Sensing,
Core Services, Swedish Meteorological and Hydrological Institute (SMHI)
www.pytroll.org
nwcsaf.smhi.se
www.smhi.se

[timo....@fmi.fi]

Hi all,

I have also some issues with the RT-STPS. First everything worked
nicely but suddenly the software started to create incorrect level 0
VIIRS files. And the problem is ONLY with the VIIRS. In most of the
overpasses, the VIIRS file has wrong date, i.e. the epoch date. See
the list of the newest files:

RNSCA-
RVIRS_npp_d19580101_t0000000_e1334191_b00001_c20120412133536331000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120412_t1323478_e1334204_b00001_c20120412133536412000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120412_t1324100_e1334181_b00001_c20120412133536444000_nfts_drl.h5
RNSCA-
RVIRS_npp_d20120412_t2203220_e2214150_b00001_c20120412221551652000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120412_t2203234_e2214172_b00001_c20120412221551724000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120412_t2203325_e2214146_b00001_c20120412221551743000_nfts_drl.h5
RNSCA-
RVIRS_npp_d19580101_t0000000_e2358060_b00001_c20120412235839927000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120412_t2343498_e2358084_b00001_c20120412235839980000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120412_t2343471_e2358030_b00001_c20120412235839996000_nfts_drl.h5
RNSCA-
RVIRS_npp_d19580101_t0000000_e0138252_b00001_c20120413013919464000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120413_t0124126_e0138262_b00001_c20120413013919499000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120413_t0124231_e0138170_b00001_c20120413013919515000_nfts_drl.h5
RNSCA-
RVIRS_npp_d19580101_t0000000_e0317458_b00001_c20120413031834297000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120413_t0304300_e0317474_b00001_c20120413031834323000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120413_t0304261_e0317345_b00001_c20120413031834399000_nfts_drl.h5
RNSCA-
RVIRS_npp_d19580101_t0000000_e0456432_b00001_c20120413045706622000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120413_t0444034_e0456450_b00001_c20120413045706673000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120413_t0443553_e0456300_b00001_c20120413045706685000_nfts_drl.h5
RNSCA-
RVIRS_npp_d19580101_t0000000_e0635104_b00001_c20120413063556188000_nfts_drl.h5
RCRIS-
RNSCA_npp_d20120413_t0622590_e0635116_b00001_c20120413063556204000_nfts_drl.h5
RATMS-
RNSCA_npp_d20120413_t0622504_e0635024_b00001_c20120413063556246000_nfts_drl.h5

Does anyone have had similar problems or any ideas what might be
wrong?

Best Regards,
Timo Ryyppö,
Finnish Meteorological Institute

[Atkinson, Nigel]

We sometimes find that the first granule that comes out of CSPP has a 1958 date. We just ignore that granule. But I've not systematically checked the dates in the RDR files. I'll look out for it.

Nigel

[timo....@fmi.fi]

I have only one granule for the whole overpass and if the date is 1958
I lose all the data. But if you find a solution, please let me know.

-Timo

On Apr 13, 12:44 pm, "Atkinson, Nigel"