PyAPS - Wrong date

573 views
Skip to first unread message

Matt Cook

unread,
May 30, 2021, 11:19:07 PM5/30/21
to MintPy
Hi everyone,

Sorry if this error has been addressed before but i could not find a past conversation with this type of error. 

I am getting an error during the correct troposphere step. I am using the Pyaps method and trying to download ERA5 weather model data. I have set up the model.cfg file to include my CDS upi and key. But when I try to run it, for some reason it is trying to download the wrong date. For example in the error below the grib file that I want to download is ERA5_S50_S30_E170_E190_20170227_17.grb - but it recognises it as year 2019, month 0 and day 04? 

Has anyone come across an issue like this before and can help me figure out where I am going wrong. 

(mintpy) mcoo951@sc-cer00279:~/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy$ smallbaselineApp.py input.txt --dostep correct_troposphere
MintPy release version v1.3.0-34, release date 2021-05-22
--RUN-at-2021-05-31 12:16:54.957237--
Current directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy
Run routine processing with smallbaselineApp.py on steps: ['correct_troposphere']
Remaining steps: ['deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Project name: input
Go to work directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy
read custom template file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/input.txt
update default template based on input custom template
No new option value found, skip updating /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/smallbaselineApp.cfg
read default template file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/smallbaselineApp.cfg


******************** step - correct_troposphere ********************
Atmospheric correction using Weather Re-analysis dataset (PyAPS, Jolivet et al., 2011)
Weather Re-analysis dataset: ERA5

tropo_pyaps3.py -f /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5 --model ERA5 -g /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 -w /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/WEATHER_DIR
weather model: ERA5 - dry (hydrostatic) and wet delay
weather directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/WEATHER_DIR
output tropospheric delay file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5
output corrected time-series file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5
read dates/time info from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5
time of cloest available product: 17:00 UTC
make directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/WEATHER_DIR/ERA5

------------------------------------------------------------------------------
downloading weather model data using PyAPS ...
number of grib files to download: 105
------------------------------------------------------------------------------

INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2
Downloading 1 of 105: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/WEATHER_DIR/ERA5/ERA5_S50_S30_E170_E190_20170227_17.grb 
{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2019', 'month': '00', 'day': '04', 'time': '17:00', 'area': '-30.00/170.00/-50.00/190.00'}
2021-05-31 12:16:56,938 INFO Welcome to the CDS
2021-05-31 12:16:57,221 INFO Request is queued
2021-05-31 12:16:58,498 INFO Request is failed
2021-05-31 12:16:58,498 ERROR Message: an internal error occurred processing your request
2021-05-31 12:16:58,498 ERROR Reason:  internal error
2021-05-31 12:16:58,498 ERROR   Traceback (most recent call last):
2021-05-31 12:16:58,498 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 160, in _consume
2021-05-31 12:16:58,498 ERROR       result = handle_locally()
2021-05-31 12:16:58,498 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/dispatcher.py", line 207, in <lambda>
2021-05-31 12:16:58,498 ERROR       lambda: self.handle_request(context, request_data),
2021-05-31 12:16:58,499 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/mappingprocessor.py", line 50, in handle_request
2021-05-31 12:16:58,499 ERROR       return self.mapping_engine.run(cdsinf, request_data)
2021-05-31 12:16:58,499 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 312, in run
2021-05-31 12:16:58,499 ERROR       request['specific'] = apply_mapping(specific, mapping)
2021-05-31 12:16:58,499 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 257, in apply_mapping
2021-05-31 12:16:58,499 ERROR       options.get("date_format", "%Y-%m-%d"))
2021-05-31 12:16:58,499 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/mapping.py", line 186, in expand_dates
2021-05-31 12:16:58,499 ERROR       raise Exception("No valid dates from year=%s month=%s day=%s" % (years, months, days))
2021-05-31 12:16:58,499 ERROR   Exception: No valid dates from year=[2019] month=[0] day=[4]
WARNING: the 1 attampt to download failed, retry it.


Thanks,

Matt

input.txt

Zhang Yunjun

unread,
Jun 3, 2021, 1:47:42 AM6/3/21
to MintPy
Hi Matt,

I think the code misinterpret the date from your directory path "ressci201900043-insar-data", and it should be an easy fix as below:

date_list2dload = [str(re.findall('\d{8}', i)[0]) for i in grib_files2dload]
to
date_list2dload = [str(re.findall('\d{8}', os.path.basename(i))[0]) for i in grib_files2dload]

Could you confirm? Then I will commit this fix next time.

Yunjun

Matt Cook

unread,
Jun 3, 2021, 4:14:04 AM6/3/21
to MintPy
Hi Yunjun,

Yes its working once I altered that line of code. Thank you so much for your help!

Matt Cook

unread,
Jun 7, 2021, 2:16:35 AM6/7/21
to MintPy
Hi Yunjun,

Sorry to bother you again, but I have another issue running the correct_troposphere step. 

I am able to download the grib file successfully but it still fails doing the correct_troposhphere stage because it does not identify some of the dates. Here is the error:


(mintpy) mcoo951@sc-cer00279:~/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy$ smallbaselineApp.py input.txt --dostep correct_troposphere
MintPy release version v1.3.0-34, release date 2021-05-22
--RUN-at-2021-06-04 11:40:30.689241--
Current directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy
Run routine processing with smallbaselineApp.py on steps: ['correct_troposphere']
Remaining steps: ['deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']
--------------------------------------------------
Project name: input
Go to work directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy
read custom template file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/input.txt
update default template based on input custom template
No new option value found, skip updating /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/smallbaselineApp.cfg
read default template file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/smallbaselineApp.cfg


******************** step - correct_troposphere ********************
Atmospheric correction using Weather Re-analysis dataset (PyAPS, Jolivet et al., 2011)
Weather Re-analysis dataset: ERA5

tropo_pyaps3.py -f /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5 --model ERA5 -g /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 -w ${WEATHER_DIR}
weather model: ERA5 - dry (hydrostatic) and wet delay
weather directory: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy
output tropospheric delay file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5
output corrected time-series file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5
read dates/time info from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5
time of cloest available product: 17:00 UTC

------------------------------------------------------------------------------
downloading weather model data using PyAPS ...
common file size: 1478520 bytes
number of grib files existed    : 105
number of grib files to download: 0
------------------------------------------------------------------------------

update mode: ON
output file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5
1) output file either do NOT exist or is NOT newer than all GRIB files.
run or skip: run
open geometry file: geometryRadar.h5
reading incidenceAngle  data from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 ...
reading height          data from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 ...
reading latitude        data from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 ...
reading longitude       data from file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/geometryRadar.h5 ...
--------------------------------------------------
create HDF5 file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5 with w mode
create dataset  : date       of |S8                       in size of (105,)               with compression = None
create dataset  : timeseries of <class 'numpy.float32'>   in size of (105, 437, 644)      with compression = None
close  HDF5 file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5

------------------------------------------------------------------------------
calculating absolute delay for each date using PyAPS (Jolivet et al., 2011; 2014) ...
number of grib files used: 105
[==================================================] ERA5_S50_S30_E170_E190_20200810_17.grb  144s /     2s

------------------------------------------------------------------------------
correcting relative delay for input time-series using diff.py
diff.py /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5 /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5 -o /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5 --force
/home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5 - ['/home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5'] --> /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5
input files are: timeseries and timeseries
WARNING: ['/home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5'] does not contain all dates in /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5
Continue and enforce the differencing for their shared dates only.
with following dates are ignored for differencing:
['20181231', '20190112', '20200506', '20200810', '20190301', '20190605', '20170603', '20180821', '20180210', '20180902', '20181207', '20181125', '20191027', '20200424', '20200107', '20200717', '20170721', '20180926', '20190313', '20180610', '20191108', '20171118', '20190617', '20181008', '20180423', '20191120', '20191226', '20170510', '20190325', '20180318', '20171013', '20190828', '20200319', '20170522', '20180914', '20190723', '20190711', '20180222', '20180517', '20170227', '20200705', '20180622', '20180411', '20170323', '20191202', '20200611', '20170428', '20180306', '20200729', '20200131', '20190418', '20170814', '20200212', '20181219', '20170627', '20191003', '20190430', '20170907', '20171130', '20170615', '20180330', '20190629', '20200530', '20191015', '20170416', '20200331', '20180728', '20180809', '20181101', '20200119', '20181113', '20190804', '20180716', '20170311', '20200412', '20171106', '20200307', '20200518', '20180105', '20190524', '20171212', '20200623', '20190512', '20170709', '20170826', '20180117', '20170919', '20191214', '20200224', '20190124', '20171224', '20181020', '20190909', '20190921', '20180704', '20180529', '20180129', '20171025', '20190406', '20170404', '20190205', '20190217', '20180505', '20190816', '20170802']
--------------------------------------------------
grab metadata from ref_file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5
grab dataset structure from ref_file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries.h5
create HDF5 file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5 with w mode
create dataset  : bperp      of float32                   in size of (105,)               with compression = None
create dataset  : date       of |S8                       in size of (105,)               with compression = None
create dataset  : timeseries of float32                   in size of (105, 437, 644)      with compression = None
close  HDF5 file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/timeseries_ERA5.h5
Traceback (most recent call last):
  File "/data/MintPy/MintPy/mintpy/smallbaselineApp.py", line 1255, in <module>
    main(sys.argv[1:])
  File "/data/MintPy/MintPy/mintpy/smallbaselineApp.py", line 1237, in main
    app.run(steps=inps.runSteps)
  File "/data/MintPy/MintPy/mintpy/smallbaselineApp.py", line 1053, in run
    self.run_tropospheric_delay_correction(sname)
  File "/data/MintPy/MintPy/mintpy/smallbaselineApp.py", line 809, in run_tropospheric_delay_correction
    tropo_pyaps3.main(iargs)
  File "/data/MintPy/MintPy/mintpy/tropo_pyaps3.py", line 791, in main
    correct_timeseries(dis_file=inps.dis_file,
  File "/data/MintPy/MintPy/mintpy/tropo_pyaps3.py", line 743, in correct_timeseries
    diff.main(iargs)
  File "/data/MintPy/MintPy/mintpy/diff.py", line 265, in main
    inps.outfile = diff_file(inps.file1, inps.file2, inps.outfile, force=inps.force)
  File "/data/MintPy/MintPy/mintpy/diff.py", line 167, in diff_file
    ref_val = readfile.read(file2[0],
  File "/data/MintPy/MintPy/mintpy/utils/readfile.py", line 209, in read
    dsname4atr = datasetName[0].split('-')[0]
IndexError: list index out of range



I noticed that when I looked at the ERA5.h5 file I noticed that it also read the dates wrong and I get the following error: 

(mintpy) mcoo951@sc-cer00279:~/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs$ view.py ERA5.h5 
run view.py in MintPy release version v1.3.0-34, release date 2021-05-22
input file is timeseries file: /home/mcoo951/ressci201900043-insar-data/isce/Projects/mintpy/Richardson/mintpy/inputs/ERA5.h5 in float32 format
file size in y/x: (437, 644)
Traceback (most recent call last):
  File "/data/MintPy/MintPy/mintpy/view.py", line 1563, in <module>
    main(sys.argv[1:])
  File "/data/MintPy/MintPy/mintpy/view.py", line 1555, in main
    obj.configure()
  File "/data/MintPy/MintPy/mintpy/view.py", line 1450, in configure
    inps, self.atr = read_input_file_info(inps)
  File "/data/MintPy/MintPy/mintpy/view.py", line 744, in read_input_file_info
    inps.sliceList = readfile.get_slice_list(inps.file)
  File "/data/MintPy/MintPy/mintpy/utils/readfile.py", line 562, in get_slice_list
    obj.open(print_msg=False)
  File "/data/MintPy/MintPy/mintpy/objects/stack.py", line 182, in open
    self.times = np.array([dt.strptime(i, self.dateFormat) for i in self.dateList])
  File "/data/MintPy/MintPy/mintpy/objects/stack.py", line 182, in <listcomp>
    self.times = np.array([dt.strptime(i, self.dateFormat) for i in self.dateList])
  File "/data/anaconda3/envs/mintpy/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/data/anaconda3/envs/mintpy/lib/python3.8/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '20190004' does not match format '%Y%m%d'

Is there another line of code I can change in one of the scripts to make sure the ERA5 is referencing the correct dates? Pleas let me know if there is any other information I can send to help with this.

Kind regards,

Matt

Zhang Yunjun

unread,
Jun 8, 2021, 12:11:57 AM6/8/21
to MintPy
Hi Matt,

It seems like the ERA5.h5 file is not properly written, due to the similar issue of date grabbing from grib file names. I have searched and change all occurrences of this grabbing part in tropo_pyaps3.py (https://github.com/insarlab/MintPy/pull/571), but I could not test it since I can not reproduce the error you showed above. Try it (update the code and re-run) and let me know pls.

Yunjun

Matt Cook

unread,
Jun 9, 2021, 1:19:13 AM6/9/21
to MintPy
Hi Yunjun,

I made the changes to the code in tropo_pyaps3.py as you did and it works fine now. 

Thank you so much for your help! 

Matt

Zhang Yunjun

unread,
Jun 9, 2021, 1:42:05 AM6/9/21
to MintPy
Great. Thank you for bug reporting.

Yunjun
Reply all
Reply to author
Forward
0 new messages