Error when running `load_data` -- KeyError: 'STARTING_RANGE'

[Trường Nguyễn]

Dear everyone,

I have encountered an error that I am unable to resolve after searching on Google (perhaps my case is quite new). I hope someone here can help me with this issue.

When I run the command smallbaselineApp.py <my_custom_template> --dostep load_data, I receive the following error. I have attached the template file below to help others understand my input.

I also tried running prep_isce.py, but I encountered errors as well. I have attached the log file for prep_isce.py below.

I am wondering which steps I should re-run to ensure that the MintPy process runs normally. Steps 5 and 9 are very time-consuming; I have 299 interferograms, so I hope I will not have to re-run the entire program:

MintPy version 1.6.0, date 2024-05-09                                                                                                                                                              
--RUN-at-2024-08-07 16:25:53.330901--                                                                                      
Current directory: /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4
Run routine processing with smallbaselineApp.py on steps: ['load_data']                                                    
Remaining steps: ['modify_network', 'reference_point', 'quick_overview', 'correct_unwrap_error', 'invert_network', 'correct_LOD', 'correct_SET', 'correct_ionosphere', 'correct_troposphere', 'deramp', 'correct_topography', 'residual_RMS', 'reference_date', 'velocity', 'geocode', 'google_earth', 'hdfeos5']                        
--------------------------------------------------                        
Project name: Choushui_TSX_Desc_v3_4                                                                                                              
Go to work directory: /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4
read custom template file: /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/Choushui_TSX_Desc_v3_4.cfg
update default template based on input custom template                        
No new option value found, skip updating /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/smallbaselineApp.cfg
read default template file: /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/smallbaselineApp.cfg                                                                  


******************** step - load_data ********************

load_data.py --template /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/smallbaselineApp.cfg /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/Choushui_TSX_Desc_v3_4.cfg --project Choushui_TSX_Desc_v3_4
processor : isce
SAR platform/sensor : unknown from project name "Choushui_TSX_Desc_v3_4"
--------------------------------------------------
prepare metadata files for isce products
prep_isce.py -m /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/merged/SLC/20230319/referenceShelve/data.dat -g /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference -b /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/baselines/ -f "/media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/Igrams/*/filt*.unw" --geom-files hgt.rdr lat.rdr lon.rdr los.rdr los.rdr shadowMask.rdr waterMask.rdr
extract metadata from ISCE/stripmapStack shelve file: /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/merged/SLC/20230319/referenceShelve/data.dat
Using default ISCE Path: /home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/isce
extract metadata from geometry files: ['hgt.rdr', 'lat.rdr', 'lon.rdr', 'los.rdr']
/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/load_data.py:707: UserWarning: prep_isce.py failed. Assuming its result exists and continue...
  warnings.warn('prep_isce.py failed. Assuming its result exists and continue...')
--------------------------------------------------
updateMode : True
compression: None
multilook x/ystep: 1/1
multilook method : nearest
--------------------------------------------------
searching geometry files info
input data files:
height          : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/hgt.rdr
latitude        : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/lat.rdr
longitude       : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/lon.rdr
incidenceAngle  : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/los.rdr
azimuthAngle    : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/los.rdr
waterMask       : /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/ISCE_output/geom_reference/waterMask.rdr
/media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/inputs/geometryRadar.h5 exists, but can not read, remove it.
--------------------------------------------------
create HDF5 file /media/davidncu/VTRUONG_1TB/JupyterLab_Workspace/TSX_Processing/Run003/MintPy_v4/inputs/geometryRadar.h5 with w mode
create dataset /height             of <class 'numpy.float32'>   in size of (2945, 1672) with compression = lzf
create dataset /latitude           of <class 'numpy.float32'>   in size of (2945, 1672) with compression = lzf
create dataset /longitude          of <class 'numpy.float32'>   in size of (2945, 1672) with compression = lzf
create dataset /incidenceAngle     of <class 'numpy.float32'>   in size of (2945, 1672) with compression = lzf
create dataset /azimuthAngle       of <class 'numpy.float32'>   in size of (2945, 1672) with compression = lzf
create dataset /waterMask          of <class 'numpy.bool_'>     in size of (2945, 1672) with compression = lzf
prepare slantRangeDistance ...
Traceback (most recent call last):
  File "/home/davidncu/miniconda3/envs/isce_project/bin/smallbaselineApp.py", line 8, in <module>
    sys.exit(main())
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/cli/smallbaselineApp.py", line 209, in main
    run_smallbaselineApp(inps)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 1155, in run_smallbaselineApp
    app.run(steps=inps.runSteps)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 908, in run
    self.run_load_data(sname)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 176, in run_load_data
    mintpy.cli.load_data.main(iargs)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/cli/load_data.py", line 123, in main
    load_data(inps)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/load_data.py", line 849, in load_data
    geom_radar_obj.write2hdf5(
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/objects/stackDict.py", line 761, in write2hdf5
    data = self.get_slant_range_distance(box=box, xstep=xstep, ystep=ystep)
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/objects/stackDict.py", line 504, in get_slant_range_distance
    data = ut.range_distance(self.extraMetadata,
  File "/home/davidncu/miniconda3/envs/isce_project/lib/python3.8/site-packages/mintpy/utils/utils0.py", line 78, in range_distance
    range_n, dR = float(atr['STARTING_RANGE']), float(atr['RANGE_PIXEL_SIZE'])
KeyError: 'STARTING_RANGE'

[Eric Fielding]

That kind of error is usually caused by having a different version of ISCE2 in your mintpy environment than the version that you used for processing the stack. I follow the recommendation of Yunjun to build a single "insar" environment that has both ISCE2 and MintPy together so that they have the compatible versions. I have not tried running stripmapStack on TerraSAR-X data. Note that you can run the `prep_isce.py` program separately and it may give you more information on why it can't get the "STARTING_RANGE".

[Trường Nguyễn]

Dear Eric,

Really thank you for your response!

I resolved the issue by identifying a malfunction in run_02_reference. (after checking the source code prep_isce.py)

The topo function didn’t perform multi-looking correctly, resulting in non-value geometric files in ./geom_reference, which is used for later MintPy processing (https://mintpy.readthedocs.io/en/latest/dir_structure/)

Then, I manually ran the topo and createWaterMask functions, which fixed the problem.

However, this could be problematic for users who run all the scripts automatically with large datasets. They might encounter errors at the end and not know what to do.