downloading weather model data using PyAPS fails

214 views
Skip to first unread message

Amin tavakkoli

unread,
Oct 31, 2022, 6:29:25 PM10/31/22
to MintPy

Dear All, 
I am new to Mintpy. I followed the instruction, provided on their website and created an account on the CDS website. Then, I created a file named .cdsapirc and added my API info to that.
To familiarize myself with this software I am using the "WellsEnvD2T399" training data set . It, however, returns the following error when I run "smallbaselineApp.py WellsEnvD2T399.txt" command.  I would really appreciate it if you can help me with this issue.

Best,
Amin 


INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2

Downloading 1 of 8: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/ERA5/ERA5_N30_N50_W120_W110_20070709_18.grb 

{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2007', 'month': '07', 'day': '09', 'time': '18:00', 'area': '50.00/-120.00/30.00/-110.00'}

2022-10-31 17:05:23,878 INFO Welcome to the CDS

2022-10-31 17:05:23,878 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels



**************************************************

WARNING: downloading failed for 3 times, stop trying and continue.

**************************************************



update mode: ON

output file: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/inputs/ERA5.h5

1) output file either do NOT exist or is NOT newer than all GRIB files.

run or skip: run

open geometry file: geometryRadar.h5

reading incidenceAngle  data from file: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/inputs/geometryRadar.h5 ...

reading height          data from file: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/inputs/geometryRadar.h5 ...

--------------------------------------------------

create HDF5 file: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/inputs/ERA5.h5 with w mode

create dataset  : date       of |S1                       in size of (0,)                 with compression = None

create dataset  : timeseries of <class 'numpy.float32'>   in size of (0, 508, 514)        with compression = None

close  HDF5 file: /Users/atavakk3/Downloads/WellsEnvD2T399/mintpy/inputs/ERA5.h5


------------------------------------------------------------------------------

calculating absolute delay for each date using PyAPS (Jolivet et al., 2011; 2014) ...

number of grib files used: 0

Traceback (most recent call last):

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/bin/smallbaselineApp.py", line 10, in <module>

    sys.exit(main())

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 1291, in main

    app.run(steps=inps.runSteps)

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 1071, in run

    self.run_tropospheric_delay_correction(sname)

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/smallbaselineApp.py", line 823, in run_tropospheric_delay_correction

    tropo_pyaps3.main(iargs)

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/tropo_pyaps3.py", line 849, in main

    calc_delay_timeseries(inps)

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/tropo_pyaps3.py", line 761, in calc_delay_timeseries

    prog_bar = ptime.progressBar(maxValue=num_date, print_msg=~inps.verbose)

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/objects/progress.py", line 52, in __init__

    self.reset()

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/objects/progress.py", line 57, in reset

    self.update_amount(0)  # Build progress bar string

  File "/Users/atavakk3/opt/anaconda3/envs/mint_py2/lib/python3.8/site-packages/mintpy/objects/progress.py", line 73, in update_amount

    percentDone = (diffFromMin / np.float(self.span)) * 100.0

ZeroDivisionError: float division by zero

Eric Fielding

unread,
Oct 31, 2022, 9:40:37 PM10/31/22
to MintPy
I don't know what problem it had with the downloading step, but it is reporting your ERA5.h5 dataset has zero (0) elements. It is possible that the Copernicus ECMWF move situation (https://confluence.ecmwf.int/display/BONDUSER/DHS+move+-+MARS+and+ECFS) is affecting the download, so I would try again tomorrow.

It can't divide by that zero elements in the pyAPS run, so that is what caused the final error.

Manudeo Singh

unread,
Nov 1, 2022, 3:51:48 AM11/1/22
to min...@googlegroups.com
Hi Amin,

In addition to what Eric has said, many times it is a licence issue for new users. Go to this link provided below and accept the license and then try downloading again - 

Remember to login to CDS before accessing this page. 

Best,
Manu

--
You received this message because you are subscribed to the Google Groups "MintPy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mintpy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mintpy/9faca853-b2bd-4d0d-8a7d-855f3f6ad83cn%40googlegroups.com.
--
_________________________________
Dr. Manudeo Singh
Alexander von Humboldt Postdoctoral Fellow
Universität Potsdam
University of Potsdam
Institut für Geowissenschaften
Institute of Geosciences
Potsdam, Germany

Amin tavakkoli

unread,
Nov 1, 2022, 3:46:02 PM11/1/22
to MintPy
Dear Eric and Manu,

Thank you for your kind replies.  
Manu you are right, I had forgotten to accept this license. Thanks for providing the link. Right now it is working

Best,
Amin

Esubalew Adem

unread,
Nov 2, 2022, 6:21:13 AM11/2/22
to MintPy
Dear Eric,

Can we use ERA5.h5 dataset from staged_data  ( "SanFranSenDT42.tar.gz ")  that we used during  2022 Short Course InSAR workshop  

Thank you
Best wish
Esubalew

Eric Fielding

unread,
Nov 2, 2022, 9:17:39 PM11/2/22
to MintPy
Dear Esubalew,

You can use the ERA5.h5 dataset from the staged dataset to perform the tropospheric corrections for that same San Francisco Sentinel-1 track 42 data, but it only has the ERA5 data for those dates and that area, not for any other locations.

++Eric

Esubalew Adem

unread,
Nov 9, 2022, 12:48:04 PM11/9/22
to MintPy
Dear Eric,

Thank you for your replay

I just downloaded ERA5*.grb  file by using  test_dload.py for my study region, How can I convert or create to ERA5.H5 format 

 ERA5.grb to ERA5.H5 


Thank you

Esubalew

Adnan Siddique

unread,
Dec 16, 2024, 2:18:47 AM12/16/24
to MintPy
Hi

Thanks. For me, it was also the license issue, but the link you shared did not work for me. Instead, I accessed the license agreement on the following page, and accepted it.

https://cds.climate.copernicus.eu/datasets/reanalysis-era5-pressure-levels?tab=download#manage-licences

Before calling the mintpy routine "correct-troposphere", I removed the existing invalid ERA5.h5, and all worked fine.

Regards
Adnan

Kurt Feigl

unread,
May 21, 2025, 4:28:08 PMMay 21
to MintPy
I am running mintpy on a stack produced by hyp3.prepare_insar_isce_burst_job.
It works fine when I set:
   mintpy.troposphericDelay.method = no 
  mintpy.troposphericDelay.method = height_correlation
but fails when I set
     mintpy.troposphericDelay.method = pyaps
The failure comes when running tropo_pyaps3.py because the download fails after making three attempts. 
Yet the following command confirms that the downloading works fine:
   python $HOME/PyAPS/tests/test_dload.py
Does anyone have any suggestions?



Suat COŞKUN

unread,
May 21, 2025, 4:40:51 PMMay 21
to min...@googlegroups.com
Have you try to install  development version? Because I think something went wrong step pyaps on realesed version. Another option you can try docker. This two thing work for me properly.
From: 'Kurt Feigl' via MintPy <min...@googlegroups.com>
Sent: Wednesday, May 21, 2025 11:28:08 PM
To: MintPy <min...@googlegroups.com>
Subject: Re: [mintpy] Re: downloading weather model data using PyAPS fails
 

Kurt Feigl

unread,
May 21, 2025, 6:43:03 PMMay 21
to MintPy
Yes. I used the development version by cloning the repositories for both MintPy and PyAPS.

mamba deactivate
mamba create -n mintpykf  -y
mamba activate mintpykf
# get standard version
mamba install mintpy -y
# overwrite with local version
cd $HOME/MintPy
git status
mamba install -y --file $HOME/MintPy/requirements.txt
which smallbaselineApp.py

# add PyAPS
cd $HOME/PyAPS
git pull
mamba install -y --file $HOME/PyAPS/requirements.txt --file $HOME/PyAPS/tests/requirements.txt

Is it possible that pyaps is getting confused because the hyp3 products are geocoded to UTM coordinates? I note that (lon, lat) and (X,Y) fields show UTM (easting, northing) in meters:

data/S1_135547_IW2_20170109_20170202_VV_INT40_7F65/S1_135547_IW2_20170109_20170202_VV_INT40_7F65_unw_phase_clipped.tif.rsc

ALOOKS                   2
ANTENNA_SIDE             -1
AZIMUTH_PIXEL_SIZE       28.2
BANDS                    1
CENTER_LINE_UTC          6642.590691
DATA_TYPE                float32
DATE12                   170109-170202
EARTH_RADIUS             6337286.638938101
EPSG                     32611
FILE_LENGTH              176
HEADING                  -13.388586868505683
HEIGHT                   693000.0
INTERLEAVE               BSQ
LAT_REF1                 4297400.0
LAT_REF2                 4297400.0
LAT_REF3                 4304440.0
LAT_REF4                 4304440.0
LENGTH                   176
LON_REF1                 381880.0
LON_REF2                 393800.0
LON_REF3                 381880.0
LON_REF4                 393800.0
NO_DATA_VALUE            0.0
NoDataValue              0.0
ORBIT_DIRECTION          ASCENDING
PLATFORM                 Sen
PROCESSOR                hyp3
P_BASELINE_BOTTOM_HDR    -7.871673723677282
P_BASELINE_TOP_HDR       -7.871673723677282
RANGE_PIXEL_SIZE         23.0
RLOOKS                   10
STARTING_RANGE           846227.8756538476
UTM_ZONE                 11N
WAVELENGTH               0.055465764662349676
WIDTH                    298
X_FIRST                  381880.0
X_STEP                   40.0
X_UNIT                   meters
Y_FIRST                  4304440.0
Y_STEP                   -40.0
Y_UNIT                   meters
beam_mode                IW
beam_swath               2
unwrap_method            snaphu_mcf

Yuan-Kai Liu

unread,
Jun 7, 2025, 12:58:16 PMJun 7
to MintPy
Hi Kurt, I have never used products from ASF Hyp3 in pyaps/mintpy. But I wonder what is the error message when it fails when you set
     mintpy.troposphericDelay.method = pyaps

Maybe we could have some clue and trace back the bug~

--Kai

Zhang Yunjun

unread,
Jun 10, 2025, 10:50:59 AMJun 10
to MintPy
Hi Kurt,

There are a few recent updates on both mintpy and pyaps. Please ensure to use the latest version of them. The installation note you posted shows you installed the latest dependency but did not install the latest mintpy/pyaps3 code themselves.

mintpy supports the UTM coordinates of HyP3 for ERA5.

And I seond Kai that the full error message with trace bug information would be very helpful for diagnosis.

Yunjun

Kurt Feigl

unread,
Jun 10, 2025, 8:03:41 PMJun 10
to MintPy
Hi Yunjun,

Thanks for your help. I have re-installed the standard released version with the following command

mamba create -n mintpy -c conda-forge mintpy

Now the pyaps routines work fine. Indeed, the GRB files now download smoothly. And the results are properly geocoded in UTM. And the results using pyaps look better than the ones from height_correlation. I attribute the previous errors to my clumsy attempts at setting up environments to run using a juypter notebook. Thanks again!

Kurt


From: min...@googlegroups.com <min...@googlegroups.com> on behalf of Zhang Yunjun <yunju...@gmail.com>
Sent: Tuesday, June 10, 2025 9:50 AM
--
You received this message because you are subscribed to a topic in the Google Groups "MintPy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mintpy/T2lyRIlYyAI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mintpy+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mintpy/8e03f4d1-1f15-4853-8628-f80581d3de22n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages