correct_troposphere

Alejandro Espin

unread,

Aug 5, 2020, 7:37:06 PM8/5/20

to MintPy

Dear

I have problems with the correct troposphere, don't download the ERA5 files

can you help me?

Thanks

******************** step - correct_troposphere ********************

Atmospheric correction using Weather Re-analysis dataset (PyAPS, Jolivet et al., 2011)
Weather Re-analysis dataset: ERA5
tropo_pyaps3.py -f timeseries.h5 --model ERA5 -g /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/geometryRadar.h5 -w ${WEATHER_DIR}
weather model: ERA5 - dry (hydrostatic) and wet delay
weather directory: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente
output tropospheric delay file: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/ERA5.h5
output corrected time-series file: timeseries_ERA5.h5
read dates/time info from file: timeseries.h5
time of cloest available product: 12:00 UTC
make directory: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/ERA5

------------------------------------------------------------------------------
downloading weather model data using PyAPS ...
number of grib files to download: 47
------------------------------------------------------------------------------

INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2
Downloading 1 of 47: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/ERA5/ERA5_S10_N10_W100_W80_20190103_12.grb
{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2019', 'month': '01', 'day': '03', 'time': '12:00', 'area': '10.00/-100.00/-10.00/-80.00'}
2020-08-05 18:19:19,159 INFO Welcome to the CDS
2020-08-05 18:19:19,160 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
WARNING: the 1 attampt to download failed, retry it.

INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2
Downloading 1 of 47: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/ERA5/ERA5_S10_N10_W100_W80_20190103_12.grb
{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2019', 'month': '01', 'day': '03', 'time': '12:00', 'area': '10.00/-100.00/-10.00/-80.00'}
2020-08-05 18:19:22,906 INFO Welcome to the CDS
2020-08-05 18:19:22,906 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
WARNING: the 2 attampt to download failed, retry it.

INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2
Downloading 1 of 47: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/ERA5/ERA5_S10_N10_W100_W80_20190103_12.grb
{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2019', 'month': '01', 'day': '03', 'time': '12:00', 'area': '10.00/-100.00/-10.00/-80.00'}
2020-08-05 18:19:29,347 INFO Welcome to the CDS
2020-08-05 18:19:29,347 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels

**************************************************
WARNING: downloading failed for 3 times, stop trying and continue.
**************************************************

open geometry file: geometryRadar.h5
reading height          data from file: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/geometryRadar.h5 ...
reading incidenceAngle data from file: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/geometryRadar.h5 ...
reading latitude        data from file: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/geometryRadar.h5 ...
reading longitude       data from file: /mnt/procesamiento/Procesamiento_Sentinel/Fernandina/Stack_Descendente/inputs/geometryRadar.h5 ...

------------------------------------------------------------------------------
calcualting absolute delay for each date using PyAPS (Jolivet et al., 2011; 2014) ...
number of grib files used: 0
Traceback (most recent call last):
File "/home/insar/tools/MintPy/mintpy/smallbaselineApp.py", line 1116, in <module>
    main()
File "/home/insar/tools/MintPy/mintpy/smallbaselineApp.py", line 1106, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
File "/home/insar/tools/MintPy/mintpy/smallbaselineApp.py", line 1056, in run
    self.run_tropospheric_delay_correction(sname)
File "/home/insar/tools/MintPy/mintpy/smallbaselineApp.py", line 791, in run_tropospheric_delay_correction
    tropo_pyaps3.main(iargs)
File "/home/insar/tools/MintPy/mintpy/tropo_pyaps3.py", line 700, in main
    calculate_delay_timeseries(inps)
File "/home/insar/tools/MintPy/mintpy/tropo_pyaps3.py", line 628, in calculate_delay_timeseries
    prog_bar = ptime.progressBar(maxValue=num_date)
File "/home/insar/tools/MintPy/mintpy/utils/ptime.py", line 357, in __init__
    self.reset()
File "/home/insar/tools/MintPy/mintpy/utils/ptime.py", line 362, in reset
    self.update_amount(0) # Build progress bar string
File "/home/insar/tools/MintPy/mintpy/utils/ptime.py", line 376, in update_amount
    percentDone = (diffFromMin / np.float(self.span)) * 100.0
ZeroDivisionError: float division by zero

Emre Havazlı

unread,

Aug 6, 2020, 3:48:05 PM8/6/20

to MintPy

Hi Alejo,

Please make sure you can access ERA5 data using the link printed out in the message.

Have you created your account and followed the guidelines to activate data download?

Best

Emre

Message has been deleted

Emre Havazlı

unread,

Aug 6, 2020, 4:24:40 PM8/6/20

to Alejandro Espin, MintPy

Hi Alejandro,

You may want to not share your username and password just in case.

Also make sure you follow the notebook (https://github.com/yunjunz/pyaps3/blob/master/examples/TestECMWF.ipynb)

there may be some hidden bells and whistles you need to make sure to access data.

Best

Emre

On Thu, Aug 6, 2020 at 1:16 PM Alejandro Espin <alejo.es...@hotmail.com> wrote:

Thanks

I created an account in https://cds.climate.copernicus.eu/api-how-to#install-the-cds-api-key and put into de model.cfg

# vim: set filetype=cfg:
#####key for ECMWF in Climate Data Store Application Program Interface
#Get it from https://retostauffer.org/code/Download-ERA5/
[CDS]
key = 53997:18618458-dfa0-443e-929e-bc61123f95

but It doesn't work.

Thanks for your help

Greetings

Alejandro

--
You received this message because you are subscribed to a topic in the Google Groups "MintPy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mintpy/Wm5ULbfacyI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mintpy+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mintpy/69cf5c5c-9c34-4ed0-8d4e-02eb577cbaf5n%40googlegroups.com.

--

Dr. Emre HAVAZLI

Signal Analysis Engineer

Radar Algorithms and Processing Group(334D)

NASA - Jet Propulsion Laboratory

Pasadena, CA, USA

Alejandro Espin

unread,

Aug 6, 2020, 4:29:13 PM8/6/20

to MintPy

I didn't realize
I'll check the web again

Thank you

Alejandro

francesco menniti

unread,

Dec 31, 2020, 6:36:04 AM12/31/20

to MintPy

Hi alejio,

I have the same problem as you.

Did you understand how to solve?

would you give me a hand?

thank you so much

FM

Russell Grew

unread,

Dec 31, 2020, 3:29:03 PM12/31/20

to MintPy

Hi all,

You need to accept the terms of use for the data set.

You need to run this example pyaps3/examples/TestECMWF.ipynb per the post above. I ran it from the shell using IPython. When it fails it gives you the link you need to visit to accept the terms. I did try and find this link through navigating their website but was not able to. It might be https://apps.ecmwf.int/datasets/data/interim-full-daily/licence/ and so maybe you need to register for that and CDS per the PyAps config.

Good luck.

francesco menniti

unread,

Jan 11, 2021, 4:37:09 PM1/11/21

to MintPy

thanks now it works! I thought I had already accepted.

Erik Rivas

unread,

Mar 3, 2021, 7:07:50 PM3/3/21

to MintPy

Dear all,

May I ask you how was this problem solved?

I already accepted the terms and conditions, and I could retrieve data following these examples:

https://cds.climate.copernicus.eu/api-how-to

However, when I try with "examples/TestECMWF.ipynb", it throws an error (attach file)

Seems like it doesn't recognize my model.cfg or it's simply not reading it

is there a particular directory where this file has to be place?

Thanks in advance

Erik

log.txt

Kurt Feigl

unread,

Apr 22, 2021, 4:00:51 PM4/22/21

to MintPy

Hi Erik

You need to have a file named model.cfg with key information in it. Here is where it is on my setup

$HOME/PyAPS/pyaps3/model.cfg

Kurt

Kurt Feigl

unread,

Apr 22, 2021, 4:18:35 PM4/22/21

to MintPy

Hi All,

I believe that we have the keys configured properly in the model.cfg file. To test, we use the following code snippet from the test notebook in examples/TestECMWF.ipynb"

>>> import os

>>> import sys

>>> import numpy as np

>>> import pyaps3 as pa

>>> pa.ECMWFdload(['20140526','20130426'],'12','./ECMWF', model='ERA5', snwe=(25,35,125,135))

Then we receive the following message:

2021-04-22 19:43:22,871 INFO Welcome to the CDS

2021-04-22 19:43:22,872 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels

2021-04-22 19:43:25,148 INFO Request is queued

After more than 30 minutes, a file is finally downloaded:

2021-04-22 20:09:49,867 INFO Request is completed

2021-04-22 20:09:49,867 INFO Downloading https://download-0003.copernicus-climate.eu/cache-compute-0003/cache/data1/adaptor.mars.internal-1619122182.7975178-1134-11-8eba1727-8ca7-4770-82ab-b41ba071a105.grib to ./ECMWF/ERA-5_20140526_12.grb (377.2K)

This delay seems long. We checked the web site: https://cds.climate.copernicus.eu/#!/home and saw:

" CDS is having technical problems. Apologies for the inconvenience this may cause. For more information please contact user support. "

Do any of you know more? Are these technical problems usual or unusual?

Thanks!

Kurt

Zhang Yunjun

unread,

Apr 22, 2021, 5:44:21 PM4/22/21

to MintPy

Hi Kurt,

Most of my ERA-5 downloading happens in <1 min, but technical problems from CDS are not uncommon.

Setting up the $WEATHER_DIR will avoid re-downloading and help one to rely less on the real-time operating status of CDS, if you have not done so.

Yunjun

Russell Grew

unread,

Apr 22, 2021, 6:03:59 PM4/22/21

to MintPy

The ECMWF did recently email about some data corruption and perhaps there has been more load since, with people refreshing their archives

https://confluence.ecmwf.int/display/CKB/ERA5+CDS%3A+Data+corruption

I too noticed it was sluggish.

+1 for the $WEATHER_DIR suggestion, and doing the same for Sentinel orbit data.

Reply all

Reply to author

Forward