Tropospheric delay correction for PySAR

[Sven]

Good morning,

I need some guidance in setting up the weather model for tropospheric delay correction.

I never used pyaps before, so I downloaded some grib files (ERA-Int*.grb) for testing and it works.

Now I'm going to process a time-series so my question is: should I download a grib file for each acquisition date, regardless of the number of ifgs?

Likely, I don't think so, as each single file has a size of about 28 MB (so the total amount should be 28 MB x n dates), whereas your ECMWF file for the Alos example is only 22 MB.

How should I do? Should I merge them in some way?

These are grib files and, as far as I understand, they must be converted into HDF files (grb -> h5): is load_hdf5.py the right script for this? If so, could you please give me the complete command? (load_hdf5.py ... )

Last point: the volcanic area I'm going to study is close to the coast, so I have water in my scenes: how can i get the water mask? I saw you used that in the S-1 example.

Sorry for disturbance and have a nice day.

Sven.

[Yunjun Zhang]

Hi Sven,

To correct tropospheric delay with PyAPS, you just need to run tropcor_pyaps.py in PySAR (run "tropcor_pyaps.py -h" for help), it will call the functions in PYAPS to calculate the corresponding tropospheric delay from ERA-I grib files, and write it to HDF5 file, i.e. INPUTS/ECMWF.h5 you saw in the example dataset. 

The ERA-Int grib file covers the whole world with ~0.75 deg resolution, and PYAPS interpolate it into the same resolution as InSAR data. I usually have a fixed directory to store all the grib files I downloaded so that they can be shared among all my datasets, such as the example below.

tropcor_pyaps.py -f timeseries.h5 -g INPUTS/geometryRadar.h5 -m ECMWF -w /Users/yunjunz/insarlab/WEATHER/

Please update your pysar code as I commited a fix to make py3-pysar works py2-pyaps.

A final note to this topic: once your pyaps runs smoothly, you can setup your template file as below, and pysarApp.py will take care of everything I said above.

pysar.troposphericDelay.method = pyaps  #[pyaps / height_correlation / no], auto for pyaps
pysar.troposphericDelay.weatherModel = auto #[ERA / MERRA / NARR], auto for ECMWF, for pyaps method
pysar.troposphericDelay.weatherDir = /Users/yunjunz/insarlab/WEATHER/ #[path2directory], auto for "./../WEATHER"


Regarding to the water mask, I use generate_mask.py, below is the jupyter notebook example with a more detailed explanation:

https://github.com/yunjunz/PySAR/blob/master/examples/water_mask.ipynb

Cheers,

Yunjun

[Sven]

Hi Yunjun,

thanks for clarification; I just need to fix the ideas.

I understand the key point is to set properly the processing template so I'll ask you again for that in the next days.

In order to test the "tropcor_pyaps.py" script (after updating PySAR), I experienced an error with the simple command:

"tropcor_pyaps.py -d date_list.txt --hour 12 -m ECMWF" (date_list.txt has one column with 3 dates in the format YYYYMMDD) with the output:

AttributeError: 'NoneType' object has no attribute 'rfind'.

I don't understand if the error is related to some missing file/files I didn't pass to the script (apart from the text file) or something related to Python itself, as I'm not a Python expert.

I have both Python and Python3 properly working: how can I fix this?

I didn't test yet "generate_mask.py" for water mask, in case of problems I'll let you know.

Moreover, I downloaded the Fernandina example and also in this case I experienced some errors (see the attached file), related to the missing 'isce' module and a missing file.

I don't have Isce on my PC, as I use Gamma as a SAR processor, b.t.w. I have not even ROI_PAC, but the Alos example works smoothly till the end of processing, with no error messages.

Fixing also this would be fine, also for other PySAR users.

Cheers,

Sven.

[Yunjun Zhang]

Hi Sven,

There was a bug related with the error, it’s fixed now, please update the code and let me know if there is any error. I also add more comment to the usage of this script (tropcor_pyaps.py -h), as shown below:

example:
  # download reanalysys dataset, calculate tropospheric delays and correct time-series file.
  tropcor_pyaps.py -f timeseries.h5 -m ECMWF -g INPUTS/geometryRadar.h5

  # download reanalysys dataset, calculate tropospheric delays
  tropcor_pyaps.py -d date_list.txt     --hour 12 -m ECMWF -g INPUTS/geometryRadar.h5 --ref-yx 30 40
  tropcor_pyaps.py -d 20151002 20151003 --hour 12 -m MERRA -g INPUTS/geometryRadar.h5 --ref-yx 30 40

  # download reanalysys dataset

  tropcor_pyaps.py -d date_list.txt     --hour 12 -m ECMWF

I have add another file (ISCE/master/data.rsc generated by prep_isce.py, which calls ISCE module) to the Fernandina dataset, you should be able to run it without ISCE now.

Cheers,

Yunjun

-- 
You received this message because you are subscribed to the Google Groups "PySAR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py-sar+un...@googlegroups.com.
To post to this group, send email to py-...@googlegroups.com.
Visit this group at https://groups.google.com/group/py-sar.
For more options, visit https://groups.google.com/d/optout.
<PySAR_Fernandina_processing.png>

[Sven]

Hi Yunjun,

the "tropcor_pyaps.py" script now works fine: I repeated the same test I did yesterday (tropcor_pyaps.py -d date_list.txt --hour 12 -m ECMWF) and I get the grib files listed in the "date_list.txt" file, though in a position different with respect to the one I would have expected.

For instance, if I create a PROJECT dir under $HOME with the "date_list.txt" inside (cd $HOME/PROJECT) I get the grib files under $HOME/WEATHER/ECMWF (newly created dirs).

The script does not end properly (No DEM/incidenceAngle/ref_yx found, skip calculating tropospheric delays ... No input timeseries file, skip correcting tropospheric delays) but of course this was just a downloading test, no files to work on ;)

With the newly inserted file, the Fernandina example woks fine till the end, thanks.

In the Google Earth web page of PySAR I noticed a very interesting *.kmz output (geo_timeseries...kmz) generated by the "generate_kmz_timeseries.py" script that does not seem to be included in PySAR: is it possible to download it from one of your web pages, or included in the next package update?

B.t.w. the only way for me to move on now is to set up the processing template and check what I get, then I will get again in touch with you: do you have an example template ready for Gamma? Of course I have to change the path/dir/file names accordingly, but this would be a good starting point.

Thanks and sorry for disturbing you.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

It’s great that tropcor_pyaps.py works for you now. I just adjust the default directory to be the current working directory so that your grib files will be $HOME/PROJECT/ECMWF by default. I recommend to always use the -w / --weather-dir option to pool your reanalysis data files in one place.

You can download generate_kmz_timeseries.py here (https://github.com/yunjunz/PySAR/blob/google_earth_kmz_timeseries/pysar/generate_kmz_timeseries.py). Note that it works but Joshua are still developing it and putting more functionalities. We will merge it to the default master branch soon.

I just uploaded an example template for GAMMA, which is very similar to the ISCE and ROIPAC template, except for the input file path in the very beginning.

https://github.com/yunjunz/PySAR/blob/master/docs/GalapagosEnvA2T061.txt

Here is an example file/directory structure for the stacking processing result from GAMMA, similar to what we discussed previously.

https://github.com/yunjunz/PySAR/blob/master/docs/Notebooks/docs/dir_gamma.ipynb

Cheers,

Yunjun

[Sven]

Hi Yunjun,

again some questions about the water mask.

As far as I understand, the water mask is a final product, to be produced after the whole time-series processing, right? I thought it was something to be included in the processing to get a complete result at the end.

So, how and when should I insert it into my processing?

About the Jupyter Notebook example from your web page, at the end I was able to reproduced it as I've installed ISCE on my PC, but something I didn't understand is how do you get the right value of the maximum height on the water from the "view.py script" and related figure.

Please correct me if I'm wrong but

1) the value should be something positive (height ON the water), your value is negative (-6 m);

2) which is the measurement unit of the colorbar on the right of your example? and why did you choose 6?

3) the output figure from my "view.py" script is reversed with respect to yours (attached file), though this should not affect the choice of the value, correct?

Just to inform you: one of the final PySAR updates doesn't bring the "save_kml" script, needed to successfully conclude the Fernandina example, which stops as PySAR is unable to generate a Google Earth *.kmz file (I retrieved "save_kml" from a previous installation), but you probably already fixed this ;)

Cheers,

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

The water mask is an optional auxiliary file, I usually use it in the pysar.networkInversion.waterMaskFile option to skip the processing on the water and give the result a "clean cut" (as shown in the Fernandina example). Then the water boundary information will be carried into the temporalCoherence.h5 and the corresponding maskTempCoh.h5, which is the reliability map of all final products. I don't use it by default because I don't find an easy and insar-processor-independent way to generate it.

From our experience, the threshold value on the water is not universal in DEM. In the case of Fernandina example, the DEM is from SRTM, which is originally referenced to the geoid, then converted into the ellipsoid by ISCE, thus you see slightly variable values around the scene. 

The unit is meter. 

When you plot the height data using view.py, you can see the current coordinates and data value information on the status bar at the bottom of your window. I just hover around the whole scene and get the max height value on water, and it's -6 in case of Fernandina dataset.

It's fine that the plot from view.py is reversed, it won't affect the choice. We reverse it on purpose for radar coded file automatically to make it looks more similar to the reality. 

Regarding the save_kml.py, I have renamed it to save_kmz.py and fixed the bug, thank you for letting me know. 

Cheers,

Yunjun

[Sven]

Hi Yunjun,

sorry again for disturbance.

I set the "pysar.networkInversion.waterMaskFile   = waterMask.h5" string into the template, though I don't understand how to interact with processing to get the proper "waterMask" file.

I mean, in your example on the PySAR web pages you show there are two options: "generate_mask.py" and "wbd.py" but in both cases you have to set up some input values (height on water and coordinates, respectively).

Does PySAR automatically set these values during processing? No interaction is needed?

And which one of the two methods is used? I ask just because I need to know if the PC I'm working on needs ISCE or not (2nd case).

B.t.w. I started processing a time-series but, as I would have expected, I got a couple of errors (see the attached file). Unfortunately, at this point, I'm not able to fix by myself the problems so I need your help.

Thanks and have a nice day.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

Regarding your questions about water mask. It’s not automatically generated, and none of the two methods shown in the notebook example are used in pysarApp.py. The water mask is not necessary for the processing workflow bypysarApp.py, as you can see that the Kuju example works fine without it. 

The water mask sometimes helps, and you need to manually generated.

Regarding the error, could you attached the following information for me to better judge the source?

NaplesRS2AF6N.txt       #the custom template, if you have

pysarApp_template.txt   #the default auto-generated template 

The error message since the beginning of the processing

Your DEM metadata file, i.e. sim_*.diff_par and/or sim_*.utm.dem.par

Yunjun

--
You received this message because you are subscribed to the Google Groups "PySAR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py-sar+un...@googlegroups.com.
To post to this group, send email to py-...@googlegroups.com.
Visit this group at https://groups.google.com/group/py-sar.
For more options, visit https://groups.google.com/d/optout.

<PySAR_processing.png>

[Sven]

Hi Yunjun,

sorry for late answer, but I was not at the office in these days.

I tried again a PySAR processing, but I still get an error message, as you can see from the attached file. More precisely, two errors, a "value error" about operands and a "file not found" error about a "geometry*.h5" file in the INPUTS folder, quite strange (?!?) because in INPUTS I get "geometryRadar.h5" + "ifgramStack.h5".

You will find also the template set by me (NaplesRS2AF6N.txt) and the one cretaed by PySAR (pysarApp_template.txt): the one set by me is a very basic one, following the Galapagos ENVISAT example.

I added and indication also for the waterMask in my template, though this is probably not requested, as the water mask is not creted by processing, right?

You will find also the DEM metadata files: maybe I did something wrong with these.

Please ask me if you need more.

Cheers,

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

As far as I can see, there are two causes for your errors: 

1. The DEM projection of your *.utm.dem.par file is “UTM”, could you re-generate it with “EQA” projection? This will result in different format of coordinates in meters or degrees for “corner_north/lat, corner_east/lon, post_north/lat, post_east/lon”,  as shown in the comparison attached below: yours on the left and mine on the right. I attached mine

After correcting this, the *.UTM_TO_RDC file will be recognized as in geo coordinates (currently it’s was recgnized as radar coordinates) and it will be loaded into ./INPUTS/geometryGeo.h5 file, similar to the Kuju* example dataset.

2. It seems like your sim*.rsc.dem file and diff*rlks.unw file have different size: one in (2023, 1713) and the other in (2024, 1713), could you confirm it?

Yunjun

--
You received this message because you are subscribed to the Google Groups "PySAR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py-sar+un...@googlegroups.com.
To post to this group, send email to py-...@googlegroups.com.
Visit this group at https://groups.google.com/group/py-sar.
For more options, visit https://groups.google.com/d/optout.

<PySAR_processing_0703.png><NaplesRS2AF6N.txt><pysarApp_template.txt><sim_120131_5rlks.diff_par><sim_120131_5rlks.utm.dem.par>

[Sven]

Hi Yunjun,

I did the changes you suggested, using an EQA projection DEM (larger than the images I've processed, does it matter?), though I was able to go only a bit ahead but not to conclude processing.

In particular, I was able to load data but I get an "input dataset ['latitude'] not found in file INPUTS/geometryRadar.h5" message error (see the attached file).

And of course you're right: the sim*.rsc.dem file and diff*rlks.unw file have different size, so I had to discard many ifgs, b.t.w. no problem, this is just a test processing. I have a long dataset so, as soon as processing works well, I'll process more ifgs.

Thanks and sorry for disturbing you.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

The *.UTM_TO_RDC is a lookup file, and is in geo coordinates while all the other files are in radar coordinates, so it’s normal that they are different in size for GAMMA.

Regarding the error message, it seems like the script find the wrong lookup file: it should be the “rangeCoord” and “azimuthCoord” datasets in ./INPUTS/geometryGeo.h5 file; not the “latitude” and “longitude” dataset in ./INPUTS/geometryRadar.h5 file, which is for ISCE products. I guess it is because the metadata got messed up. Could you empty the PYSAR folder, and remove all *.rsc files in the GAMMA folder, and run pysarApp.py again?

Your ./INPUTS folder should be similar to the Kuju* example dataset and has 3 files:

geometryGeo.h5

geometryRadar.h5

ifgramStack.h5

I would like to draw your attention to the newly available Jupyter Notebook for pysarApp: https://github.com/yunjunz/PySAR/tree/master/docs/Notebooks.You should be able to get more familiar with the processing there with explanations step by step.

Cheers,

Yunjun

--
You received this message because you are subscribed to the Google Groups "PySAR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to py-sar+un...@googlegroups.com.
To post to this group, send email to py-...@googlegroups.com.
Visit this group at https://groups.google.com/group/py-sar.
For more options, visit https://groups.google.com/d/optout.

<Latitude_not_found.png>

[Sven]

Hi Yunjun,

I tried to generate the right lookup table in GAMMA (from the "gc_map_inversion" script for lookup table inversion) but I'm still unable to obtain the "geometryGeo.h5" file (only the *radar one): I don't understand where I'm wrong.

Also the Jupyter Notebook example doesn't help too much, simply stating that GAMMA and ROI_PAC lookup files are in geo coordinates...

Maybe if you provide on your web pages also the GAMMA example this could help in some way...

Every suggestion is welcome.   

Cheers,

Sven.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

I uploaded an Gamma example in the link below. Note that all *.rsc files are generated by PySAR and the rest are from Gamma. Only one interferogram is included, but it will serve the purpose here.
(I did not upload the whole dataset online because this dataset is very challenging with a lot of unwrapping errors in other interferograms). I hope to upload a full version of another test area in one or two weeks.
https://miami.box.com/s/9vsmwrk7b0q33jhvgqg2705r4zw7fusq

Here is an script written by Yunmeng. He uses both “gc_map” and “gc_map_fine” to generate the final UTM_TO_RDC file. I don’t know the difference between “gc_map_inversion” and “gc_map_fine” though, Yunmeng can comment more on that (I cced him here).
https://github.com/ymcmrs/PyINT/blob/3661aade1ec0fc2daaa12daf55ffd2622fe5cce9/pyint/CreateRdcDem_Gamma.py#L288

Once you have the UTM_TO_RDC file, you could run the following two commands to prepare the .rsc file and plot the data, the plotting should have the lat/lon coordinate automatically, as the attachment below.
prep_gamma.py sim_040313_4rlks.UTM_TO_RDC 
view.py sim_040313_4rlks.UTM_TO_RDC range

Yunjun

[Yunjun Zhang]

Hi Sven,

I would like to let you know that there is a new example dataset processed by Gamma available here: https://github.com/insarlab/PySAR/wiki/Example

Cheers,

Yunjun

On Monday, March 11, 2019 at 3:08:53 PM UTC-4, Yunjun Zhang wrote:

Hi Sven,

I uploaded an Gamma example in the link below. Note that all *.rsc files are generated by PySAR and the rest are from Gamma. Only one interferogram is included, but it will serve the purpose here.
(I did not upload the whole dataset online because this dataset is very challenging with a lot of unwrapping errors in other interferograms). I hope to upload a full version of another test area in one or two weeks.
https://miami.box.com/s/9vsmwrk7b0q33jhvgqg2705r4zw7fusq

Here is an script written by Yunmeng. He uses both “gc_map” and “gc_map_fine” to generate the final UTM_TO_RDC file. I don’t know the difference between “gc_map_inversion” and “gc_map_fine” though, Yunmeng can comment more on that (I cced him here).
https://github.com/ymcmrs/PyINT/blob/3661aade1ec0fc2daaa12daf55ffd2622fe5cce9/pyint/CreateRdcDem_Gamma.py#L288

Once you have the UTM_TO_RDC file, you could run the following two commands to prepare the .rsc file and plot the data, the plotting should have the lat/lon coordinate automatically, as the attachment below.
prep_gamma.py sim_040313_4rlks.UTM_TO_RDC 
view.py sim_040313_4rlks.UTM_TO_RDC range

Yunjun

On Mar 11, 2019, at 10:07 AM, Sven wrote:

Hi Yunjun,

I tried to generate the right lookup table in GAMMA (from the "gc_map_inversion" script for lookup table inversion) but I'm still unable to obtain the "geometryGeo.h5" file (only the *radar one): I don't understand where I'm wrong.

Also the Jupyter Notebook example doesn't help too much, simply stating that GAMMA and ROI_PAC lookup files are in geo coordinates... 

Maybe if you provide on your web pages also the GAMMA example this could help in some way...

Every suggestion is welcome.    

Cheers,
Sven.
Sven.


Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:
Good morning,

I need some guidance in setting up the weather model for tropospheric delay correction.

I never used pyaps before, so I downloaded some grib files (ERA-Int*.grb) for testing and it works.

Now I'm going to process a time-series so my question is: should I download a grib file for each acquisition date, regardless of the number of ifgs?
Likely, I don't think so, as each single file has a size of about 28 MB (so the total amount should be 28 MB x n dates), whereas your ECMWF file for the Alos example is only 22 MB.

How should I do? Should I merge them in some way?

These are grib files and, as far as I understand, they must be converted into HDF files (grb -> h5): is load_hdf5.py the right script for this? If so, could you please give me the complete command? (load_hdf5.py ... )

Last point: the volcanic area I'm going to study is close to the coast, so I have water in my scenes: how can i get the water mask? I saw you used that in the S-1 example.

Sorry for disturbance and have a nice day.
Sven.

-- 
You received this message because you are subscribed to the Google Groups "PySAR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to py-sar+unsubscribe@googlegroups.com.

[Sven]

Hi Yunjun,

thanks for the info.

I suppose it will be very useful for me.

Cheers,

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Sven]

Hi Yunjun,

I'm slowly moving forward with PySAR processing, though I get still some errors.

I tried to process a cropped stack of CSK images and I got a "broadcast" error message (see the "Terminal" attached file).

My feeling is that Pysar is not able to match something with the range and azimuth values of the cropped images (too small? should I work on the full scene?): see the "date1_5rlks.amp.par" file for these values.

The lookup file now should be ok (see the "sim_date1_date2_5rlks" file).

I'm also attaching the processing templates: one made by me, very basic one (IschiaCskA) and the one generated by PySAR (pysarApp_template).

A basic question, which is my concern: do you think I could have problems as my Aoi (the island) lies completely inside a non coherent area? (the water, of course, see the "rmli" file).

I remember in ROI_PAC there was a script ("bridge", or something like that) which "carried" phase values from the inland to the island but, obviously, the land should be in the scene, which is not my case.

A lot of questions so please take your time for the answer.

Thanks again and sorry for disturbance.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

pysar should load anything, including cropped images (both Kuju and Fernandina examples are cropped too), as long as they are in the same size (length and width) with correct metadata file. The loading error you shown seems still related to data size. Does the "WIDTH" and "LENGTH" in your diff*.unw.rsc file has the correct value? It should be 1300 by 1360 in your case. I would recommend:

A) debug the prep_gamma.py script, to find the file name potential conflict, which causes the incorrect metadata value. You can display the .unw file with view.py to check the program can read it right or not.

or B) attached one entire interferogram folder (without the binary data file is fine, they are large to attach) and I can debug it for you, such as "20170627_20170814".

Your lookup file looks good to me too.

Regarding your question on the island: I don't think you will have a problem. Fernandina example is the same as yours in this sense. But be aware that you will have to select a reference point on this island later in the processing (automatic by pysar or manual; InSAR can only measure relative deformation), thus your result is relative to this pixel and could be biased if this pixel is deforming also.

FYI, PySAR have a script unwrap_error_bridging.py, similar to roipac bridge.pl, it implements a strategy to connect different islands.

Best,

Yunjun

[Sven]

Hi Yunjun,

1) you're right: "WIDTH" and "LENGTH" in my "diff*.unw.rsc" file have completely different values with respect to those ones in the "date1_5rlks.amp.par" (6800, 5200 vs. 1300, 1360).

The error is clearly evident from the attached file (wrong_unw) when I used "view.py" as you suggested.

B.t.w. I solved the problem with a correct use of "prep_gamma.py": before PySAR processing I gave "prep_gamma.py *.unw *.cor" which gave me the error.

Now I've used extended file names ("prep_gamma.py diff_20170627_20170814_5rlks.unw filt_20170627_20170814_5rlks.cor") which fixed the error, as you can see from the "correct_unw" attached file.

2) I've already set in the processing template as the reference point the starting point (the most stable one) of the leveling network of the island, so this should be ok.

3) nice to know PySAR has a script to implement a strategy to connect different islands: my AoI is a gulf with two different volcanic areas in the inland and two islands at the edge of it.

Last but not least, did you give a look to my processing template? (IschiaCskA, attached). Is it too poor? too basic? should I add something more for a successfull processing?

B.t.w. now i'll correct all the ifgs and try a processing: I'll let you know in case of problems.

Thanks for your suggestions.

And have a nice Easter time.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

I am glad to hear that you have solved the size problem.

Your template file looks fine to me. You may want to play with network modification if your dataset has some very low coherent interferograms (which are prone to have unwrapping errors), in order to increase the spatial coverage (pixel density). I always recommend generating more interferograms (with GAMMA) and excluding low coherent ones in the time-series stage (with PySAR). Wells Envisat dataset is an example of that (https://github.com/insarlab/PySAR/wiki/Example#the-2008-wells-nevada-earthquake-from-envisat-with-gamma-zenodo-280m-in-size).

The current unwrapping error correction methods in PySAR required connected components files, which can be generated by SNAPHU or ICU methods, both are not included in GAMMA, thus, not applicable, unfortunately. 

I would be glad to hear if you went through the whole processing with pysarApp too,

Regards,

Yunjun

[Sven]

Hi Yunjun,

processing is fine until I reach the "tropospheric correction" step, where PySAR is unable to import pyaps (see the "PySAR_processing" attached file).

I have both an old py2 pyaps installation under $HOME/giant/GIAnT/pyaps, following the indications from the GIAnT Caltech wiki and now pyaps3 under $HOME/python/PyAPS, following the indications from the PySAR installation web page: both PATH and PYTHONPATH are set accordingly.

In the PyAPS folder I've installed also cdsapi-0.1.4.

B.t.w., if I import pyaps from python2, I've no problems, whereas if I try to import pyaps3 from python3, I get an import error message, with no module named cdsapi, which is regularly installed next to pyaps3, as I told you (see the "pyaps&pyaps3" attached file).

Maybe some conflicts? Should i remove the old py2 pyaps?

Or simply a PATH setting issue?

Sorry but I tried to fix this many times before writing to you.

I think if I'm able to fix this, processing will go smoothly till the end.

Cheers,

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

For pyaps in python2 from Caltech, I guess the error is caused by the conflict of "python", which is directed to 'python3' instead of "python2" and led to the import error when you run tropo_pyaps.py. You could constrain the python version to "python2" in here: https://github.com/insarlab/PySAR/blob/master/pysar/tropo_pyaps.py#L1.

For pyaps3 in python3 from Github, were you able to import cdsapi in python3 environment? Could you tell me more how did you installed it? Note that I updated the conda.txt file recently, to include cdsapi through conda install command, if you are using conda for your python environment installation. Note that tropo_pyaps3.py is still experimental (not used by default in pysarApp.py yet), so bugs are expected, we are working together with Angelique Benoit and Romain Jolivet to make it more robust.

A side note: since pyaps/pyaps3 is used as a python module, PATH is not needed, setup PYTHONPATH is enough, as shown in PySAR installation page. But I don't think this will cause any conflict.

Regards,

Yunjun

[Sven]

Hi Yunjun,

with regard to "cdsapi", the installation was done as usual with:

python setup.py build

sudo python setup.py install.

Cdsapi has been set also in the .bashrc file (see the attached "bashrc" file for its related part), and this is also seen by python (both 2&3): see the other attachment (pyaps_pyaps3 ... ).

I've also changed the shebang of the "tropo_pyaps.py" script from ... python into ... python2.

Then, updated PySAR through the "git pull" command and inserted the hidden ".cdsapirc" script into my $HOME for ERA5 download.

B.t.w. if I process my data ("pysarApp.py IschiaCskA.txt"), PySAR stops with a "Value Error: cannot reshape array of size 1798064 into shape (1879)".

I suppose this is related to the new "prep_gamma.py" script and the fact I have to regenerate new files in the "geom_master" folder, right?

In this regard, in the GAMMA processing I've generated the terrain geocoding and calibration of SAR images including lookup-table refinement + resampling of the DEM to SAR Range-Doppler Coordinates (RDC): with the new "prep_gamma.py" script should I generate something more in the GAMMA processing before using your script?

Sorry, I'm a bit confused now.

Sven.

Il giorno venerdì 8 febbraio 2019 12:58:40 UTC+1, Sven ha scritto:

[Yunjun Zhang]

Hi Sven,

If you have old pyaps working for you, nothing needs to change. 

Could you tell me more about the error you have?

should I generate something more in the GAMMA processing before using your script?

No, you don't need to.

Yunjun

[Sven]

Hi Yunjun,

with regard to my previous error message "Value Error: cannot reshape array of size 1798064 into shape (1879)", I fixed this: it was related to a wrong *.UTM_TO_RDC lookup file in my "geom_master" folder.

Then I started a regular processing but PySAR ended at the "correct_troposphere" stage with a PyAPS error message, though both pyaps and pyaps3 are regularly set.

It may sound strange but I solved the problem typing correctly the command in the terminal (tropo_pyaps.py -f timeseries.h5 --model ECMWF -g /home/pc-gamma/IschiaCskA/PYSAR/INPUTS/geometryRadar.h5 -w /home/pc-gamma/WEATHER), because PySAR gives 2 empty spaces between the "tropo_pyaps" script and the "-f" flag. Then the download of the "grb" files started regularly.

I concluded my processing with "pysarApp.py template.txt --start deramp --end hdfeos5".

Just to know, may I change somewhere in PySAR this command? Let's say, a config file, the template itself or something similar?

Cheers,

Sven.

Just a side note: processing was not so good but only because I was dealing with few images/ifgs, in a quite decorrelated area, b.t.w. this was just to test PySAR, it seems to be great ;)

[Yunjun Zhang]

Hi Sven,

py2-pyaps should be working with pysarApp.py now. Please update your pysar code to try it. Thank you for pointing it out.

I am glad to hear you went through the end now! Your result would be better once you have more redundancy (number of interferograms / number of SAR images), that's how the SBAS algorithm gains its strength.

Cheers,

Yunjun