Questions on setting up MintPy from raw ISCE2 output

[Jatin Mathur]

Hi,

First time MintPy user here (and ISCE2). I had some questions on setting up the expected files for MintPy.

Looking at https://mintpy.readthedocs.io/en/latest/dir_structure/#isce_stripmapstack, I am missing:

- Igrams/.../filt_20060624_20061225.cor. Looking at the interferogram folder in the ISCE2 output, I have a phsig.cor and a topophase.cor, if those work

- ionosphere/.../iono.bil.unwCor.filt. Looking at the ionosphere folder in the ISCE2 folder, I have a dispersive.bil.unwCor and a nondispersive.bil.unwCor

- geom_reference/hgt.rdr. Not sure which file this might correspond to in the raw ISCE2 output. Same with geom_reference/incLocal.rdr and geom_reference/shadowMask.rdr.

One thing to keep in mind is I ran ISCE of ISCE master (hash 0dbb16). Perhaps I should run off the version installed in the MintPy Dockerfile?

Also, a few other questions:

1. Is "merged/SLC/.../referenceShelve" and "merged/baselines" generated with `prep_isce.py`? 

2. Is there some existing script to setup files in the file tree MintPy wants from raw ISCE output? 

Thanks!

Jatin

[Jatin Mathur]

Update: I realized "stripmapStack" was hyperlinked so I followed it and realized the stack stripmap processor is being used for MintPy, not the "stripmapApp" processor that I originally used. I tried following the instructions, but ran into an issue on the first runfile where it seemingly a) wanted to download data (I was confused why cause I gave it the DEM file and I thought that's all it needed), and b) after I setup the .netrc it had an issue downloading data. Here are the complete logs from where it failed to download data:

command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W158.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   501    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   294    0   294    0     0    223      0 --:--:--  0:00:01 --:--:--   359
  0   269    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0   196    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:06,587 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W158.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W157.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:07,085 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W157.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W156.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:07,571 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W156.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W155.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:08,055 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N71W155.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W158.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:08,545 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W158.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W157.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:09,031 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W157.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W156.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:09,514 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W156.SRTMSWBD.raw.zip. Requested file seems not present on server.
command = curl -n  -L -c $HOME/.earthdatacookie -b $HOME/.earthdatacookie -k -f -O https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W155.SRTMSWBD.raw.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   196    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (22) The requested URL returned error: 404
2023-09-03 00:54:09,994 - isce.iscesys.DataRetriever - WARNING - There was a problem in retrieving the file  https://e4ftl01.cr.usgs.gov/MEASURES/SRTMSWBD.003/2000.02.11/N70W155.SRTMSWBD.raw.zip. Requested file seems not present on server.
API open (R): /permafrost-prediction/src/py/methods/swbdLat_N70_N72_Lon_W158_W154.wbd
API close:  /permafrost-prediction/src/py/methods/swbdLat_N70_N72_Lon_W158_W154.wbd
Writing geotrans to VRT for /permafrost-prediction/src/py/methods/swbdLat_N70_N72_Lon_W158_W154.wbd
converting water mask file to radar coordinates ...
API open (R): /permafrost-prediction/src/py/data/alos_palsar/try_stack_stripmap/geom_reference/waterMask.rdr
API close:  /permafrost-prediction/src/py/data/alos_palsar/try_stack_stripmap/geom_reference/waterMask.rdr

I am not quite sure what data it wanted to download or how to bypass this. Is this the file (run file 1) that produces interferograms? If so, that explains why I don't see any "Igrams" folder in the output. I went ahead and ran steps 2-7 and those had no issue.

Let me know if I should provide more info.

Thanks,

Jatin

[Eric Fielding]

Hi Jatin,

The stripmapStack workflow uses a water mask. It tries to download the SRTM Water Body Database from the NASA Eros archive center operated by the USGS. Check to see that you have the most recent version of ISCE2. Sometimes they move the data archive locations and we have to update the ISCE2 code.

You have to run all the steps in the stripmapStack workflow in order.

++Eric

[Eric Fielding]

Hi Jatin,

I tried to download the water body database file myself, and now I see that you are working at latitude 70 N. The SRTM DEMs and water body data stop at 60 N, so there is no water mask data from the SRTM database for your area. I have only run the stripmapStack workflow on areas where SRTM data was available. It is possible that you have found a bug where it fails to make the waterMask at high latitudes and does not complete the rest of steps. The "run_01" step calculates the geometry and masks for the reference scene, so it is a required first step.

I would recommend posting this as an issue in the ISCE2 repository: 

https://github.com/isce-framework/isce2/issues

++Eric

[Zhang Yunjun]

Just want to add one more point on top of Eric's post earlier: the water mask is the last sub-step of the "run_01" step (https://github.com/isce-framework/isce2/blob/0dbb1679b61b4ac385c537269b91208e57024672/contrib/stack/stripmapStack/Stack.py#L346-L351), and the generated water mask file is not really used anywhere afterward in the isce2 stripmap stack processing if I remember it right. So your isce processing may be fine without it.

ARIA-tools (https://github.com/aria-tools/ARIA-tools) uses the GSHHS water mask dataset from the GMT community, which should have water mask data over 60 N.

I second Eric's recommendation for opening a github issue on the isce2 repo, so that other users and developers could also be aware of this issue and maybe fix it.

Yunjun

[Jatin Mathur]

Hi Eric and Yunjun,

Thanks for the help. I am now running with the latest version of ISCE2 (hash 0dbb16) and it was fine with the water mask missing. I ran all the steps, but I do not see a folder "Igrams" or "ionosphere" being created. I have logs from all the steps running, if that would help.

Side note: What kind of development infrastructure do you guys use? I would ideally like ISCE2 source code and MintPy editable/recompilable within a Docker container, but also bind-mounted in from the host so that all my progress can't get wiped if the container gets taken down. I have a modified version of the ISCE2 Dockerfile that does this for me, but I am not sure if I should combine it with MintPy. What do you guys do?

Thanks,

Jatin

[Jatin Mathur]

Hey,

Any update on this?

[Zhang Yunjun]

Hi Jatin,

I don't use Docker much, because there is a visualization issue with mintpy (https://github.com/insarlab/MintPy/issues/801), I am not sure about this on isce2.

Here (https://github.com/yunjunz/conda_envs) is the code development/maintenance note I used. Basically, I use isce2 core module from conda-forge (because it's relatively complicated to install the development version), and the development version of isce2 stack processors and mintpy using path setup or pip editable installation.

Without more details, I could not know what's going with your stripmapStack processing. From my experience, the stack processors requires advanced python programing knowledge and a relative good understanding of the InSAR workflow itself, because there are so many processing done there, and it's difficult for the code (and its developer) to catch all possible data and human related issues.

Yunjun

[Eric Fielding]

Hi Jatin,

I use a Mac and a Linux server for my development work in ISCE2. I don't use Docker containers for my work. It seems they are good for running jobs after development but not efficient for working on development. There are services like the Open Science Lab (OpenSARLab) at the Alaska Satellite Facility that provide a JupyterHub virtual machine that is actually inside a Docker container, but they use a separate non-volatile file storage system that keeps your files. It has the visualization issue that Yunjun mentioned. The command-line tools in MintPy and ISCE2 that use X Window System displays don't work.

++Eric

[Jatin Mathur]

Huh, in my Docker setup view.py works fine on Linux. I wrote my own Dockerfile, and I simply volume mounted the X11 socket. Would you guys be interested in a PR?

[Zhang Yunjun]

Hi Jatin,

It's nice to hear you have view.py working! Yes, a PR on GitHub is more than welcomed!

Regards,

Yunjun