Error in step 2, using the outputs of ISCE

[Leo]

Dear David,

I generated the StaMPS directory set-up using the output from the isce topsStack processors (SLC workflow). But when i run StaMPS is step 2, the following error occurred:

Can you help me with this problem? I've been beating my brains out with it, but I just can't solve it.

By the way, when I preprocessed this batch of data with SNAP, I made no mistakes in StaMPS.

Thank You!

Leo

Warning: Rank deficient, rank = 0, tol = NaN.
> In ps_topofit (line 53)
  In ps_est_gamma_quick (line 247)
  In stamps (line 326)
Warning: Rank deficient, rank = 0, tol = NaN.
PS_EST_GAMMA_QUICK: gamma_change_change=NaN
GETPARM: gamma_change_convergence=0.005
GETPARM: gamma_max_iterations=3
Subscript indices must either be real positive integers or logicals.

Error in ps_est_gamma_quick (line 303)
                Prand_ps=Prand(round(coh_ps*1000)+1)';

Error in stamps (line 326)
                    ps_est_gamma_quick(est_gamma_parm);
 

[Stoorm]

I got the same errors. Is there any solution?

Thx

воскресенье, 17 марта 2019 г., 16:30:13 UTC+3 пользователь Leo написал:

[David Bekaert]

Hi,

the gamma_change_change is Nan, so i suspect there is a nan leaking somewhere in the data.

Could you perhaps check which of the variable used within the calculation is introducing a nan?

Based on that i might be able to provide furhter insight in what we could do to get rid of the bug.

Thanks,

D.

--
You received this message because you are subscribed to the Google Groups "MAINSAR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mainsar+u...@googlegroups.com.
To post to this group, send email to mai...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stoorm]

Hi David,

How could I do that?

Processing log before crashing:

21-Mar-2019 00:15:52 STAMPS           ########################################

21-Mar-2019 00:15:52 STAMPS           ################ Step 2 ################

21-Mar-2019 00:15:52 STAMPS           ########################################

21-Mar-2019 00:15:52 STAMPS           Directory is PATCH_1

21-Mar-2019 00:15:52 PS_EST_GAMMA_QUICK Starting

21-Mar-2019 00:15:52 PS_EST_GAMMA_QUICK Estimating gamma for candidate pixels

21-Mar-2019 00:15:52 GETPARM          filter_grid_size=50 

21-Mar-2019 00:15:52 GETPARM          filter_weighting='P-square'

21-Mar-2019 00:15:52 GETPARM          clap_win=32 

21-Mar-2019 00:15:52 GETPARM          clap_low_pass_wavelength=800 

21-Mar-2019 00:15:52 GETPARM          clap_alpha=1 

21-Mar-2019 00:15:52 GETPARM          clap_beta=0.3 

21-Mar-2019 00:15:52 GETPARM          max_topo_err=20 

21-Mar-2019 00:15:52 GETPARM          lambda=0.031 

21-Mar-2019 00:15:52 GETPARM          gamma_change_convergence=0.005 

21-Mar-2019 00:15:52 GETPARM          gamma_max_iterations=3 

21-Mar-2019 00:15:52 GETPARM          small_baseline_flag='n'

21-Mar-2019 00:15:57 PS_EST_GAMMA_QUICK n_trial_wraps=0.117576

21-Mar-2019 00:15:57 PS_EST_GAMMA_QUICK Initialising random distribution...

21-Mar-2019 00:16:19 PS_EST_GAMMA_QUICK 2962029 PS candidates to process

21-Mar-2019 00:16:19 PS_EST_GAMMA_QUICK iteration #1

21-Mar-2019 00:16:19 PS_EST_GAMMA_QUICK Calculating patch phases...

21-Mar-2019 00:17:55 PS_EST_GAMMA_QUICK Estimating topo error...

четверг, 21 марта 2019 г., 0:35:51 UTC+3 пользователь David Bekaert написал:

[Leo]

Hi David,

As shown in the attachment, the nan values appeared after registration.

This error will not appear when I use gdalwarp to subset registration files , but the end StaMPS result looks wrong.

Then we add a line of code (coh_ps(isnan(coh_ps))=0) at line 292 to change the nan value to 0. The problem seems to have been solved but I am not sure whether it will affect the unwrap or other steps.

Thanks,

Leo

                

在 2019年3月21日星期四 UTC+8上午5:35:51，David Bekaert写道：

[David Bekaert]

Hi Leo,

I assume the crop is large than the overlap of data in the stack, i.e. black band of data

I suggest to drop the points rather than forcing the coherence to zero, as there is no data and for PS,SB you need the whole stack connected in time.

This would however need to be propagated to all the variables used in the calculation there.

e.g. if coherence is compared to baseline or phase, then all should have remove the nan-pixels.

so likely something like

ix_drop = isnan(coh_ps)

coh_ps(ix_drop,:)=[]

othervariables(ix_drop,:)=[]

Cheers,

D.

[Leo]

Hi David,

Thank you for your guidance.I did it again according to your method and got a better result.

Sincerely

Leo

在 2019年3月22日星期五 UTC+8下午11:19:51，David Bekaert写道：

[Stoorm]

Hi, Leo
Сould you tell me which line in the ps_est_gamma_quick.m script you added the code to?

Line 292?

ix_drop = isnan(coh_ps)

coh_ps(ix_drop,:)=[]

othervariables(ix_drop,:)=[]

Thx!

воскресенье, 24 марта 2019 г., 15:39:16 UTC+3 пользователь Leo написал:

[Stoorm]

Hi, David

I checked and removed NaN from the input variables (from the first step - ph, lonlat etc)), but I still get the same error in the second step. I checked the interferograms - they look properly. 

If I try to add the code that you suggested, the processing is unrealistically long (in fact-the result is not completed) for one patch with < 1 million pixel. 

Could you please advise what else can I do?

Thanks a lot.

[David Bekaert]

Hi Leo,

Could you issue a PR on GitHub with the fixes you made.

I will inspect the changes to incorporate in main branch.

Thanks,

David

[David Bekaert]

Is there a way for you to share me privately one of the patch folders.

I can have a closer look if I have the data.

Thanks,

D.

On Tue, 26 Mar 2019 at 09:02, Stoorm <vitaliy....@gmail.com> wrote:

[Stoorm]

David, 

I sent you a link to patch folder in a private message.

Thx

To unsubscribe from this group and stop receiving emails from it, send an email to mai...@googlegroups.com.

[Stoorm]

Hi, David

I found the cause of the error is an empty (full of zero and NaNs) bperp_mat variable. How can I check at what stage of processing I have incorrect values? 

When running the run_7_grid_baseline from contrib/stackStripMap.py? Make_single_master_stack_isce or after step 1 in Matlab?

I also tried to convert the baseline txt  files using step_baseline_stack.py but I got the following error: ValueError: could not convert string to float.

If I use baseline grids (from merge dir) in this case after the step 1 I get an empty bperp_mat file again.

I processed different CSK datasets and got the same error in baselines data.

Thanks

On Tuesday, March 26, 2019 at 7:24:14 PM UTC+3, David Bekaert wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to mai...@googlegroups.com.

[David Bekaert]

I saw a few other posts of people with similar issue.

Q: Can you check if the baseline grids were non-zero and were made for all the pairs (i.e. after running run_7_grid_baseline)?

step_baseline_stack.py script would just load these grid and compute an average which goes into the txt files.

Q: Are you using the same master for the stack as for stamps PS processing?

Depending if you choose a different master, there is a slight variation in the step_baseline_stack.py code.

I will need to check  a sample dataset for of the stripmap output to see what is happening.

Hopefully i can find some time for this next week.

Cheers,

D.

[JW]

Hi,

baseline grid files (30 by 30 dimension in my case) generated after running run_7 have non-zero values (perp baselines) in the first row and all zero values in other rows. 

The zero values loaded into STAMPS cause problems in ps_topofit. Because the perp baselines in a small cropped area have a slight variation, the zero values can be filled with the existing baselines. Then, 2nd step of STAMPS worked well anyway. 

But I guess that python script used in run_7 can be edited to avoid the problem.

thanks

JW 

[David Bekaert]

Hi,

When we are making the baseline grid we initialie it as an empty zero matrix.

we made some changes intenrally to give this a unique no-data value as gdal will capture a no-data in the meta-data.

This would be mainly for the master grid as that would have zero's.

However the problem here is different as the baseline grid is typical non-zero for slave combinations.

Could you investigate what the messages are when running the baseline grid in step 7 for one of them?

I believe you do not have enough state-vectors to fill the complete grid.

i.e. we give a +- delta in time as the baseline grid is slightly larger than the IFG.

D.

[Stoorm]

Hi, David

I checked all baselines data throughout the all processing chain. 

All baselines files (masterdate_slavedate.txt) from main processing directory looking normally:

PERP_BASELINE_BOTTOM 389.7613050350376

PERP_BASELINE_TOP 384.21776228421413

Baselines grid files from the masterShelve and slaveShelve looking not so normally:

gdalinfo -stats 20180908

Driver: ISCE/ISCE raster

Files: 20180908

       20180908.xml

Size is 30, 30

Coordinate System is:

GEOGCS["WGS 84",

    DATUM["WGS_1984",

        SPHEROID["WGS 84",6378137,298.257223563,

            AUTHORITY["EPSG","7030"]],

        AUTHORITY["EPSG","6326"]],

    PRIMEM["Greenwich",0,

        AUTHORITY["EPSG","8901"]],

    UNIT["degree",0.0174532925199433,

        AUTHORITY["EPSG","9122"]],

    AUTHORITY["EPSG","4326"]]

Origin = (0.000000000000000,0.000000000000000)

Pixel Size = (1.000000000000000,1.000000000000000)

Corner Coordinates:

Upper Left  (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)

Lower Left  (   0.0000000,  30.0000000) (  0d 0' 0.01"E, 30d 0' 0.00"N)

Upper Right (  30.0000000,   0.0000000) ( 30d 0' 0.00"E,  0d 0' 0.01"N)

Lower Right (  30.0000000,  30.0000000) ( 30d 0' 0.00"E, 30d 0' 0.00"N)

Center      (  15.0000000,  15.0000000) ( 15d 0' 0.00"E, 15d 0' 0.00"N)

Band 1 Block=30x1 Type=Float32, ColorInterp=Undefined

  Minimum=-384.954, Maximum=0.000, Mean=-12.778, StdDev=68.809

  Metadata:

    STATISTICS_MAXIMUM=0

    STATISTICS_MEAN=-12.777554423014

    STATISTICS_MINIMUM=-384.9538269043

    STATISTICS_STDDEV=68.809468956147

Another example from the same CSK data set but with zero as minimum:

gdalinfo -stats baselineGRID_20180702

Driver: ISCE/ISCE raster

Files: baselineGRID_20180702

       baselineGRID_20180702.xml

Size is 30, 30

Coordinate System is:

GEOGCS["WGS 84",

    DATUM["WGS_1984",

        SPHEROID["WGS 84",6378137,298.257223563,

            AUTHORITY["EPSG","7030"]],

        AUTHORITY["EPSG","6326"]],

    PRIMEM["Greenwich",0,

        AUTHORITY["EPSG","8901"]],

    UNIT["degree",0.0174532925199433,

        AUTHORITY["EPSG","9122"]],

    AUTHORITY["EPSG","4326"]]

Origin = (0.000000000000000,0.000000000000000)

Pixel Size = (1.000000000000000,1.000000000000000)

Corner Coordinates:

Upper Left  (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)

Lower Left  (   0.0000000,  30.0000000) (  0d 0' 0.01"E, 30d 0' 0.00"N)

Upper Right (  30.0000000,   0.0000000) ( 30d 0' 0.00"E,  0d 0' 0.01"N)

Lower Right (  30.0000000,  30.0000000) ( 30d 0' 0.00"E, 30d 0' 0.00"N)

Center      (  15.0000000,  15.0000000) ( 15d 0' 0.00"E, 15d 0' 0.00"N)

Band 1 Block=30x1 Type=Float32, ColorInterp=Undefined

  Minimum=0.000, Maximum=109.001, Mean=3.582, StdDev=19.291

  Metadata:

    STATISTICS_MAXIMUM=109.00128173828

    STATISTICS_MEAN=3.5820626661513

    STATISTICS_MINIMUM=0

    STATISTICS_STDDEV=19.290707316395

Could it be a problem with the baselineGrid.py?

About the second question. I tried using different masters but got the same error as a result.

Thx

[Stoorm]

David, 

step 7 messages:

Running: baselineGrid

['--master', '/home/insar/1/ISCE/proc/CSK/proc/merged/SLC/20130718/masterShelve', '--slave', '/home/insar/1/ISCE/proc/CSK/proc/merged/SLC/20130718/slaveShelve', '--baseline_file', '/home/insar/1/ISCE/proc/CSK/proc/merged/baselines/20130718/20130718']

This is the Open Source version of ISCE.

Some of the workflows depend on a separate licensed package.

To obtain the licensed package, please make a request for ISCE

through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.

Alternatively, if you are a member, or can become a member of WinSAR

you may be able to obtain access to a version of the licensed sofware at

https://winsar.unavco.org/software/isce

Baseline file /home/insar/1/ISCE/proc/CSK/proc/merged/baselines/20130718/20130718 dims: 30L x 30P

Polynomial Order: 0 - by - 0 

0

Polynomial Order: 0 - by - 0 

0

Polynomial Order: 0 - by - 0 

0

The zeros go to the end of processing.

Thx

[Stoorm]

Hi, David

I sent you a link to the CSK stripmap output data. 

I also tried to process Alos2 stripmap data and got the same error as for CSK.

Best

Thx

[David Bekaert]

Thnx for sharing.

This helps a bit.

Could you email me a few of the basline grid files and metadata?

D.

[Stoorm]

Hi, David

I sent you a link to the Alos2 baseline grid data.

Best

Thnx

[Stoorm]

Hi, David, 

Is there a solution to this issue?

Thanks

[David Bekaert]

Would recommend reaching out on the isce forum to see what can be done to extent the statevectors for the baseline grid to ensure it has data in it rather than zeros.

Alternatively modify in stamps to fill the the baseline grid with the average value.

D.

To view this discussion on the web visit https://groups.google.com/d/msgid/mainsar/241ac609-0604-45c8-92c0-4de7e413c16e%40googlegroups.com.

[Stoorm]

Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to mai...@googlegroups.com.

[Stoorm]

Hi,

The issue has been solved. Details here http://earthdef.caltech.edu/boards/4/topics/3016?r=3087

To unsubscribe from this group and stop receiving emails from it, send an email to mai...@googlegroups.com.

[raincxy]

Can u teach me how to use stackStripMap.py ?  I have followed  the instructions on Github but i cant get run_files. https://github.com/isce-framework/isce2/blob/master/contrib/stack/stripmapStack/README.md

Can u give me some advice ?

[raincxy]

Hi Stoorm

I got the same errors as you.Have you solved this problem?
thanks