Issue with the surface reflectance calculation using the 6s atmospheric model (SREF)

[Antonio Falciano]

Hi all,

first of all I'd like to thank the developers of such awesome software. I discovered ARCSI about an year ago when searching an open source software able to perform atmospheric correction of Landsat images using a robust model (6s) and possibly in batch mode. The mission was accomplished successfully thanks to ARCSI. Now I have the same need for another research project, but I've accidentally lost the previous conda environment. Thus I've installed everything from scratch using python>=3.6 and testing both the latest ARCSI stable version of conda-forge (3.6.1) and the development one (3.2.3) of au-eoed channel. The result was quite the same in every configuration I have tested (except one that I'll report later):

[...]

IDENTIFIED AOT:  0.05
AOT Search Range = [0.05, 0.45]
AOT Value: 0.05
Converting Image Projection
Min X: 3365320.70122
Max X: 3183757.10239
Min Y: 2186672.97067
Max Y: 1825238.18108
MinX = 3183757.10239
MaxX = 746415
Input Header: '/mnt/z/rs_data/uncompressed/L5/LT05_L1GS_188032_20081012_20180116_01_T2.tar/LT05_L1GS_188032_20081012_20180116_01_T2_MTL.txt'
Output Basename: 'LS5TM_20081012_lat17lon403_r32p188'
Error: Images and Envelope do not overlap in the X axis
Error: The following products were not generated:
SREF METADATA
Input Header: '/mnt/z/rs_data/uncompressed/L5/LT05_L1GS_188032_20081012_20180116_01_T2.tar/LT05_L1GS_188032_20081012_20180116_01_T2_MTL.txt'
Output Basename: 'LS5TM_20081012_lat17lon403_r32p188'
ARCSI took 00:20:42 to process the input image. Thank you for using ARCSI.

or:

IDENTIFIED AOT:  0.05
AOT Search Range = [0.05, 0.45]
AOT Value: 0.05
Converting Image Projection
Min X: 3365320.70122
Max X: 3183757.10239
Min Y: 2186672.97067
Max Y: 1825238.18108
MinX = 3183757.10239
MaxX = 746415
Input Header: '/mnt/z/rs_data/uncompressed/L5/LT05_L1GS_188032_20081012_20180116_01_T2.tar/LT05_L1GS_188032_20081012_20180116_01_T2_MTL.txt'
Output Basename: 'LS5TM_20081012_lat17lon403_r32p188'
Error: Images and Envelope do not overlap in the X axis
Error: The following products were not generated:
SREF METADATA
Input Header: '/mnt/z/rs_data/uncompressed/L5/LT05_L1GS_188032_20081012_20180116_01_T2.tar/LT05_L1GS_188032_20081012_20180116_01_T2_MTL.txt'
Output Basename: 'LS5TM_20081012_lat17lon403_r32p188'
Traceback (most recent call last):
  File "/home/ant/miniconda3/envs/arcsienv0/bin/arcsi.py", line 605, in <module>
    arcsilib.arcsirun.runARCSI(args.inputheader, args.imagefile, args.cloudmask, args.sensor, args.inwkt, args.format, args.outpath, args.outbasename, args.outwkt, args.outproj4, args.projabbv, args.ximgres, args.yimgres, args.prods, args.stats, args.aeropro, args.atmospro, args.aeroimg, args.atmosimg, args.grdrefl, args.surfacealtitude, args.atmosozone, args.atmoswater, atmosOZoneWaterSpecified, args.aerowater, args.aerodust, args.aerooceanic, args.aerosoot, aeroComponentsSpecified, args.aot, args.vis, args.tmpath, args.minaot, args.maxaot, args.lowaot, args.upaot, args.dem, args.demnodata, args.aotfile, (not args.localdos), args.dosout, args.simpledos, args.debug, args.scalefac, args.interp, args.interpresamp, args.cs_initdist, args.cs_initminsize, args.cs_finaldist, args.cs_morphop, args.fullimgouts, args.checkouts, args.classmlclouds, args.cloudtrainclouds, args.cloudtrainother, args.resample2lowres, args.keepfileends)
  File "/home/ant/miniconda3/envs/arcsienv0/lib/python3.7/site-packages/arcsilib/arcsirun.py", line 1319, in runARCSI
    calculateSREF(paramsObj)
  File "/home/ant/miniconda3/envs/arcsienv0/lib/python3.7/site-packages/arcsilib/arcsirun.py", line 1147, in calculateSREF
    statsElev = rsgislib.imagecalc.getImageStatsInEnv(paramsObj.outDEMName, 1, paramsObj.demNoDataVal, paramsObj.sensorClass.latTL, paramsObj.sensorClass.latBR, paramsObj.sensorClass.lonBR, paramsObj.sensorClass.lonTL)
_imagecalc.error: Images and Envelope do not overlap in the X axis

Finally, after reading the open Issue #33, I've discovered that the only way to bypass the problem was to install ARCSI using python=2.7 and arcsi=3.1.6 (and so GDAL 2.4.2):

conda create -c conda-forge -n arcsienv python=2.7 arcsi=3.1.6

In conclusion, my opinion is that ARCSI is not yet ready to use GDAL>=3.x (too young to be used in production), because it may cause some disalignments in imagecalc calculations.

Cheers,

Antonio

[Peter Bunting]

Dear Antonio, 

Great than you have found ARCSI useful. It should be working with python 3.X - in fact I don’t test with python 2.7.X anymore. I have found a knew issues with RSGISLib/ARCSI since switching to GDAL 3.0.1 but these should be fixed in the latest RSGISLib release (4.0.6). However, I haven’t got ARCSI built on conda-forge against RSGISLib v4 yet. 

Can you send me the command you ran to get this error and a link to download the landsat scene you are running?

You could also try running via Docker as I have created images with what I believe are working versions of the software. There are details on how to run with docker here:

https://www.remotesensing.info/2019/09/24/rs-gis-data-processing-using-docker-and-singularity/

Best wishes, 

Pete

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rsgislib-support/d9f263cb-f5fd-4eca-bbf9-8bdf49df1613%40googlegroups.com.

[Antonio Falciano]

Dear Peter,

thank you for your quick reply. The command I ran to get this error is:

arcsi.py -s ls5tm -f KEA -p DOSAOTSGL SREF METADATA \
--dem ./dem20bas.kea --demnodata -32768 -t ./tmp \
-o $basepath/atmospherically_corrected/l5 --fullimgouts \
-i $basepath/uncompressed/L5/"$line".tar/"$line"_MTL.txt \
--keepfileends srefdem.kea meta.json --ncores 4

IMHO the issue is not scene dependent because I've tried the same command randomly with several Landsat 5, 7 and 8 scenes downloaded from GloVis and the error was exactly the same.

For completeness, I'm working with the Ubuntu (linux-64) shell on a win10 os and images are stored on an external NAS. Maybe this configuration puts his own...

Thank you for your kind remarks and suggestion. I'll try the Docker images as soon as I can.

Best regards,

Antonio

[Robin Wilson]

Hi Pete and Antonio,

I've found the same problem today, with a new conda environment created as specified in the docs.

I found that downgrading to ARCSI v3.1.6 solved the problem for me (conda struggled to downgrade the environment, so I just created a new one as 'conda create -n arcsi-old arcsi=3.1.6').

Running the older version worked fine, and gave me the full surface reflectance output. However, there seems to be a significant difference between the AOT outputs from v3.1.6 and v3.2.4 - I'll create a separate thread about this.

Thanks,

Robin

On Friday, 27 September 2019 17:07:21 UTC+1, Antonio Falciano wrote:

Dear Peter,

thank you for your quick reply. The command I ran to get this error is:

arcsi.py -s ls5tm -f KEA -p DOSAOTSGL SREF METADATA \
--dem ./dem20bas.kea --demnodata -32768 -t ./tmp \
-o $basepath/atmospherically_corrected/l5 --fullimgouts \
-i $basepath/uncompressed/L5/"$line".tar/"$line"_MTL.txt \
--keepfileends srefdem.kea meta.json --ncores 4

IMHO the issue is not scene dependent because I've tried the same command randomly with several Landsat 5, 7 and 8 scenes downloaded from GloVis and the error was exactly the same.

For completeness, I'm working with the Ubuntu (linux-64) shell on a win10 os and images are stored on an external NAS. Maybe this configuration puts his own...

Thank you for your kind remarks and suggestion. I'll try the Docker images as soon as I can.

Best regards,

Antonio

Il giorno ven 27 set 2019 alle ore 17:46 Peter Bunting <peteb...@gmail.com> ha scritto:

Dear Antonio, 

Great than you have found ARCSI useful. It should be working with python 3.X - in fact I don’t test with python 2.7.X anymore. I have found a knew issues with RSGISLib/ARCSI since switching to GDAL 3.0.1 but these should be fixed in the latest RSGISLib release (4.0.6). However, I haven’t got ARCSI built on conda-forge against RSGISLib v4 yet. 

Can you send me the command you ran to get this error and a link to download the landsat scene you are running?

You could also try running via Docker as I have created images with what I believe are working versions of the software. There are details on how to run with docker here:

https://www.remotesensing.info/2019/09/24/rs-gis-data-processing-using-docker-and-singularity/

Best wishes, 

Pete

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib...@googlegroups.com.

[Peter Bunting]

Hi Robin, 

Can you try using the latest development version as there are some issues with the 3.2.4 release which I hadn’t picked up related to changes in GDAL v3 and proj4 v6. 

I think those are now resolved and I will do an updated release soon. 

if you can install from my au-eoed conda channel - note there are only linux 64bit binaries for Python 3.7. The latest version is v3.6.4.

conda install -c au-eoed -c conda-forge arcsi

This version has some improvements for cloud masking - Landsat Collection 1 you can now use the cloud mask provided as it is generated using FMask. For Sentinel-2 there is a machine learning based cloud masking which can be used as an alternative to FMask. 

Best wishes, 

Pete

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rsgislib-support/77db49eb-1373-40dd-8122-96bf3c28ba67%40googlegroups.com.

[Robert Richard]

Hi all,

I've also run into the "Images and Envelope do not overlap in the X axis" error. I first was using python 3.6, recreated the environment and installed arcsi via conda install -c au-eoed -c conda-forge arcsi with gdal 3, but I continued to get the error. I did get arcsi to run successfully when creating a new environment in python=2.7, arcsi=3.1.6, and gdal=2.4.2.

Hope that's helpful.

Robbie