Unexpected surface reflectances calculated with new arcsi version

[silke.and...@bayer.com]

Dear all,

in April 2016 I used arcsi to derive the surface reflectances of some Landsat8 scenes using the „inversion for AOT“ method and the SRTM DEM.

The input line was something like:

arcsi.py -s ls8 –p TOA DOSAOTSGL SREF –f KEA --stats –i  <path to LC81920232015109LGN00_MTL> -o  <path to out dir> -t <…> --aeroimg <path to WorldAerosolParams.kea> --atmosimg <path to WorldAtmosphereParams.kea > --dem <path to SRTM DEM> --minaot 0.05 --maxaot 0.6 --simpledos

I have converted the output to GTiff using gdal_translate.

The result from April appeared to be reasonable (i.e. slight enhancement compared to the RAW data, very close to the SREF images provided by others (USGS) and with the NDVI lying between -1 and 1). The attached image "old.png" is a RGB image of a section of the SREF from scene LC81920232015109 and "ndvi_old" shows the NDVI for the complete scene.

A few days ago I installed the newest version of arcsi via miniconda3 on a Ubuntu 14 machine. The installation went smoothly except for the “proj.4” error that was discussed and solved in this forum (see “Conda Installation Changes”). Unfortunately the SREFs that I calculated with this version of arcsi appear to be wrong. They are very different from the results from April (see the attached RGB image "new.png" from the same Landsat8 scene) and the range of the NDVI is strange (see "ndvi_new.png"). The same issue is appearing for Rapideye images. Also, sometimes there seem to be artefacts in the images, which have not been there before.

I have tested this several times and with a new installation and I am wondering whether anybody else is having the same problem? Or have I missed something?

Any hint is welcome :)

Thanks!
Silke

[William Ray]

Hi Silke,

What do you mean by artifacts? I've had problems with Landsat 8 also when trying to derive SREF using the inversion for AOT method. In my examples, you can clearly see the DEM thresholds within the SREF product, where different AOT thresholds have been applied. However, when I run ARCSI using a global surface altitude it works absolutely fine.

I can provide an example it this helps as well?

Thanks,

Will

[Pete Bunting [pfb]]

Hi Silke,

Thanks for your email, I have make quite a few changes to ARCSI over the last few months but I haven’t seen this issue.

I am just running a test at the moment on an ls8 scene using the options you’ve specified. I don’t tend to use the --simpledos option so maybe there is an issue here - I am trying with and without this option at the moment to test this.

A minor point which won’t make any difference on this issue but you don’t need to specify the aerosol and atmosphere images (unless you have custom version) as they are now installed in installpath/share and arcsi automatically picks them up and uses them.

When you say ‘new’ version of arcsi, can you let me know which version? (i.e., arcsi.py -v) and when you did the installation? There was a new version updated on conda this week so want to check I know which version you’re using.

Many thanks,

Pete

> --
> You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
> To post to this group, send email to rsgislib...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rsgislib-support.
> For more options, visit https://groups.google.com/d/optout.
> <ndvi_old.png><new.PNG><old.PNG><ndvi_new.png>



****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
*
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”



--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

[Pete Bunting [pfb]]

Hi William, 

That seems very odd that you would be able to see the DEM steps from the LUT in the output. I have never seen that and the tests I did would suggest that the output reflectance is at most changed by 0.2% using 100 m intervals. Can you send me an example - ideally the landsat scene where you found this and the command you ran so I can check this.

However, the latest (yet to be released) version applies a linear interpolation between the LUT layers for completeness but as I said even in the high mountains with elevation ranges > 4000 m it makes minimal difference (< 0.2% reflectance).

Best wishes, 

Pete  

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[silke.and...@bayer.com]

Hi Pete and Will,

thank you for the comments! I have repeated my computations without the --simpledos option (and without specifying --atmosimg and --aeroimg). This does not make a difference.

Versions: I have installed ARCSI on two different computers. One was updated mid-December 2016 and tells me that it has ARCSI version 0.16.0. For doublechecking I re-installed everything on a second computer at the beginning of this week. That one has version 1.0.0. On both computers I see the issues described above.

Artifacts:  I also had the impression that the artifacts were related to the DEM. In my case they appeared especially in cloudy regions. Attached are the following images (which are sections from the Landsat8 scene I mentioned before): one example of what I think are artifacts in RGB "artefact_RGB.png". The same section of the image combining bands 1, 2 and 3 ("artefacts_bands123"). The SRTM DEM for the same region. There are no high mountains in this scene (it's Germany/Poland and close to the sea).

Hope this helps. Many thanks!
Silke

[Pete Bunting [pfb]]

Hi Silke,

Thanks for this, I can see what you mean but I haven’t found an example of this in the scenes I’m looking at. Can you send me the scene file name so I can download the same one or send a link for me to download the file?

You can also try the very latest version (1.1.0) which I have not my development conda channel. To use create a new conda environment (http://conda.pydata.org/docs/using/envs.html#create-an-environment):
conda create --name arcsitest python=3.5
source activate arcsitest
conda install -c conda-forge -c au-eoed arcsi

Using this environment you can test the updated version without removing your old version. This new version includes does a linear interpolation between the steps in the LUT so if it is the DEM based LUT causing this issue this should fix it.

Best wishes,

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
*
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”

> --
> You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
> To post to this group, send email to rsgislib...@googlegroups.com.
> Visit this group at https://groups.google.com/group/rsgislib-support.
> For more options, visit https://groups.google.com/d/optout.

> <artefacts_bands123.PNG><artefacts_RGB.PNG><SRTM_dem.PNG>

[silke.and...@bayer.com]

Hi Pete,

the sceenshots come from the Landsat 8 scene LC81920232015109LGN00 which I downloaded from earthexplorer. See attachment, I marked the edge where the artifacts show up.

Best
Silke

[silke.and...@bayer.com]

... and I installed ARCSI v1.1.0. The installation worked (proj.4 error seems to be gone), but I get errors when executing it:

Start Time: 06:21:51, Thu Jan 01 2017. Reading header file Extracting Header Values
ERROR 4: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 4: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.  Input imagery as listed in header file is present.  Get aero profile from image...
ERROR 4: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files. Converting Image Projection ERROR 6: No translation for an empty SRS to PROJ.4 format is known. Input Header: './LS8/Inputs/LC81920232015109LGN00/LC81920232015109LGN00_MTL.txt' Error: Coordinate System Transformation failed (Min).
ARCSI took in less than a second to process the input image. Thank you for using ARCSI

Best wishes
Silke

[Pete Bunting [pfb]]

Hi Silke,

Try setting the GDAL_DATA variable, run the following within the terminal you are using and then re-run arcsi - if you have a different environment name than my previous email change accordingly.

export GDAL_DATA=~/miniconda3/envs/arcsidev/share/gdal

Best wishes,

Pete


****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
*
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”

[William Ray]

Hi both,

I have run the old and new versions of ARCSI on a LS8 image and still get the same results. I've attached a screenshot of an RGB image of the scene being corrected. The "blue" areas seem to correspond with the 100m steps in the Digital Elevation Model (SRTM).

I can upload the image, DEM and python script for you Pete if you want?

Many Thanks,

Will

[Pete Bunting [pfb]]

Hi William, 

Yes, if you could upload so I can download (i have an ftp server if you need somewhere to upload too) those that would be useful as I can then workout what is going on. 

Strange I haven’t seen that in any of my data… 

Cheers, Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

[Pete Bunting [pfb]]

Hi William and Silke,

I’ve processed those images and I can’t see the artefacts you are getting, can you check the following outputs:

Silke:

https://www.dropbox.com/s/ragow2irosr89vm/LS8_20150419_lat53lon1495_r23p192_vmsk_rad_srefdem.kea?dl=0

William:

https://www.dropbox.com/s/qb3fh4i4itwni72/LS8_20161125_lat1lon1607_r59p182_vmsk_rad_srefdem.kea?dl=0

I assume you are both using Linux for your processing? I am running on a Mac so I will try and test on one of linux systems are well - maybe there is something odd going on there… 

Best wishes, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

[silke.and...@bayer.com]

Hi Pete,

your output looks great and it is very close to my results from April 2016 (the cirrus clouds over the water body on the upper left corner are gone now). And yes, so far I have used Ubuntu 14 computers. I am a bit clueless now... maybe I will re-do everything from scratch another time. Maybe try Ubuntu 16... There is one more difference, which is this small gap in my January2017 output (see attached images) which is not there in your output (and which is not in the DEM).

Best wishes
Silke

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsub...@googlegroups.com.

To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.

To post to this group, send email to rsgisli...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[William Ray]

Hi Pete,

Thanks for testing this, I can see your output looks a lot better than my equivalent one!

I'm using my Linux virtual machine, but I will test this on my Mac at home and see if there is any change.

Many Thanks,

Will

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsub...@googlegroups.com.

To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.

To post to this group, send email to rsgisli...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[silke.and...@bayer.com]

Hi Pete and Will,

maybe one more hint: the AOT seems to run into the upper limit during my computation (it's something like 0.600000001 if I specify --maxaot 0.6). I have increased it to --maxaot 1.0 and again it runs into the limit. Also the search range appears to be wrong. I am attaching a screenshot with more details.

Best wishes
Silke

[Pete Bunting [pfb]]

Hi Silke and William, 

I have tried these images on Linux and I get exactly the same results (i.e., pixel values are identical). I am therefore at a bit of loss as to why you are getting different results. 

Therefore, can you confirm which versions of all the packages and dependencies you have installed? The easiest way is if you run the following within the conda environment you run arcsi.

conda list > pkgs.txt

and just email me the pkgs.txt I can then compare to what I have setup and maybe replicate your setup. 

Many thanks, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

<arcsi_AOT.PNG>

[Pete Bunting [pfb]]

HI Silke and all, 

With the help of Will I think I have got to the bottom on what is going on here. 

We’d updated 6S to 2.1 when we moved over to using conda-forge and rios channels back in September and this (at least the version we’ve compiled) doesn’t seem to work. On my dev channel I had gone back to 6S 1.1 and I think that is causing the issues. 

Therefore, if you are using ARCSI I strongly recommend that you run the following commands within your environment:

conda remove pysolar py6s sixs arcsi

conda install -c conda-forge -c au-eoed arcsi

You should now have 6S 1.1 on your system and the artefacts Will and Silke had identified should have been removed. 

Many thanks, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

[silke.and...@bayer.com]

Hi Pete and Will,

sorry for my slow response, I was still busy moving everything to Ubuntu16. However, installing the older S6 version as suggested by Pete does the job. Thank you!!

Out of curriosity: in Petes result for the Landsat scene the clouds over the water body are removed, in my result this is not the case. Are you precisely using the input line

arcsi.py -s ls8 -p TOA DOSAOTSGL SREF -f KEA --stats -i <.../LC81920232015109LGN00_MTL.txt > -o <...> -t ./tmp/ --dem <...> --minaot 0.05 --maxaot 0.6

or are there any further changes that I have missed? Or does the newer ARCSI (dev) version remove the clouds?

However, many thanks again!
Silke

<arcsi_AOT.PNG>

--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.

To post to this group, send email to rsgisli...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[William Ray]

Hi Silke,

If you want to apply a cloud mask as well you want to add to CLOUDS to the product list (-p). You can also add TOPOSHADOW to create a topographic shadow mask too. 

In addition, you can use METADATA in the product list to generate a metadata file for your processing

Will

[Pete Bunting [pfb]]

Hi Silke, 

As Will said there are new options for applying FMask for cloud masking within the latest version of ARCSI. There are options CLOUDS - for masking clouds, CLEARSKY which will mask the image to large areas of continuous clear sky (i.e., without cloud) and TOPOSHADOW performs a ray tracing with the DEM to find topographic shadowing - however this can be slow. If you add all these options it can take about 2 hours for a Landsat scene. 

With regards to your scene, I don't know. I was a little confused by that. It looks to me like the scene I downloaded was different to yours in some way... USGS have been reprocessing data so I don't know where the issue if here?

Best wishes, 

Pete

Get Outlook for iOS

_____________________________
From: William Ray <william....@gmail.com>
Sent: Wednesday, January 11, 2017 11:13 am
Subject: Re: [rsgislib-support] Unexpected surface reflectances calculated with new arcsi version
To: RSGISLib Support <rsgislib...@googlegroups.com>
Cc: <william....@gmail.com>, Pete Bunting [pfb] <p...@aber.ac.uk>

[silke.and...@bayer.com]

Thanks a lot! I will try this.

Silke

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe...@googlegroups.com.