ARCSI Valid pixel mask does not exist when reprojecting

[Matt Debont]

Hi,

I have been trying to process some Sentinel 2 data, currently just running a small sample with three granules of data, however I am getting an issue when arcsi reprojects the valid pixel mask, I pretty consistently get an error saying the valid pixel mask is not a supported file format;

---

Define re-projected image BBOX.
Output Image BBOX (TL, BR): [( 526830.00,  469910.00), ( 644090.00,  368080.00)]
gdalwarp -t_srs /data/sentinel/2/toolchain/proj4/osgb.wkt -tr 10.0 -10.0 -ot Byte -wt Float32 -te 526830.0 368080.0 644090.0 469910.0 -r near -tap -srcnodata 0 -dstnodata 0 -of KEA -overwrite /data/sentinel/2/output_scr_2/SEN2_20170820_lat54lon081_T31UCV_ORB137_utm31n_valid.kea /data/sentinel/2/output_scr_2/SEN2_20170820_lat54lon081_T31UCV_ORB137_utm31n_osgb_valid.kea
ERROR 4: `/data/sentinel/2/output_scr_2/SEN2_20170820_lat54lon081_T31UCV_ORB137_utm31n_valid.kea' not recognized as a supported file format.
Error: 'Reprojected valid image mask is not present: /data/sentinel/2/output_scr_2/SEN2_20170820_lat54lon081_T31UCV_ORB137_utm31n_osgb_valid.kea'

----

However when looking at the file system I can see the valid.kea file as expected and then running the printed gdalwarp command works (copy paste the command after the process errors). After creating the file manually and re-running the original command arcsi carries on and produces output as expected so I am at a bit of a loss (it still throws the same error about the mask not being a recognized supported file format, but as the file exists on the file system it doesn't complain beyond that).

To me this feels like a process not completely flushing the file to disk before calling the next step, as occasionally the process works without issue, but usually I run into this.

The original command I am pushing in is;

arcsi.py -s sen2 --stats -f KEA --fullimgouts -p RAD SHARP SATURATE CLOUDS TOPOSHADOW STDSREF DOSAOTSGL METADATA --interpresamp near --interp cubic --outwkt /data/sentinel/2/toolchain/proj4/osgb.wkt --projabbv osgb -t /data/sentinel/2/tmp_scr_2/ -o /data/sentinel/2/output_scr_2 --dem /data/sentinel/2/toolchain/dem/dem.kea --multi --inputheader /data/sentinel/2/list_Sentinel2A_20170820.txt

Where the osgb.wkt is just the EPSG:27700 definition in a file;

---

PROJCS["OSGB 1936 / British National Grid",GEOGCS["OSGB 1936",DATUM["D_OSGB_1936",SPHEROID["Airy_1830",6377563.396,299.3249646]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",49],PARAMETER["central_meridian",-2],PARAMETER["scale_factor",0.9996012717],PARAMETER["false_easting",400000],PARAMETER["false_northing",-100000],UNIT["Meter",1]]

---

The software is install on an Ubuntu box in AWS, from a conda install in its own environment, happy to provide more info if something is missing here or would help though.

Thanks,

Matt

[Pete Bunting [pfb]]

Hi Matt,

 

I have updated rsgislib in my conda channel (au-eoed). If you could update and try that.

 

Cheers, Pete

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************

"Please consider the environment before printing this email or any documents attached”

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

--------------------------------------------------------------------
Prifysgol Aberystwyth www.aber.ac.uk
Prifysgol y Flwyddyn ar gyfer Ansawdd Dysgu - The Times & The Sunday Times 2018.

Aberystwyth University www.aber.ac.uk
University of the Year for Teaching Quality - The Times & The Sunday Times 2018.

[Matt Debont]

Hi Pete, Thanks for this so far, as yesterday I had managed to get a working version of arcsi by creating an install with only RSGISLIB installed (and the dependencies for it) with python=3.5 and then manually installing arcsi by checking out the source code from bitbucket and installing manually (version 2.2.24) this got me a working copy of arcsi but I had to manually put in the areosol and atmosphere parameters which as far as I had got to so far was an automatic thing, however today it looks like you have merged what I assume was your mpidev branch and I get a different error again this time with GDAL libraries

-------

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/arcsi/bin/arcsi.py", line 56, in <module>
    import arcsilib.arcsirun
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/site-packages/arcsilib/arcsirun.py", line 61, in <module>
    from arcsilib.arcsiutils import ARCSISensorFactory
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/site-packages/arcsilib/arcsiutils.py", line 46, in <module>
    import osgeo.gdal as gdal
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/site-packages/osgeo/__init__.py", line 21, in <module>
    _gdal = swig_import_helper()
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/site-packages/osgeo/__init__.py", line 17, in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: /home/ubuntu/miniconda3/envs/arcsi/lib/python3.5/site-packages/osgeo/../../../libgdal.so.20: undefined symbol: _ZN6kmldom5ParseERKSsPSs

-----

So something still is a bit off with the dependencies somewhere (I have also checked out the environment as a whole with conda install -c au-eoed -c conda-forge arcsi, which seems to install at least now), have tried stepping back through versions of gdal a little but start getting other errors with poppler and geos libraries.

Current gdal version is 2.2.3 and I can provide more info if it helps, still a bit new to all this so help is greatly appreciated.

Many Thanks,
Matt

To post to this group, send email to rsgisli...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[Pete Bunting [pfb]]

Hi Matt,

 

I merged the development branch into the default branch yesterday and merged the Pull request from Dan Clewley with the fix for the latest ESA header format change. This version is 3.1.6 (when I updated to support processing with MPI and bumped the development version from 2.X to 3.X) so I used this version (3.1.5) for processing the UK Sentinel-2 data therefore you should be using the 3.1.6 version of ARCSI and the latest version of rsgislib is 3.5.3 (you can check version using rsgis-config --version).

 

Surprised the conda install -c au-eoed -c conda-forge rsgislib into a new environment doesn’t work. I am also using GDAL 2.2.3. You might need to run conda install -c conda-forge openblas after installing arcsi / rsgislib, there is a dependency missing from a package somewhere on that. What error do you get when you still arcsi using the au-eoed channel into a new environment?

 

Cheers, Pete

 

 

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK
* 
* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743

* Email: pete.b...@aber.ac.uk

To post to this group, send email to rsgislib...@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[Gwawr Jones]

Hi Pete,

*Message from Felix Mason*

I'm working with Matt on this and i've picked up from where he left off. I've set up a new environment from scratch following matts instructions and also got the GDAL error outlined above. As part of this I have pulled the latest code from the default branch and installed it manually into my environment. I was able to resolve the GDAL error, or at least mask it with another one, by running conda update --all. This included a update for gdal. Unforntuanlty when I run arcsi as specified by matt i now get the following error: 

Traceback (most recent call last): File "/home/ubuntu/miniconda3/envs/rsgislib2/bin/arcsibuildmultifilelists.py", line 151, in <module> arcsiObj.buildCmds(args.input, args.output, args.header, args.depth, args.sensor) 

File "/home/ubuntu/miniconda3/envs/rsgislib2/bin/arcsibuildmultifilelists.py", line 97, in buildCmds sensorClass = sensorFact.getSensorClassFromName(sensor, False, None) File "/home/ubuntu/miniconda3/envs/rsgislib/lib/python3.5/site-packages/arcsilib/arcsiutils.py", line 542, in getSensorClassFromName from arcsilib.arcsisensorsentinel2 import ARCSISentinel2Sensor 

File "/home/ubuntu/miniconda3/envs/rsgislib/lib/python3.5/site-packages/arcsilib/arcsisensorsentinel2.py", line 44, in <module> from .arcsisensor import ARCSIAbstractSensor 

File "/home/ubuntu/miniconda3/envs/rsgislib/lib/python3.5/site-packages/arcsilib/arcsisensor.py", line 67, in <module> import rsgislib.imagecalc 

File "/home/ubuntu/miniconda3/envs/rsgislib/lib/python3.5/site-packages/rsgislib/imagecalc/__init__.py", line 6, in <module> from ._imagecalc import * ImportError: libboost_filesystem.so.1.65.1: cannot open shared object file: No such file or directory 

I should point out that actually my interest is in processing multiple files. So at the moment I'm focussed on gettting the output of the arcsibuildmultifilelists.py command. Running this command also results in the same error as above, so it would seem the arcsi package as a whole is affected.

Thanks,

Felix

[Daniel Clewley]

Hi Gwawr and Felix,

I think the conda-forge builds of RSGISLib need to be updated for a different version of boost, there is a pull request open for it:

https://github.com/conda-forge/rsgislib-feedstock/pull/9

Should be fixed in the next couple of days - will post when it is for you to test. 

I did have a quick look see if there was a work around by forcing package versions but didn’t have any luck.

I’m not quite sure why the conda-forge builds keep breaking, I guess it just needs someone to keep on top of it.

Thanks,

Dan

[Daniel Clewley]

Hi,

conda-forge binaries for RSGISLib have just been updated so everything should be working OK now. Let me know any problems.

Thanks,

Dan