Installing ARCSI on a minimal Ubuntu OS (Docker)

[Matt Debont]

Hi, I have been working with a few of my colleagues to try and create a containerized version of ARCSI via Docker, have managed to get most of the dependencies sorted via a combination of trial and error using a standard conda install;

conda install -c conda-forge arcsi

conda update -c conda-forge --all

I am using the following command to start the process off;

arcsi.py -s sen2 --stats -f KEA --fullimgouts -p RAD SHARP SATURATE CLOUDS TOPOSHADOW STDSREF DOSAOTSGL METADATA FOOTPRINT --interp near --outwkt /data/proj/osgb.wkt --projabbv OSGB -t /data/tmp/ -o /data/outputs/ --dem /data/dtm/DTM_UK_10m_OSGB_CompImg.kea -i /data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/MTD_MSIL1C.xml

It looks like I am getting lost in the Cloud masking section, but the error message itself is coming back as blank so hard to tell exactly what is happening.

Perform Cloud Classification...
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
Processing input file /data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/GRANULE/L1C_T30UWE_A015764_20180629T112537/IMG_DATA/T30UWE_20180629T112111_B01.jp2.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
Processing input file /data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/GRANULE/L1C_T30UWE_A015764_20180629T112537/IMG_DATA/T30UWE_20180629T112111_B09.jp2.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
Processing input file /data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/GRANULE/L1C_T30UWE_A015764_20180629T112537/IMG_DATA/T30UWE_20180629T112111_B10.jp2.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
New image width = 10825 height = 11134 bands = 3
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
There are 2 images to stack
/data/outputs/SEN2_20180629_lat54lon217_T30UWE_ORB037_utm30n_OSGB_vmsk_sharp_rad_toa.kea
/data/tmp/SEN2_20180629_lat54lon217_T30UWE_ORB037_utm30n_OSGB_clouds/SEN2_20180629_lat54lon217_T30UWE_ORB037_utm30n_OSGB_clouds_B01B09B10Rescaled.kea
New image width = 10825 height = 11134 bands = 13
Max. block size: 256
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
New image width = 10825 height = 11134 bands = 13
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Cloud layer, pass 1
  Twater= None Tlow= None Thigh= None NIR_17= 0.24
Cloud layer, pass 2
  landThreshold= 0.77
Cloud layer, pass 3
Potential shadows
Clumping clouds
Making 3d clouds
Input Header: '/data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20180629_lat54lon217_T30UWE_ORB037_utm30n_OSGB'
Error:
Error: The following products were not generated:
SREF DOSAOTSGL STDSREF METADATA CLOUDS TOPOSHADOW
Input Header: '/data/inputs/S2A_MSIL1C_20180629T112111_N0206_R037_T30UWE_20180629T151157.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20180629_lat54lon217_T30UWE_ORB037_utm30n_OSGB'
ARCSI took 02:16:08 to process the input image. Thank you for using ARCSI.

Have done a few trials with this (force installing python-fmask as that has been an issue in the past / copy pasting the conda env from a working copy on a more 'full fat' OS install) and just come back to this same point, so I am definitely missing something but as the error is blank it becomes hard to say exactly what is missing here.

As the error always happens after the 'Making 3d clouds' step it follows that its the next step that has trouble but I could not find that particular line in RSGISLib or ARCSI so I guess its in one of the dependencies underneath somewhere which makes this a bit difficult to nail down from the outside. If anyone has any ideas on what may be missing they would be greatly appreciated.

Many Thanks,

Matt

[Peter Bunting]

Hi Matt. 

Could you run with --debug?

Best wishes, 

Pete

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.

[Markus Neteler]

On Tue, Jul 3, 2018 at 5:11 PM, Matt Debont <matt....@gmail.com> wrote:
> Hi, I have been working with a few of my colleagues to try and create a
> containerized version of ARCSI via Docker,

You may take a look at our existing Docker and re-use the related Dockerfile:

https://github.com/mundialis/docker-arcsi

Best,
Markus

--
Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

[Matt Debont]

Hi Pete,

Thanks for letting me know about the --debug argument, completely missed it (guess I was just looking for verbose).

It brought up the error, looks like a memory error rather than some missing OS package, managed to get past that stage and processing the rest of an input file after increasing the base memory a bit, will let you know if it runs all the way through without error now. Been a bit manic here so sorry for not getting back to you sooner.

Thanks,

Matt

[Matt Debont]

Thanks for the heads up, it actually looks pretty similar to what we have already done with a few extras.

Cheers,

Matt