arcsi.py Required argument 'format' (pos 3) not found

144 views
Skip to first unread message

vinay...@digital.cabinet-office.gov.uk

unread,
Jun 20, 2017, 4:16:59 AM6/20/17
to RSGISLib Support
I am trying to generate S2 ARD using the command below but get following message and no products are created "Required argument 'format' (pos 3) not found"

I am running this on ubuntu 14.04.

arcsi.py -s sen2 -i ./S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml --stats -f GTIFF -o ./Outputs -p RAD TOA CLOUDS
 DOS SREF DOSAOTSGL METADATA --resample2lowres --interp average -t /tmp/ --dem ./Output.tif

Console Log

Start Time: 13:58:47, Mon Jun 06 2017.
Reading header file
Found Sentinel-2 image file product 'S2MSI1C'

Input imagery as listed in header file is present.

Get aero profile from image...
Aerosol Profile =  Maritime

Get atmos profile from image...
Northern Hemisphere
Atmosphere Profile =  MidlatitudeSummer

Image Base Name: SEN2_20170611_lat52lon07_T30UYC

Checking Input Images are valid
Resampling image bands to match one another.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Generate valid image mask
Input Header: './S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170611_lat52lon07_T30UYC'
Error: Required argument 'format' (pos 3) not found
Error: The following products were not generated:
RAD METADATA SREF DOS DOSAOTSGL TOA SATURATE CLOUDS
Input Header: './S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170611_lat52lon07_T30UYC'



ARCSI took 00:07:26 to process the input image. Thank you for using ARCSI.

Pete Bunting [pfb]

unread,
Jun 20, 2017, 4:44:10 AM6/20/17
to vinay...@digital.cabinet-office.gov.uk, RSGISLib Support
Hi Vinay, 

What version of rsgislib and arcsi do you have installed? ARCSI and corresponding functions in RSGISLib are under quite a bit of development at the moment so I’m updating the versions on my development conda channel, where you got these binaries from, quite regularly. 

To check you versions run the following:

arcsi.py --version
ARCSI 1.4.2

rsgis-config --version
3.2.1361

If you need to update then run:

conda update -c conda-forge -c au-eoed rsgislib arcsi

If you still get this error then please provide the Sentinel-2 download and I will debug. 

As an FYI, I would suggest that you run the analysis with the output format as KEA as I still haven’t had chance to deal with the issues when running with other formats. KEA supports functions which some other formats don’t support (e.g., attribute tables) and I use these within ARCSI. I haven’t tested running with another format in a while so you might encounter issues. KEA also produces smaller file sizes as well. Personally, I do my processing in KEA and then, if I need a different format, I convert using gdal_translate. 

Best wishes, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK

* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”

--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.



--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

Vinay Patel

unread,
Jun 20, 2017, 8:50:26 AM6/20/17
to Pete Bunting [pfb], RSGISLib Support
Hi Pete

updated the application now getting different error.

Console Output

Start Time: 12:34:47, Tue Jun 06 2017.
Reading header file
Found Sentinel-2 image file product 'S2MSI1C'

Input imagery as listed in header file is present.

Get aero profile from image...
Aerosol Profile =  Maritime

Get atmos profile from image...
Northern Hemisphere
Atmosphere Profile =  MidlatitudeSummer

Image Base Name: SEN2_20170611_lat52lon07_T30UYC

Checking Input Images are valid
Resampling image bands to match one another.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Generate valid image mask
0) ./Outputs/SEN2_20170611_lat52lon07_T30UYC_B02_20m.kea
1) ./Outputs/SEN2_20170611_lat52lon07_T30UYC_B03_20m.kea
2) ./Outputs/SEN2_20170611_lat52lon07_T30UYC_B04_20m.kea
3) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B05.jp2
4) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B06.jp2
5) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B07.jp2
6) ./Outputs/SEN2_20170611_lat52lon07_T30UYC_B08_20m.kea
7) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B8A.jp2
8) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B11.jp2
9) /home/ubuntu/S2-Files/S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/GRANULE/L1C_T30UYC_A010287_20170611T111012/IMG_DATA/T30UYC_20170611T110621_B12.jp2
New image width = 5490 height = 5490 bands = 1
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100..110. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100..110. Complete.
Started.0..10..20..30..40..50..60..70..80..90. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Get Image Min and Max.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Get Image Histogram.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Adding Histogram and Colour Table to image file
./ARD.sh: line 1: 15104 Segmentation fault      (core dumped) arcsi.py -s sen2 -i ./S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml --stats -f GTIFF -o ./Outputs -p RAD TOA CLOUDS DOS SREF DOSAOTSGL METADATA --resample2lowres --interp average -t /tmp/ --dem ./Output.tif


But it has generated the following files in the output directory

-rw-rw-r-- 1 ubuntu ubuntu  36034269 Jun 20 12:36 SEN2_20170611_lat52lon07_T30UYC_B02_20m.kea
-rw-rw-r-- 1 ubuntu ubuntu  36539107 Jun 20 12:38 SEN2_20170611_lat52lon07_T30UYC_B03_20m.kea
-rw-rw-r-- 1 ubuntu ubuntu  37224344 Jun 20 12:40 SEN2_20170611_lat52lon07_T30UYC_B04_20m.kea
-rw-rw-r-- 1 ubuntu ubuntu  38743349 Jun 20 12:42 SEN2_20170611_lat52lon07_T30UYC_B08_20m.kea
-rw-rw-r-- 1 ubuntu ubuntu  30184380 Jun 20 12:44 SEN2_20170611_lat52lon07_T30UYC_valid.tif
-rw-rw-r-- 1 ubuntu ubuntu       389 Jun 20 12:44 SEN2_20170611_lat52lon07_T30UYC_viewangletmp.kea.aux.xml
-rw-rw-r-- 1 ubuntu ubuntu 241209774 Jun 20 12:44 SEN2_20170611_lat52lon07_T30UYC_viewangle.tif


Regards

Vinay


To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe@googlegroups.com.
To post to this group, send email to rsgislib-support@googlegroups.com.


--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk



--
Regards

Vinay

 
Cabinet Office - Government Digital Service

​Vinay Patel


Government Digital Service, Cabinet Office
Aviation House​, 3rd
 Floor, 
125 Kingsway, London WC2B 6NH
M: 
+44 7956 131984 



Pete Bunting [pfb]

unread,
Jun 20, 2017, 11:59:50 AM6/20/17
to Vinay Patel, RSGISLib Support
Dear Vinay, 

You need to set the output file format to KEA - see my earlier email. 

Best wishes, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK

* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”

vinay...@digital.cabinet-office.gov.uk

unread,
Jun 28, 2017, 11:09:53 AM6/28/17
to RSGISLib Support
Hi Pete

thrown another type of error:

Input Header: './S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170611_lat52lon07_T30UYC'
Error:
Error: The following products were not generated:
DOSAOTSGL METADATA CLOUDS DOS SREF
Input Header: './S2A_MSIL1C_20170611T110621_N0205_R137_T30UYC_20170611T111012.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170611_lat52lon07_T30UYC'

I've added file console.txt which is the console output.

can't upload S2 file as returns an error when attempting to add

Vinay
console.txt

Daniel Clewley

unread,
Jun 30, 2017, 4:45:42 AM6/30/17
to vinay...@digital.cabinet-office.gov.uk, RSGISLib Support
Hi Vinay,

I’m not sure what the problem is but to debug I would edit the file arcsilib/arcsirun.py and get it to raise the exception where it being caught at line 1339, as shown below (change in bold):

1339     except Exception as e:
                 raise
1340         print('Input Header: \'' + inputHeader + '\'', file=sys.stderr)
1341         if (paramsObj is not None) and (paramsObj.outBaseName is not None):
1342             print('Output Basename: \'' + paramsObj.outBaseName + '\'', file=sys.stderr)
1343         print(“Error: {}".format(e), file=sys.stderr)

If you make this change run again it should print more information about where the error is occurring.

You can find out where the file is installed using something like:

python -c “from arcsilib import arcsirun; print(arcsirun.__file__.lstrip(‘c’))”

Thanks,

Dan


--
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.
<console.txt>

Vinay Patel

unread,
Jul 2, 2017, 2:52:18 PM7/2/17
to Daniel Clewley, RSGISLib Support
Hi Dan

I found the file (using unix command find) but Unusually it only had 1200 lines, but I think I located the exception code in the file, below is the output from the run

Start Time: 18:00:29, Sun Jul 07 2017.
Reading header file
Found Sentinel-2 image file product 'S2MSI1C'

Input imagery as listed in header file is present.

Get aero profile from image...
Aerosol Profile =  Maritime

Get atmos profile from image...
Northern Hemisphere
Atmosphere Profile =  MidlatitudeSummer

Image Base Name: SEN2_20170621_lat52lon07_T30UYC

Checking Input Images are valid
Resampling image bands to match one another.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Generate valid image mask
0) ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B02_20m.kea
1) ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B03_20m.kea
2) ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B04_20m.kea
3) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B05.jp2
4) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B06.jp2
5) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B07.jp2
6) ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B08_20m.kea
7) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B8A.jp2
8) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B11.jp2
9) /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B12.jp2
New image width = 5490 height = 5490 bands = 1
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100..110. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100..110. Complete.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Get Image Min and Max.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Get Image Histogram.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Adding Histogram and Colour Table to image file
Calculating Image Pyramids.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Calculating Image Pyramids.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.

Generate Saturation Image
Opening: ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B02_20m.kea
Opening: ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B03_20m.kea
Opening: ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B04_20m.kea
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B05.jp2
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B06.jp2
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B07.jp2
Opening: ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_B08_20m.kea
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B8A.jp2
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B11.jp2
Opening: /home/ubuntu/S2-Files/S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/GRANULE/L1C_T30UYC_A010430_20170621T111222/IMG_DATA/T30UYC_20170621T110651_B12.jp2
New image width = 5490 height = 5490 bands = 10
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Calculating Statistics...
Get Image Min and Max.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Calculating Image Pyramids.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.

Converting to Radiance
New image width = 5490 height = 5490 bands = 10
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Masking to valid data area.
New image width = 5490 height = 5490 bands = 10
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Deleting file: ./Outputs2/SEN2_20170621_lat52lon07_T30UYC_rad.kea
Setting Band Names...
Calculating Statistics...
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Calculating Image Pyramids.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.

Converting to TOA
New image width = 5490 height = 5490 bands = 10
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Setting Band Names...
Calculating Statistics...
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Calculating Image Pyramids.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.
Started .0..10..20..30..40..50..60..70..80..90..100. Complete.

Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
Started.0..10..20..30..40..50..60..70..80..90..100. Complete.
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
New image width = 5490 height = 5490 bands = 3
Started .0..10..20..30..40..50..60..70..80..90. Complete.
There are 2 images to stack
./Outputs2/SEN2_20170621_lat52lon07_T30UYC_vmsk_rad_toa.kea
/tmp/SEN2_20170621_lat52lon07_T30UYC_clouds/SEN2_20170621_lat52lon07_T30UYC_clouds_B01B09B10Rescaled.kea
New image width = 5490 height = 5490 bands = 13
Max. block size: 256
Started.0..10..20..30..40..50..60..70..80..90. Complete.
New image width = 5490 height = 5490 bands = 13
Started .0..10..20..30..40..50..60..70..80..90. Complete.
Cloud layer, pass 1
  Twater= None Tlow= None Thigh= None NIR_17= 0.264
Cloud layer, pass 2
  landThreshold= 0.95
Cloud layer, pass 3
Potential shadows
Error: The following products were not generated:
DOSAOTSGL METADATA DOS SREF CLOUDS
Input Header: './S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170621_lat52lon07_T30UYC'



Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/bin/arcsi.py", line 595, in <module>
    args.cloudtrainother, args.resample2lowres)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsirun.py", line 762, in runARCSI
    cloudsImage = sensorClass.generateCloudMask(toaImage, saturateImage, thermalBrightImage, validMaskImage, outFilePath, outName, outFormat, tmpPath, scaleFactor)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsisensorsentinel2.py", line 1231, in generateCloudMask
    fmask.doFmask(fmaskFilenames, fmaskConfig)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/fmask/fmask.py", line 157, in doFmask
    potentialShadowsFile = doPotentialShadows(fmaskFilenames, fmaskConfig, NIR_17)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/fmask/fmask.py", line 691, in doPotentialShadows
    NIR_filled = scaledNIR_filled.astype(numpy.float) / fmaskConfig.TOARefScaling
MemoryError


Regards

Vinay

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe@googlegroups.com.
To post to this group, send email to rsgislib-support@googlegroups.com.

Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.
<console.txt>




--
Regards

Vinay

 
Cabinet Office - Government Digital Service

​Vinay Patel


Government Digital Service, Cabinet Office
The White Chapel Building
10 Whitechapel High Street
London E1 8QS
M: 
+44 7956 131984 



Pete Bunting [pfb]

unread,
Jul 3, 2017, 8:57:37 AM7/3/17
to Vinay Patel, Dan Clewley, RSGISLib Support
Dear Vinay, 

I have tested with the scene you are using and I haven’t been able to reproduce the error. However, I have been continuing to develop arcsi for Sentinel-2 so the latest version is 2.2.2. Update with the following command:

arcsi.py update -c conda-forge -c au-eoed arcsi rsgislib

Best wishes, 

Pete

****************************************************
* Dr Pete Bunting
* Reader in Remote Sensing
* Earth Observation and Ecosystem Dynamics Group
* Department of Geography and Earth Sciences
* Aberystwyth University
* Aberystwyth
* Ceredigion
* SY23 3DB
* UK

* Ph: +44 (0) 1970 622615
* Mob: +44 (0) 7917 842743
* Email: p...@aber.ac.uk
* ORCID: http://orcid.org/0000-0002-7435-0148
****************************************************
"Please consider the environment before printing this email or any documents attached”

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.

Vinay Patel

unread,
Jul 5, 2017, 9:24:26 AM7/5/17
to Pete Bunting [pfb], Dan Clewley, RSGISLib Support
Hi Pete

I think I have shared library issue, as my ubuntu 16.04 is a fresh install I think there are additional packages which need to be manually installed e.g. sudo apt-get install libopenblas-base

Runtime error points to missing library libCGAL.so.13 

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/bin/arcsi.py", line 54, in <module>
    from arcsilib.arcsirun import ARCSIRun
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsirun.py", line 65, in <module>
    from arcsilib.arcsisensor import ARCSIAbstractSensor
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsisensor.py", line 67, in <module>
    import rsgislib.imagecalc
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/rsgislib/imagecalc/__init__.py", line 6, in <module>
    from ._imagecalc import *
ImportError: libCGAL.so.13: cannot open shared object file: No such file or directory
 
Regards

Vinay

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsub...@googlegroups.com.

To post to this group, send email to rsgislib-support@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.
<console.txt>



-- 
Regards

Vinay

 
Cabinet Office - Government Digital Service

​Vinay Patel


Government Digital Service, Cabinet Office
The White Chapel Building
10 Whitechapel High Street
London E1 8QS
M: 
+44 7956 131984 




-- 
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsub...@googlegroups.com.

To post to this group, send email to rsgislib-support@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.
--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr.
(Arolwg Cenedlaethol y Myfyrwyr 2016)
www.aber.ac.uk

Top 4 UK university and best in Wales for student satisfaction
(National Student Survey 2016)
www.aber.ac.uk

vinay...@digital.cabinet-office.gov.uk

unread,
Jul 7, 2017, 2:17:13 PM7/7/17
to RSGISLib Support, p...@aber.ac.uk, daniel....@gmail.com
Hi Pete,Dan

I managed to overcome the library file error by copying libCGAL.so.11 and naming it libCGAL.so.13, but on executing the application it failed with the same error message, so updated the code as per Dan's instruction, now I get a different message,see below

This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; See website (http://www.rsgislib.org/arcsi).
Bugs are to be reported to rsgislib...@googlegroups.com.

Start Time: 14:54:29, Fri Jul 07 2017.
SREF CLOUDS DOS METADATA DOSAOTSGL
Input Header: './S2A_MSIL1C_20170621T110651_N0205_R137_T30UYC_20170621T111222.SAFE/MTD_MSIL1C.xml'
Output Basename: 'SEN2_20170621_lat52lon07_T30UYC'



Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/bin/arcsi.py", line 595, in <module>
    args.cloudtrainother, args.resample2lowres)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsirun.py", line 762, in runARCSI
    cloudsImage = sensorClass.generateCloudMask(toaImage, saturateImage, thermalBrightImage, validMaskImage, outFilePath, outName, outFormat, tmpPath, scaleFactor)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/arcsilib/arcsisensorsentinel2.py", line 1231, in generateCloudMask
    fmask.doFmask(fmaskFilenames, fmaskConfig)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/fmask/fmask.py", line 157, in doFmask
    potentialShadowsFile = doPotentialShadows(fmaskFilenames, fmaskConfig, NIR_17)
  File "/home/ubuntu/miniconda3/lib/python3.5/site-packages/fmask/fmask.py", line 690, in doPotentialShadows
    NIR = scaledNIR.astype(numpy.float) / fmaskConfig.TOARefScaling
MemoryError


Regards

Vinay

Daniel Clewley

unread,
Jul 10, 2017, 2:03:54 AM7/10/17
to vinay...@digital.cabinet-office.gov.uk, RSGISLib Support, Pete Bunting
Hi Vinay,

Both the errors are memory errors - you’ve probably already checked by are you reaching the limit of how much RAM you have just before it fails?

Dan

To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-suppo...@googlegroups.com.
To post to this group, send email to rsgislib...@googlegroups.com.

Vinay Patel

unread,
Jul 26, 2017, 8:19:36 AM7/26/17
to Daniel Clewley, RSGISLib Support, Pete Bunting
Hi Dan

sorry for late reply as I have been on leave over the last couple of weeks.
I created another VM with 32GB of memory and 8 vCPU's on AWS and after a number of config issues I have managed to run the process successfully, I just need to get the output checked by domain expert.
I'm now planning to run some test using different size VM's to determine the best performance. 

Thanks 

Vinay 

On 10 July 2017 at 07:03, Daniel Clewley <daniel....@gmail.com> wrote:
Hi Vinay,

Both the errors are memory errors - you’ve probably already checked by are you reaching the limit of how much RAM you have just before it fails?

Dan
On 7 Jul 2017, at 19:17, vinay...@digital.cabinet-office.gov.uk wrote:

Hi Pete,Dan

I managed to overcome the library file error by copying libCGAL.so.11 and naming it libCGAL.so.13, but on executing the application it failed with the same error message, so updated the code as per Dan's instruction, now I get a different message,see below

This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; See website (http://www.rsgislib.org/arcsi).
Bugs are to be reported to rsgislib-support@googlegroups.com.


Regards

Vinay
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe...@googlegroups.com.

To post to this group, send email to rsgislib-support@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.
<console.txt>



-- 
Regards

Vinay
 

 
Cabinet Office - Government Digital Service

​Vinay Patel
 

Government Digital Service, Cabinet Office
The White Chapel Building
10 Whitechapel High Street
London E1 8QS
M:  
 
+44 7956 131984  




-- 
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsubscribe...@googlegroups.com.

To post to this group, send email to rsgislib-support@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.


--------------------------------------------------------------------
Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd myfyrwyr. 
(Arolwg Cenedlaethol y Myfyrwyr 2016) 
www.aber.ac.uk 

Top 4 UK university and best in Wales for student satisfaction 
(National Student Survey 2016) 
www.aber.ac.uk



-- 
Regards

Vinay

 
Cabinet Office - Government Digital Service

​Vinay Patel


Government Digital Service, Cabinet Office
The White Chapel Building
10 Whitechapel High Street
London E1 8QS
M: 
+44 7956 131984 




-- 
You received this message because you are subscribed to the Google Groups "RSGISLib Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rsgislib-support+unsub...@googlegroups.com.
To post to this group, send email to rsgislib-support@googlegroups.com.
Visit this group at https://groups.google.com/group/rsgislib-support.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages