not enough memory to run prepareCal (?)

[lasu...@gmail.com]

Hi,

I have finally installed and run the segmentation using segmentation_ArcGIS_MShift successfully. I am trying to run the prepareCal function but apparently I have memory issues (which I really don't think so)...

memory.size()

[1] 458.66

memory.limit()

[1] 65365

The error highlighted in green 

> calData <- prepareCalData(rstSegm = segmRst,

+                           trainData = trainDataRst,

+                           rstFeatures = classificationFeatures,

+                           thresh = 0.5, 

+                           funs = c("mean", "sd"),

+                           bylayer = TRUE,

+                           minImgSegm = 30, verbose = TRUE, progressBar = TRUE)

-> [1/3] Loading train data into image segments...

done.

-> [2/3] Calculating feature statistics for all image segments...

Error in calculateSegmentStats(rstFeatures = rstFeatures, rstSegm = rstSegm,  : 

  Not enough memory to process the input raster files when using option bylayer!

Warning message:

In prepareCalData(rstSegm = segmRst, trainData = trainDataRst, rstFeatures = classificationFeatures,  :

  An error occurred while calculating segmentation statistics for feature data!

Is there something I can do to fix this? 

Thanks in advance for your help :) 

Cheers,

Angelica

[João Gonçalves]

Hi Angelica,

Thanks for using SegOptim and sorry for this inconvenience. SegOptim was initially designed to handle relatively 'small' images in memory for optimizing segmentation parameters and as such very large images can cause this sort of errors. This is something that I am currently improving to make the package more memory friendly. This will take a couple of weeks more to implement in code.

The error you are getting is being thrown by the following R code which is veryfying if your system has enough memory to hold two copies of raster segments plus one layer from raster features dataset (used for classification), here:

if(!raster::canProcessInMemory(raster::stack(rstSegm, rstFeatures[[1]]), n = 2))
    stop("Not enough memory to process the input raster files when using option bylayer!")

So if you are having this issue it is highly likely that your image data is quite large and therefore available memory in your system is not enough. Using a machine with greater memory resouces may help to overcome this issue. Other options to save memory are to use an integer-like raster datatype (by conversion and rounding or truncation) or down-sampling the original images (if feasible).

 

Can you please tell me more info about the size of the images and the amount of memory available in your system? Use print to output raster metadata and please past it here:

print(segmRst)
print(trainDataRst)
print(classificationFeatures)



Cheers

João

- - -

[João F Gonçalves]

Hi Angélica,

Thanks for the feedback.

Indeed these images (with roughly 280 million pixels) will require processing by tiles which is currently under development in SegOptim. I will keep you posted on advances on this front. Currently these features are being implemented and tested on the experimental build of the package but not yet released.

Best regards,

João

- - -

Hi,

This is what I got...

> print(segmRst)
class      : RasterLayer
dimensions : 15105, 18540, 280046700  (nrow, ncol, ncell)
resolution : 0.3, 0.3  (x, y)
extent     : 437390.4, 442952.4, 6174744, 6179276  (xmin, xmax, ymin, ymax)
crs        : +proj=utm +zone=55 +south +ellps=GRS80 +units=m +no_defs
source     : C:/Scripts/SegOptim/data/segmRaster_MS_R.tif
names      : segmRaster_MS_R
values     : 1, 624470  (min, max)

> print(trainDataRst)
class      : RasterLayer
dimensions : 15105, 18540, 280046700  (nrow, ncol, ncell)
resolution : 0.3, 0.3  (x, y)
extent     : 437390.4, 442952.4, 6174744, 6179276  (xmin, xmax, ymin, ymax)
crs        : +proj=utm +zone=55 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs
source     : C:/Scripts/SegOptim/data/TRAIN_AREAS/Citrus_NSW_Calibration_TCA_Training_v3.tif
names      : Citrus_NSW_Calibration_TCA_Training_v3
values     : 0, 1  (min, max)

> print(classificationFeatures)
class      : RasterStack
dimensions : 15105, 18540, 280046700, 22  (nrow, ncol, ncell, nlayers)
resolution : 0.3, 0.3  (x, y)
extent     : 437390.4, 442952.4, 6174744, 6179276  (xmin, xmax, ymin, ymax)
crs        : +proj=utm +zone=55 +south +ellps=GRS80 +units=m +no_defs
names      :         NDI_1,         NDI_2,         NDI_3,         NDI_4,         NDI_5,         NDI_6,         NDI_7,         NDI_8,         NDI_9,        NDI_10,        NDI_11,        NDI_12,        NDI_13,        NDI_14,    SpecBand_1, ...
min values :   -0.99122804,   -0.97282606,   -0.86203539,   -1.77049184,   -0.86842108,   -0.74736840,   -0.77639753,   -0.82300884,   -8.74870396,  -15.48746777,   -0.31707317, -376.00000000,    0.02717391,    0.09708738,  138.00000000, ...
max values :     0.5231054,     3.5411763,    28.9137459,     0.5206897,     0.6102418,     0.5440932,     0.9961832,     0.9976190,    33.7906075,    25.8950958,     0.9941691,   376.0000000,     4.5411763,   839.0000000,   692.0000000, ... 

Regards,

L.A. Suarez

Gracias, Gracias, Gracias

[Karim Mehour]

Hello,

I'm posting on this thread because I encountered the same problem, with the same error as Angelica while running the  prepareCalData() with the original size of my image (13500x12371).

I tested  the function with a resampled version of my data (3590 x 3158)  and by using the "bylayer" method and it worked flawlessly.

But still, i was wondering if there was any update on the "by tiles" method, or any other solution that would allow to work with the original size.

Thank you.

Karim

[João Gonçalves]

Hi Karim,

Thanks for your feedback. The tiling method for prepareCalData() is already working in the experimental version of SegOptim. To use this version of the package, you will have to reinstall the it following the instructions here.

Then, in the prepareCalData function you will have available the tiles parameter defined as the:

Number of times that the image will be divided along the x and y axes. This means that the original raster data will be split into a number of blocks equal to tiles^2 (e.g., if tiles = 5 this will generate 25 blocks/tiles) for reading. This number should be larger for large Raster* objects. This means that some fine tuning may be necessary to adjust this value according to available memory and raster input size.

Try it out and let me know how it works for you.

Cheers

João Gonçalves

- - -