ENMEval vector storage issue

[Megan Sebasky]

Hello,

I am trying to run ENMEval using 9 rasters at 30m resolution for the entire state of Wisconsin. I've converted each raster to TIF, loaded into R, created a raster stack, and tried running the ENMEval code. I've tried on computers with 32GB, 64GB RAM, and even a server with 500GB RAM, and continue to get errors about allocating vector space (I've tried both running in parallel and not).

This instance, on a 32GB RAM computer, gave the error that it could not write a 13.2GB raster:

eval2<- ENMevaluate(OCCS, Kstack, method='randomkfold', kfolds = 3, RMvalues=c(1,2), fc='LQP', algorithm='maxnet')

On a 64GB RAM computer...

> eval2 <- ENMevaluate(OCCS, Kstack, method='checkerboard2', RMvalues=c(1,2,3), fc=c('L','LQ','LQP','LQPT','LQHPT'), algorithm='maxnet')
*** Running ENMevaluate using maxnet v.0.1.2 ***
Doing evaluations using checkerboard 2...
There are 88 background records with NA for at least one predictor variable.
                  Removing these records from analysis, resulting in 9912 records...
  |=======                                                                                                        |   7%
 1583336 grid cells found with at least one NA value: these cells were excluded from raster predictions.
Error: cannot allocate vector of size 10.7 Gb

and...

> eval3 <- ENMevaluate(OCCS, Kstack, method='checkerboard2', RMvalues=c(1,2,3), fc=c('L','LQ','LQP','LQPT','LQHPT'), algorithm='maxnet', parallel=TRUE)
*** Running ENMevaluate using maxnet v.0.1.2 ***
Doing evaluations using checkerboard 2...
There are 117 background records with NA for at least one predictor variable.
                  Removing these records from analysis, resulting in 9883 records...
Of 16 total cores using 16
Running in parallel...
Error in { : task 1 failed - "Failure during raster IO

On a 500GB RAM server, I tried the below and got the following error message, and 96% of the memory on the server was being used:

> eval4 <- ENMevaluate(OCCS, Kstack, method='randomkfold', kfolds=2, RMvalues=c(1,2), fc=c('L','LQ','LQP'), algorithm='maxnet', parallel=TRUE)
*** Running ENMevaluate using maxnet v.0.1.2 ***
Doing random k-fold evaluation groups...
There are 101 background records with NA for at least one predictor variable.
                  Removing these records from analysis, resulting in 9899 records...
Of 48 total cores using 48
Running in parallel...
Error in { :
  task 2 failed - "long vectors not supported yet: ../include/Rinlinedfuns.h:519"

Is there something I can be doing to make the files smaller or write intermediate files so it doesn't all stay in memory?

Thanks,

Megan

[Tereza]

Hi Megan,

Were you able to solve this issue? I am running into the "long vectors not supported yet" error with ENMeval. Thank you, Tereza

[Lilian Sales]

Hello,

I am running into the same error. Has someone found a way to deal wit this please?

Thank you,

Lilian

[gafna jeff]

Hello Lilian,

Close your R studio, Open it, then run this line ( begins with options: see example below) before you begin running any package. Should be something like what I write below. Hope it works. Am very good at ENMeval package. I used it in my paper (https://doi.org/10.1016/j.ecochg.2023.100070). Let me know if you have any problems.

options(java.parameters = "-Xmx200g" )

library(raster)#raster files
library(MASS)
library(corrplot)

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/6241e59f-7c84-4c71-8081-b69448ef9994n%40googlegroups.com.

[Lilian Sales]

Thanks Jeff,

I will try this code and let you know how it goes.

By the way, congratulations on the paper!  =)

Cheers,

Lilian P. Sales

Ph.D. in Ecology & Evolution

Post-Doctoral Researcher at UBC Okanagan

ResNet Canada

You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/lBQgeLEpHu0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/CAABPRJdPTJkvaWVtzZG-8F8GYvXxLGKBzfcrAWK6EMQFm6DPUg%40mail.gmail.com.

[gafna jeff]

Dear Lilian,

It seems you have a very big study area, and I guess you will still have a memory problem. The BEST option for you is to extract your values from the raster and use those values in building the model instead of running your model on the entire raster of your study area. 

Hope this helps 

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/CAOrG%2BQ75H_Qu7uqUrub7554qP1nH%2Bhw4GzbgrwvGKaNOijhxmg%40mail.gmail.com.

[Thanh Trang Pham]

your studied areas are very big or/and your variables' resolution are fine.

[Thanh Trang Pham]

Hi everyone,

Can you please help me to resolve the problem as shown below? 

Best regards,

Trang

To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/CAABPRJeFK-ocPNJcWzkcX70_cC09d1mJY-OMS3eK4Fi8D8LnTw%40mail.gmail.com.

--

Pham Thanh Trang, 
Lecturer, Department of Forest plant, Faculty of Forest Resource and Environmental Management, Vietnam Forestry University, Xuan Mai, Chuong My, Ha Noi, Viet Nam

Mobile: +84.918.910.986

PhD student, Environmental and Conservation Sciences Department, College of Science, Health, Engineering and Education, Murdoch University, 90 South street, Murdoch, Perth, WA 6150 Australia.

Email: phamtrang...@gmail.com/ tra...@vnuf.edu.vn/ trang...@murdoch.edu.au

[Lilian Sales]

Thanks a lot, Thanh and Jeff!

I extracted the values from the raster and use those values in building the model instead of running the model on the entire raster of my study area, as suggested by Jeff.

Now, the models run perfectly! Thanks!

Lilian P. Sales

Ph.D. in Ecology & Evolution

Post-Doctoral Researcher at UBC Okanagan

ResNet Canada

To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/e6d595f5-9d38-4e27-bca5-1d82a5283c6an%40googlegroups.com.

[Bede-Fazekas Ákos]

Dear Trang,

please open a new thread for a new problem.
Anyway, I guess that
ncol(bg) == 3
and
ncol(occs) == 2
therefore you cannot simply copy the column names.
Please first check the colnames(), and then update only the relevant columns, e.g.
colnames(bg)[1:2] <- colnames(occs)

HTH,
Ákos
_______
Ákos Bede-Fazekas
Centre for Ecological Research, Hungary

To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/CALE7RkOALf_%3Dk6gL22LNN6xe6BKOzm86kS_GrWz%3DHRUzmN2QzA%40mail.gmail.com.

[Thanh Trang Pham]

Dear Askos,

Thank you so much for your help and reminder.

Best regards,

Trang

To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/0d88800c-3ccc-d793-c04c-e6f6a45b0054%40gmail.com.