iRSFs - model validation and scale of inference

[Daniel Eacker]

Hello,

I've been able to use the ctmm package to fit winter iRSF models for deer and conduct model selection using rsf.select(). I was wondering if there was a recommended way to validate these models such as a k-fold cross-validation routine as is traditionally done in RSF modeling?

I've attempted to do this but I wanted to ask for your advice. It seems that each individual model is based on their utilization distribution, but traditional validation models for a 2nd-order analysis would compare the used values with predicted relative probability of selection across the population-level UD or home range. 

Do you have any recommendations for this? Should we be validating these models separately at the individual UD or somehow combining them to produce a correlation statistic similar to what is traditionally done?

Also, my understanding of the approach is that it is invariant to scale as far as the order of selection. Is this correct?

Thanks,

Dan

 

[Christen Fleming]

Hi Dan,

I would suggest some kind of block cross validation, with the number of blocks based on the effective sample size. I was actually planning on making some utility code to automate the generation of the blocks, which I could implement quickly if someone needs it.

After that, I would probably look at what is considered in the spatstat package for point-process models. A lot of what I see done for RSF and SDM models (like AUC) doesn't make any sense to me with a presence-only point-process model. If there's something that looks good that I can help facilitate, then please let me know.

Best,

Chris

[Daniel Eacker]

Good morning Chris,

Thank you for the speedy response. I haven't done a block cross validation but I should be able to figure it out.

After a quick internet search, I saw this R package called blockCV that could facilitate this perhaps (here is the publication about the package: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13107). If you can work it into the ctmm package, I could see this benefiting a other users conducting similar analyses.

I'll take a look at the spatstat package and I appreciate the suggestion. I still have more to learn about the mechanics of the iRSF models, but my basic understanding is that the auto-correlated KDE surface weights the likelihood in the model.

As far as the effective sample size, I think this is just the mean of the DOF here in the summary? This is a summary for one individual, but I think you would use the effective sample size for the population-level summary? $DOF mean area diffusion speed 61.33980 58.19108 379.60802 59.34827 $CI low est high southing (1/southing) -4.559837 -3.134717 -1.709597 slope (1/slope) 2.338430 3.458393 4.578356 area (hectares) 40.096912 52.779275 67.177636 τ[position] (hours) 11.688161 14.629546 18.311144 τ[velocity] (minutes) 18.958086 27.540194 40.007325 speed (kilometers/day) 1.918477 2.197900 2.476889 diffusion (hectares/day) 7.417580 8.224193 9.071838

Thanks,

Dan

[Christen Fleming]

Hi Dan,

I'll take a look at that more closely, but I don't see a specific test performed. Temporal or spatial blocking is fairly straightforward, but point process cross validation techniques are less obvious.

I was going to use DOF[area] but DOF[mean] is usually pretty close.

Best,

Chris

[Daniel Eacker]

Thanks Chris.

I'll just research it for now and let you know what I find out.

Cheers,

Dan