Error in makegkParalleldcpp : negative length vectors are not allowed

37 views
Skip to first unread message

Olivier Devineau

unread,
Aug 28, 2023, 4:23:45 AM8/28/23
to secr

Hi everyone,

I hope someone can help me debug or at least explain an openCR error...

I am trying to fit a CJSsecr model to non-invasive genetic
sampling data covering a fairly large geographic area. I have been expressly asked to not share these data, so I have to
keep descriptions vague... The attached document contains *some*
summary information about the data which might be useful.

For the purpose of the analysis, I use a 20-km grid of "traps" over the whole
study area, and the following model fitting code :

    mod <- openCR.fit(
      dat,
      type = "CJSsecr",
      model = formX,
      mask = msk,
      detectfn = "HHR",
      movementmodel = "BVT",
      edgemethod = "truncate",
      details = list(
        CJSp1 = TRUE,
      )
    )

Models with a single built-in or user-provided covariate work fine, but as soon
as I include two covariates (for the same parameter, or 'across' parameters), I
get a segmentation fault (ie, R crashes), and / or the following error :

Error in makegkParalleldcpp(as.integer(data$detectfn), as.integer(.openCRstuff$sigmai[type]),  :
  negative length vectors are not allowed


For example, these formula / models work fine :

    form1 <- list(lambda0 ~ 1, sigma ~ 1, z ~ 1, phi ~ 1)
    form2 <- list(lambda0 ~ 1, sigma ~ Session, z ~ Session, phi ~ Session)

But these result in the error :

    form3 <- list(lambda0 ~ 1, sigma ~ 1, z ~ 1, phi ~ forest + Session)
    form4 <- list(lambda0 ~ 1, sigma ~ forest, z ~ 1, phi ~ Session)

The error message makes me think of some kind of too-big-an-object or out-of-memory
problem related to parallel computing, but it also happens if I force the use of
a single core...

If such an error is to be expected given the geographic range and fairly high
number of traps , and/or an inappropriate movement model for example, please let
me know...

I am a bit at a loss with respect to where to look / what to do next to debug or
explain the error... Any help or suggestion would be greatly appreciated!

Thank you in advance,

Olivier



data_info.txt

Murray Efford

unread,
Aug 28, 2023, 3:13:04 PM8/28/23
to secr
You really do have a covariate 'forest'? I see 'frst' in data_info.txt.

Olivier Devineau

unread,
Aug 29, 2023, 2:58:36 AM8/29/23
to secr
 I changed the names in the data_info.txt to gain a bit of space, but the covariates I tried to use in the model formula did match with the covariates in the data.

Murray Efford

unread,
Aug 29, 2023, 3:42:22 PM8/29/23
to secr
OK, but you don't make it easy. I'm not clear whether we're dealing with one error or two. A trap covariate should not work in models for phi. You appear to have covariates with many levels - these can slow things down and demand extra space.  Size alone should not cause the errors, unless the product of the dimensions of an internal array gets ridiculous. Certainly openCR should not crash R - I can debug  given a minimal example that fails (my own example works). There's plenty you can do to make your own tests - drop HHR and BVT for a start. You have an extra comma in your code near the end.

Olivier Devineau

unread,
Aug 30, 2023, 5:00:37 AM8/30/23
to secr
Apologies for the confusion with the variable names. Below are the first few lines of the traps covariate table again. The only covariate with 'many levels' is region, with 21 levels. The other covariates are (standardized) continuous measures of the amount of some habitat within a buffer around the "trap".

[ins] r$> covariates(trpssub) |> head()
region roadddens nat2000prop humdens forest grassland other seminat built croplnd water sheeppdens nneighbours
A      -1.19     -0.46       -0.51   0.78    1.36     -0.28  1.55   -0.71 -1.42   -0.25  0.08      -0.41
A      -1.08     -0.55       -0.52   2.09   -0.03     -0.31  0.08   -0.65 -1.42   -0.25 -0.22      -0.41
A      -0.94     -0.55       -0.46   0.97    0.01      0.00  2.66   -0.47 -1.22   -0.25 -0.24      -0.41
A      -1.13     -0.37       -0.49   2.23   -0.58     -0.31 -0.11   -0.71 -1.42    0.58 -0.21      -0.41
A       0.67     -0.55       -0.41   2.21   -0.30     -0.31 -0.30   -0.66 -1.28   -0.25 -0.30      -0.41
A       0.43     -0.55       -0.37   2.01   -0.77     -0.12 -0.39   -0.15 -1.04   -0.25 -0.36      -0.41
...

Below are a few "test" formula and what they give.
In short, the default detectfn / movement model leads to the likelihood being NA. The HHR detection function works fine with both a static or BVT movement model, if  and only if, the included covariate is the same for both sigma and phi (even with a trap covariate on phi, but see my next comment below about this).  As soon as two different variables are involved one way or another, I get the Error in markgkParalleldcpp. At least it seems it doesn't segfault anymore.


# default detectfn = "HHN", movement model = "static",
list(lambda0 ~ 1, sigma ~ 1, z ~ 1, phi ~ 1)                                                          # --> NA likelihood
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ Session)                       # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ forest)                           # --> NA likelihood
list(lambda0 = ~ 1, sigma = ~ Session, z = ~ 1,  phi = ~ Session)                     # --> NA likelihood
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ Session)      # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ 1)                  # --> error in makegkParalleldcpp

# detectfn = "HHR", movement model = "static",
list(lambda0 ~ 1, sigma ~ 1, z ~ 1, phi ~ 1)                                                           # --> OK
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ forest)                             # --> OK
list(lambda0 = ~ 1, sigma = ~ Session, z = ~ 1,  phi = ~ Session)                      # --> OK
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ Session)                          # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ Session)        # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ 1)                    # --> error in makegkParalleldcpp

# detectfn = "HHR", movement model = "BVT",
list(lambda0 ~ 1, sigma ~ 1, z ~ 1, phi ~ 1)                                                             # --> OK
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ forest)                              # --> OK
list(lambda0 = ~ 1, sigma = ~ Session, z = ~ 1,  phi = ~ Session)                       # --> OK
list(lambda0 = ~ 1, sigma = ~ forest, z = ~ 1,  phi = ~ Session)                           # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ Session)         # --> error in makegkParalleldcpp
list(lambda0 = ~ 1, sigma = ~ forest + Session, z = ~ 1,  phi = ~ 1)                     # --> error in makegkParalleldcpp



I initially went for the CJSsecr model rather than one of the JS*secr models, and considered forest and other covariates on phi because we are actually interested in spatial variation in survival ...  However, I obviously misunderstood something and did not realise that trap (ie "spatial") covariates could not be used / should not work with the survival parameter phi ...  This kind of defeats the purpose of this analysis... Given this, which approach / strategy would you recommend to model spatial variation in survival, then?

Still, I would expect that "more than one covariate" is possible... @murrrray, I send you the dataset and script in a private message.


Olivier

Murray Efford

unread,
Aug 30, 2023, 8:55:29 PM8/30/23
to secr
Oliver

Thanks for sending example data that allowed me to find the source of the 'negative length vector' error. Turns out that the product of the dimensions of an internal array exceeded the allowed size of integer in R. I have added an explicit error message, now on GitHub. In your case it reads
Error: integer overflow in open.secr.loglifkn: 4667 parameter levels is too many given the numbers of detectors (1811) and mask cells (1949)
The code prepares a large array with size equal to the product of these dimensions, in your case 16472825213 (the largest integer allowed is about 2*10^9 - see ?as.integer). So you would need to reduce the number of parameter combinations by a factor of about 10 for it to work. I have yet to look into how you might do this.

The 'number of parameter levels' is the number of unique combinations of the levels of the different predictor variables. (Perhaps 'parameter' is misused here - may change that). I imagine the problem is the escalating number of unique combinations when predictors are combined, rather than the number of predictor variables per se.

Thinking aloud: Modelling spatial variation in phi is a hard problem because, conceptually, you would need to integrate over the possible unknown locations of animals between capture times and after last capture. Perhaps go down the Nimble path? Or partition your data according to first-capture location (I've just done this in another study)? (Most of your animals are in one clump, so chasing spatial effects may be unsatisfying). The CJSsecr model is messy because despite the name it's not a direct analogue of non-spatial CJS. I see why you would want to use it, but I don't think we (I) understand its limits.

I suggest using 'HEX' instead of 'HHR'. Failure with halfnormal (HHN) probably reflects the extreme improbability of distant detections under that model; HEX allows occasional long movements without requiring an extra parameter (HHR also leads to buffer-dependence, but that is probably not an issue for you).

Murray
Reply all
Reply to author
Forward
0 new messages