Inquiry Regarding Model Errors in NIMBLE

35 views
Skip to first unread message

B Motwaly

unread,
Oct 9, 2024, 9:10:06 PM10/9/24
to nimble-users

Dear NIMBLE Group,

I hope this message finds you well. I am currently working on a population dynamics model using NIMBLE and have encountered several warnings that I would like your guidance on:

  1. Backwards Indexing Warnings:

    • [Warning] Detected backwards indexing in '10:9'. This is likely unintended and will likely not produce valid model code.
    • [Warning] Detected backwards indexing in '12:11'. This is likely unintended and will likely not produce valid model code.
    • [Warning] Detected backwards indexing in '14:13'. This is likely unintended and will likely not produce valid model code.
  2. Non-Constant Indexes Note:

    • [Note] Detected use of non-constant indexes: z[1, 1, 1], z[1, 2, 1], z[1, 3, 1], z[1, 4, 1], z[1, 5, 1], ... For computational efficiency, we recommend specifying these in 'constants'.

I would appreciate any insights into why these errors may be occurring and how I can resolve them. I have attached the relevant model code for your reference.

# write nimble model code
code <- nimbleCode({
    
  # intercepts
  for (s in 1:3) {
    phi_alpha[s] ~ dbeta(1, 1)
    p_alpha[s] ~ dbeta(1, 1)
  }
 
  # sex effects
  phi_beta ~ dnorm(0, 1)
  p_beta ~ dnorm(0, 1)
 
  # random effect SDs
  for (k in 1:2) {
    phi_sigma[k] ~ dexp(1)
    p_sigma[k] ~ dexp(1)
  }
 
  # random year effects
  for (t in 1:max(n_year)) {
    phi_year_z[t] ~ dnorm(0, 1)
    phi_year[t] <- phi_year_z[t] * phi_sigma[2]
    p_year_z[t] ~ dnorm(0, 1)
    p_year[t] <- p_year_z[t] * p_sigma[2]
  }
 
  # probability of transitioning to adulthood
  tau_alpha ~ dbeta(1, 1)
 
  # probability of being female for unobserved individuals
  pr_female ~ dbeta(5, 5)  
 
  # open site loop
  for (m in 1:n_site) {
   
    # random site effects
    phi_site_z[m] ~ dnorm(0, 1)
    phi_site[m] <- phi_site_z[m] * phi_sigma[1]
    p_site_z[m] ~ dnorm(0, 1)
    p_site[m] <- p_site_z[m] * p_sigma[1]
   
    # open individual loop
    for (i in 1:n_ind[m]) {
     
      # sex (being female) imputation
      female[m, i] ~ dbern(pr_female)
     
      # open years loop
      for (t in first[m, i]:(n_year[m] - 1)) {
       
        # survival probabilities
        for (s in 1:3) {
          phi[s, m, i, t] <- expit(phi_alpha[s]                       # age-specific intercept
                                   + phi_beta * (female[m, i] - 0.5)  # sex effect (specified at -0.5, 0.5 for male/female)
                                   + phi_site[m]                      # random site effects
                                   + phi_year[t]                      # random year effects
          )
        } # s
       
        # transition probability
        # note: this is probabably a "nuisance parameter"; it doesn't tell you much, because a tortoise MUST transition to adulthood at some point which is more a function of its survival
        tau[m, i, t] <- tau_alpha  # intercept only
       
        # ecological process
       
        # departing state: <1 year
        ps[1, m, i, t, 1] <- 0                    # survives as <1, doesn't transition to juvenile (latter impossible)
        ps[1, m, i, t, 2] <- phi[1, m, i,t]       # survives as <1, MUST transition to juvenile!
        ps[1, m, i, t, 3] <- 0                    # survives as <1, transitions to adult (impossible)
        ps[1, m, i, t, 4] <- 1 - phi[1, m, i, t]  # doesn't survive as <1 (and thus didn't transition to juvenile)
       
        # departing state: juvenile
        ps[2, m, i, t, 1] <- 0                                     # transitions to <1 (impossible)
        ps[2, m, i, t, 2] <- phi[2, m, i, t] * (1 - tau[m, i, t])  # survives as juvenile, doesn't transition to adult
        ps[2, m, i, t, 3] <- phi[2, m, i, t] * tau[m, i, t]        # survives as juvenile, transitions to adult
        ps[2, m, i, t, 4] <- 1 - phi[2, m, i, t]                   # doesn't survive as juvenile
       
        # departing state: adult
        ps[3, m, i, t, 1] <- 0                    # transitions to <1 (impossible)
        ps[3, m, i, t, 2] <- 0                    # transitions to juvenile (impossible)
        ps[3, m, i, t, 3] <- phi[3, m, i, t]      # survives as adult
        ps[3, m, i, t, 4] <- 1 - phi[3, m, i, t]  # doesn't survive as adult
       
        # departing state: dead
        ps[4, m, i, t, 1] <- 0
        ps[4, m, i, t, 2] <- 0
        ps[4, m, i, t, 3] <- 0
        ps[4, m, i, t, 4] <- 1
       
        # detection probabilities
        for (s in 1:3) {
          p[s, m, i, t] <- expit(p_alpha[s]
                                 + p_beta * (female[m, i] - 0.5)  # sex effect (specified at -0.5, 0.5 for male/female)
                                 + p_site[m]                      # random site effects
                                 + p_year[t]                      # random year effects
          )
        } # s
       
        # observation process
        # note: this can be inside of the same time loop
       
        # departing state: <1 year
        po[1, m, i, t, 1] <- p[1, m, i, t]
        po[1, m, i, t, 2] <- 0
        po[1, m, i, t, 3] <- 0
        po[1, m, i, t, 4] <- 1 - p[1, m, i, t]
       
        # departing state: juvenile
        po[2, m, i, t, 1] <- 0
        po[2, m, i, t, 2] <- p[2, m, i, t]
        po[2, m, i, t, 3] <- 0
        po[2, m, i, t, 4] <- 1 - p[2, m, i, t]
       
        # departing state: adult
        po[3, m, i, t, 1] <- 0
        po[3, m, i, t, 2] <- 0
        po[3, m, i, t, 3] <- p[3, m, i, t]
        po[3, m, i, t, 4] <- 1 - p[3, m, i, t]
       
        # departing state: dead
        po[4, m, i, t, 1] <- 0
        po[4, m, i, t, 2] <- 0
        po[4, m, i, t, 3] <- 0
        po[4, m, i, t, 4] <- 1
       
      } # t
     
      # likelihood
     
      for (t in (first[m, i] + 1):n_year[m]) {
       
        # ecological process
        z[m, i, t] ~ dcat(ps[z[m, i, t - 1], m, i, t - 1, 1:4])
       
        # observation process
        y[m, i, t] ~ dcat(po[z[m, i, t], m, i, t - 1, 1:4])
       
      } # t
    } # i
   
    # derived quantities
   
    # site- and age-specific survival and recapture
    phi_m[m, 1:3] <- expit(phi_alpha[1:3] + phi_site[m])
    p_m[m, 1:3] <- expit(p_alpha[1:3] + p_site[m])
   
    # site- and sex-specific survival and recapture
    phi_female[m] <- expit(phi_alpha[3] + phi_site[m] + phi_beta * 0.5)
    phi_male[m] <- expit(phi_alpha[3] + phi_site[m] + phi_beta * -0.5)
    p_female[m] <- expit(p_alpha[3] + p_site[m] + p_beta * 0.5)
    p_male[m] <- expit(p_alpha[3] + p_site[m] + p_beta * -0.5)
   
  } # m
 
  # year-specific survival and recapture for adults
  for (t in 1:(max(n_year) - 1)) {
    phi_t[t] <- expit(phi_alpha[3] + mean(phi_site[1:n_site]) + phi_year[t])
    p_t[t] <- expit(p_alpha[3] + mean(p_site[1:n_site]) + p_year[t])
  }
 
})

# constants, data, initial values, and parameters to monitor
consts <- list(n_site = n_site,
               n_ind = n_ind,
               n_year = years$n_year,
               first = first) |>
  glimpse()
data <- list(y = ch,
             z = z_known,
             female = female) |>
  glimpse()
inits <- list(z = z_init) |>
  glimpse()

params <- c("alpha", "beta", "sigma", "site", "year", "m", "t", "female", "male")
mons <- c(str_c("phi_", params),
          str_c("p_", params),
          "tau_alpha",
          "pr_female")  # updated parameter name

# model
Rmodel <- nimbleModel(code, consts, data, inits, check = F, calculate = F)


Thank you for your assistance!

Best regards,


Motwaly



Chris Paciorek

unread,
Oct 11, 2024, 8:25:23 PM10/11/24
to B Motwaly, nimble-users
Hi Motwaly,

The warning is probably coming from this line in your model code:

```
for (t in first[m, i]:(n_year[m] - 1)) {
```

You'll want to check that  for all the {m,i} pairs that that indexing produces a valid, non-empty, increasing sequence given your values of `first` and `n_year`. As the warning suggests, I wouldn't proceed without tracking down what is going on.

As far as the note about `z` that is just there for the case the `z` is fixed and a data node or unknown parameter node, in which case it is best to provide it in constants to avoid nimble doing unnecessary calculations behind the scenes that result from nimble not knowing that `z` will never change. Usually an index would either be fixed in advance or an unknown parameter in the model. In your case, it's complicated because it appears you are specifying the `z`s as data, but then there are a few of them that don't appear on the left-hand side of a declaration (the first cases that are observed). Those are the ones that Nimble's checking is flagging because it thinks they could be specified in constants. It should be fine to proceed despite that note. 

I haven't looked closely enough, but it might be possible to rewrite the model a bit so that the `z`'s that do not appear on the left-hand side of a declaration are declared in constants, but you'd need to separate them out as a separate variable that is not part of the `z` that are actually observations with a distribution specified for them. That would silence the note. But I don't think it will make much difference in this case in terms of computational efficiency. 

-Chris 

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/5f6c0946-da22-4ceb-811f-23badbe32438n%40googlegroups.com.

B Motwaly

unread,
Oct 14, 2024, 7:06:26 PM10/14/24
to nimble-users, paci...@stat.berkeley.edu
Dear NIMBL's users, 

I’m a bit puzzled as this model has been running smoothly for the past six months, so I’m unsure why I'm encountering this warning now. Therefore, I uninstalled NIMBLE version 1.2.0 and reverted to version 1.1.0. Interestingly, no warnings appeared when running the model in this older version. What could this indicate? I expected the newer update to enhance NIMBLE's accuracy.

Best,

Motwaly

Chris Paciorek

unread,
Oct 14, 2024, 9:06:37 PM10/14/24
to B Motwaly, nimble-users
Hi Motwaly,

Yes, in version 1.2.0 we started warning about backward indexing. Nimble will generally not properly handle cases where the for loop has indexes such as `10:9`.

Now if everything was working before, this is just a warning, and you could ignore the warning.

However, I would be cautious. I suggest that you check the values of 

```
for (t in first[m, i]:(n_year[m] - 1)) {
```

You'll want to check that  for all the {m,i} pairs that that indexing produces a valid, non-empty, increasing sequence given your values of `first` and `n_year`.
For example, I suspect there is a pair of {m,i} values for which `first[m, i]` equals 10 and `(n_year[m] - 1)` equals 9. (I.e., that the first time for the ith individual is too large relative to the number of possible times for that year.)

More generally, you can run `Rmodel$getNodeNames()` and check that the `z[m, i, t]` and `y[m, i, t]` nodes that you expect are all in the model.

-chris


Reply all
Reply to author
Forward
0 new messages