dimension mismatch in variable

[Glenn Stauffer]

I am trying to run a population growth model, and when I try to define my model, I get the following message:

defining model...

Error in genVarInfo3() : 

  Dimension of r is 1, which does not match its usage in 'mu[j, t] <- N[j, t - 1] * exp(r[hunt[t]] * (1 - log(N[j, t - 1] + 1)/log(K + 1))) + iota'.

It is true that r has dimension 1, the prior is defined as:

for(i in 1:2){

r[i] ~ dnorm(0,1)

}

But I can't see how the usage in the offending line of code indicated is mismatched. The variable "hunt" is a data vector of ones and twos used to index r. It seems to me it is consistent with r having 1 dimension. 

What am I missing that generates a dimension mismatch in the LHS and RHS usage of r? 

Thanks

[Chris Paciorek]

Hi Glenn,

As of version 0.6-5 of NIMBLE we don't yet handle having indices be
unknown/dynamic. So hunt[j] would need to be specified in the model as
a part of 'constants'. If it's not, that will trigger an error (though
it should be triggering a different error than you seem to be having,
which has me puzzled).

So, is hunt[t] a constant? If it's not, that's the issue. We're
working right now on allowing dynamic indexes and hope to have that
available as of version 0.6-6 in a few weeks.

If hunt[t] is a constant, please send the BUGS code for the full model
so I can trap the specific problem here.

-Chris

> --
> You received this message because you are subscribed to the Google Groups
> "nimble-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nimble-users...@googlegroups.com.
> To post to this group, send email to nimble...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/nimble-users/a5993200-46eb-4c76-9b72-d9bd5bb0a7a3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Glenn Stauffer]

Ah, thanks, I should have thought of that! It does seem to work as expected now when I include hunt in “constants” rather than “data”. And that's good news about soon allowing dynamic indexing.

Glenn

[Chris Paciorek]

In your defense, that error message is not particularly helpful. I'll
make a note for us to trap that case better.

[Glenn Stauffer]

Does nimble support multiple levels in nested indices? The problem cropped again for, this time when I am trying to sum matrix columns, where some cells are NA and my nested indexing identifies the not-NA values.

My code looks like this:
for(t in 1:nYear){
Ntot[t] <- sum(N[vind[ind[t,1]:ind[t,2]],t])
}

Where vind is a vector of row indices giving the rows in each column of N that are not NA, and ind is a 2-column matrix with nYear rows where the 2 columns give the index values for vind that apply to each column in N. N is a matrix with nYear columns and some rows having a variable number of leading NA cells up until there were data to estimate values (effort began in different years at different sites).
Both ind and vind are provided as constants.

The error produced is the same as before:

defining model...
Error in genVarInfo3() :

Dimension of N is 2, which does not match its usage in 'Ntot[t] <- sum(N[vind[ind[t, 1]:ind[t, 2]], t])'.

Maybe the doubly nested indexing is not allowed? As an alternative, does nimble support providing the index values as list elements (where each list element could be a different length)? Or, the model does run if I set those NAs to zero and then sum over all rows....

[Perry de Valpine]

Glenn,

What you are trying to do should be supported, but you've found a bug mostly about error-trapping cases where dimensions are used incorrectly.  Your code should be valid but is incorrectly trapped as invalid.  I'll file an issue and fix it in a future release, but for now here is a workaround.  Since I don't have your full model I set it up in a toy model.

sumRange <- nimbleFunction(

    run = function(Ncol = double(1), inds = double(1)) { ## all model variables are doubles, even if they should be integers

        return(sum(Ncol[inds]))

        returnType(double())

    })

m <- nimbleModel(

    nimbleCode({

        for(t in 1:4)

            for(j in 1:3)

                N[j, t] ~ dbinom(prob = 0.5, size = 3)

        for(t in 1:4)

            Ntot[t] <- sumRange(N[1:3, t], vind[ind[t,1]:ind[t, 2]]) ## sum(N[ vind[ind[t,1]:ind[t, 2]], t])

    }),

    constants = list(

        ind = t(matrix(c(1, 3, 4, 6, 7, 9, 10, 12), nrow = 2)),

        vind = rep(1:3, 4)

)

)

The idea is to push summing over the row index vector into a separate function.  The model code can now correctly identify the graph structure (what depends on what) by seeing that Ntot[t] depends on N[1:3, t].  And the calculation you want will be done in sumRange.

By the way, a similar workaround can be used to achieve the effect of stochastic indices as a workaround until we support them in a future release soon.

Perry

>> send an email to nimble-users+unsubscribe@googlegroups.com.

>> To post to this group, send email to nimble...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/nimble-users/a5993200-46eb-4c76-9b72-d9bd5bb0a7a3%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users+unsubscribe@googlegroups.com.

To post to this group, send email to nimble...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/017901d2f691%24bc462490%2434d26db0%24%40gmail.com.

[Glenn Stauffer]

Thanks Perry, it looks like that works.

And now that you mention it, I do remember a previous post where you suggested the exact same solution for stochastic indexing. If I had remembered sooner, I might have tried it here!

 

I will note that when I fiddled with your example so that some of the cells of N were left NA (see code below), it still worked, but when I ran the model a warning was generated:

 

warning: value of right hand side only node not initialized

 

I suppose this warning results from those NA values in N, and then using N in the custom nimbleFunction. Nonetheless, the summation does appear to be correct (e.g., N[1,1]==Ntot[1]; sum(N[c(1,3),2)==Ntot[2], etc).

 

 

mm <- nimbleModel(

    nimbleCode({

        for(j in 1:3){

                for(t in f[j]:4){

                   N[j, t] ~ dbinom(prob = 0.5, size = 3)

                }

       }

       for(t in 1:4){

            Ntot[t] <- sumRange(N[1:3, t], vind[ind[t,1]:ind[t, 2]]) ## sum(N[ vind[ind[t,1]:ind[t, 2]], t])

                                }

    }),

    constants = list(

       ind = t(matrix(c(1, 1, 2, 3, 4, 6, 7, 9), nrow = 2)),

        vind = c(1,1,3,1:3,1:3),

        f = c(1,3,2)

    )

)

 

 

From: Perry de Valpine [mailto:pdeva...@berkeley.edu]
Sent: Friday, July 7, 2017 6:14 AM
To: Glenn Stauffer <gesta...@gmail.com>
Cc: Chris Paciorek <christophe...@gmail.com>; nimble-users <nimble...@googlegroups.com>
Subject: Re: dimension mismatch in variable

 

Glenn,

>> send an email to nimble-users...@googlegroups.com.

>> To post to this group, send email to nimble...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/nimble-users/a5993200-46eb-4c76-9b72-d9bd5bb0a7a3%40googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.

To post to this group, send email to nimble...@googlegroups.com.

[Chris Elrod]

This function calculates the product A' * A, where A is the product of the symmetric block diagonal matrix "diag" and the kronecker product of lower triangular "L" and an identity matrix.

kron_diag_quad_form <- nimbleFunction(
  run = function(L = double(2), diag = double(3)){
    returnType(double(2))
    diag_dim <- dim(diag)
    T <- diag_dim[2]
    K <- diag_dim[3]
    KT <- K*T
    kdqf <- matrix(double(0, KT^2, init=FALSE), ncol=T)
   
    for (kr in 1:K){
      kr_step <- (kr-1)*T
      for (kc in 1:(kr-1)){
        kc_step <- (kc-1)*T
        kdqf_sub <- L[kr,kr] * L[kr,kc] * diag[1:T,1:T,kr]
        for (kj in (kr+1):K){
          kdqf_sub <- kdqf_sub + L[kj,kr] * L[kj,kc] * diag[1:T,1:T,kj]
        }
        kdqf[(kr_step+1):(kr_step+T),(kc_step+1):(kc_step+T)] <- kdqf_sub
        kdqf[(kc_step+1):(kc_step+T),(kr_step+1):(kr_step+T)] <- kdqf_sub
      }
      kdqf_sub <- L[kr,kr]^2 * diag[kr]
      for (kj in (kr+1):K){
        kdqf_sub <- kdqf_sub + L[kj,kr]^2 * diag[1:T,1:T,kj]
      }
      kdqf[(kr_step+1):(kr_step+T),(kr_step+1):(kr_step+T)] <- kdqf_sub
    }
    return(kdqf)
  }
)

Trying to compile the model gives me the following error:

defining model...
Error in genVarInfo3() :

  Dimension of Li_K is 2, which does not match its usage in 'fullPrecision <- kron_diag_quad_form(Li_K, ARarray)'.

I tried a couple iterations of moving pieces within the function off into other functions, but it did not help.

For example, attempting to follow pdevalpine's approach:

calculate_off_sub_matrix <- nimbleFunction(
  run = function(v1 = double(1), v2 = double(1), diag = double(3)){
    returnType(double(2))
    T = dim(diag)[1]
    kdqf_sub <- (v1[1] * v2[1]) * diag[1:T,1:T,1];
    for (i in 2:length(v1)){
      kdqf_sub <- kdqf_sub + (v1[i] * v2[i]) * diag[1:T,1:T,i];
    }
    return(kdqf_sub)
  }
)
calculate_on_sub_matrix <- nimbleFunction(
  run = function(v = double(1), diag = double(3)){
    returnType(double(2))
    T = dim(diag)[1]
    kdqf_sub <- (v[1]^2) * diag[1:T,1:T,1];
    for (i in 2:length(v)){
      kdqf_sub <- kdqf_sub + (v[i]^2) * diag[1:T,1:T,i];
    }
    return(kdqf_sub)
  }
)

kron_diag_quad_form <- nimbleFunction(
  run = function(L = double(2), diag = double(3)){
    returnType(double(2))
    diag_dim <- dim(diag)
    T <- diag_dim[2]
    K <- diag_dim[3]
    KT <- K*T;
   
    for (kr in 1:K){
      kr_step <- (kr-1)*T;
      for (kc in 1:(kr-1)){
        kc_step <- (kc-1)*T;
        kdqf_sub <- calculate_off_sub_matrix(L[kr:K,kr], L[kr:K,kc], diag[1:T,1:T,kr:K])
        kdqf[(kr_step+1):(kr_step+T),(kc_step+1):(kc_step+T)] <- kdqf_sub;
        kdqf[(kc_step+1):(kc_step+T),(kr_step+1):(kr_step+T)] <- kdqf_sub;
      }
      kdqf_sub <- calculate_on_sub_matrix(L[kr:K,kr], diag[1:T,1:T,kr:K])
      kdqf[(kr_step+1):(kr_step+T),(kr_step+1):(kr_step+T)] <- kdqf_sub;
    }
    return(kdqf);
  }
)

but got the same error.

I find this error bizarre. If the function is somehow implying that `kron = double(2)` is anything other than the explicitly stated `double(2)`, I'd hope defining the function would immediately throw an error -- for it not to wait until compiling the model.


[Chris Paciorek]

Hi Chris, a couple things here:

1) You need to initialize 'kdqf' before you assign into a subset of
it, using numeric() (aka nimNumeric()).
Otherwise NIMBLE can't figure out its size.
Not sure if that's the only issue as I would need code for your model
to reproduce this. But please start there
and then respond if that doesn't solve the problem.

2) More generally, as you note, that error message seems misleading.
If you can provide a reproducible example (either your
full model, or ideally a simplified version that just focuses on the
error that is occurring), we can
take a look and see if we can figure out why the message occurs when
it does and if we can improve that.

chris

> --
> You received this message because you are subscribed to the Google Groups "nimble-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
> To post to this group, send email to nimble...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/deca9f3c-c96a-4406-b2ea-7b8f0c19b766%40googlegroups.com.

[Chris Elrod]

This is so simplified that I wouldn't bet it is the same problem anymore, but:

nimble_Wish_code <- nimbleCode({
  W[1:K,1:K] ~ dwish(R = I_K[1:K,1:K], df = K + 1)
  Li_K[1:K,1:K] <- chol(W)
})

NIMBLE_model <- nimbleModel(
  nimble_Wish_code,
  constants = list( K = 15 )
)

defining model...
Error in genVarInfo3() :

  Dimension of I_K is 2, which does not match its usage in 'lifted_chol_oPI_K_oB1to15_comma_1to15_cB_cP[1:15, 1:15] <- chol(I_K[1:15, 1:15])'.

[Daniel Turek]

Two issues.  First, you need full indexing on the matrix W in the model, specifically in chol(W).

Second, to get the model to build without an error about chol() of a singular matrix, then provide initial values for W and I_K, through the "inits" list argument.  Easiest choice is identity matrix.

Both changes shown below, works fine:

library(nimble)

nimble_Wish_code <- nimbleCode({

    W[1:K,1:K] ~ dwish(R = I_K[1:K,1:K], df = K + 1)

    Li_K[1:K,1:K] <- chol(W[1:K,1:K])

})

NIMBLE_model <- nimbleModel(

    nimble_Wish_code,

    constants = list( K = 15 ),

    inits = list( I_K = diag(15), W = diag(15) )

)

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/19a5c781-3168-4452-b9d7-a4696cdf5a37%40googlegroups.com.

[Chris Elrod]

> library(nimble)
nimble version 0.6-13 is loaded.
For more information on NIMBLE and a User Manual,
please visit http://R-nimble.org.

Attaching package: ‘nimble’

The following object is masked from ‘package:stats’:

    simulate

>
> nimble_Wish_code <- nimbleCode({
+     W[1:K,1:K] ~ dwish(R = I_K[1:K,1:K], df = K + 1)
+     Li_K[1:K,1:K] <- chol(W[1:K,1:K])
+ })
>
> NIMBLE_model <- nimbleModel(
+     nimble_Wish_code,
+     constants = list( K = 15 ),
+     inits = list( I_K = diag(15), W = diag(15) )
+ )

defining model...
Error in genVarInfo3() :
  Dimension of I_K is 2, which does not match its usage in 'lifted_chol_oPI_K_oB1to15_comma_1to15_cB_cP[1:15, 1:15] <- chol(I_K[1:15, 1:15])'.

This does not work for me.

[Chris Elrod]

My install was broken. I reinstalled, and now your code runs.

I'll update when I have an appropriate minimal working example from my original model / if the problem is fixed via liberal sprinkling of [1:upper_bound,1:upper_bound].

[Daniel Turek]

Good, good, good luck and keep us posted.

Yes, NIMBLE model code always requires *full indexing* for all vectors & matrices, that is, having no missing indices such as vec[], or matrix[i,], those would rather have to be vec[1:N], and matrix[i, 1:M]

Unless, only exception is if you provide addition information about the dimension of your variable(s) via the "dimensions = list(...)" argument to nimbleModel().

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/1d5e0b48-75bf-4d9a-ac2f-a0569aebcec1%40googlegroups.com.