nesting nimble lists

[David Pleydell]

Dear Nimble-Users

I'm sure this is simple, but I'm struggling. 

I want to make a nimbleList where each element is also a nimbleList containing one integer scalar and a character vector of variable lengths.  For example, in R I can do this... 

myList <- list(node1=list(n=3, someText=c("A", "B", "C")), node2=list(n=2, someText=c("D", "A")), node3=list(n=1, someText=c("B")))

But I want to be able to do this inside some setup code where the character vectors are obtained via iterative calls to  model$getDependencies(someNodes[iterationNumber]). Here n is the length of the character vector, which is defined by the DAG generated by a nimble model. I don't see how I can handle the different and unknown lengths returned by each call to getDependencies.  

I could alternatively use modelValues instead, but I still don't see how to handle the mix of dimensions (character vector lengths). 

Any thoughts?  

David

[David Pleydell]

Cracked it! This does what I need.

#################

## Toy Example ##

#################

rm(list=ls())

library(nimble)

toyCode <- nimbleCode({

    rate[1,1] <- state[1,1] 

    rate[2,1] <- state[1,1] + state[2,1] + state[3,1] 

    rate[3,1] <- state[1,1] + state[2,1] 

    rate[1,2] <- state[1,1] + state[2,2] + state[2,3] 

    rate[2,2] <- state[1,1] + state[3,3] 

    rate[3,2] <- state[1,1] + sum(state[3,])

})

inits    <- list(state = matrix(1:9, nrow=3))

toyModel <- nimbleModel(toyCode, inits=inits)

toyModel[["state"]]

toyModel[["rate"]]

(stateIndices <- as.vector(outer(1:dim(toyModel[["state"]])[1], 1:dim(toyModel[["state"]])[2], paste, sep=",")))

(stateNodes   <- paste0("state[",stateIndices,"]"))

(nStateNodes  <- length(stateNodes))

(stateIndices <- paste0("row",sub(",","col",stateIndices)))

(rateIndices <- as.vector(outer(1:dim(toyModel[["rate"]])[1], 1:dim(toyModel[["rate"]])[2], paste, sep=",")))

(rateNodes   <- paste0("rate[",rateIndices,"]"))

(nRateNodes  <- length(rateNodes))

## Define list structures

innerList <- nimbleList(n = integer(0), txt = character(1))

(outerDef  <- paste0("nimbleList(",paste0(stateIndices, "=innerList()", collapse=","),")"))

outerList <- eval(parse(text=outerDef))

## Create outer list

dependentRatesList <- outerList$new()

## Create inner list & fill outer list

for (iSN in 1:nStateNodes) { ## iSN=2

    (deps <- toyModel$getDependencies(stateNodes[iSN]))

    (deps <- deps[grep("rate", deps)])

    eval(parse(text=paste0("dependentRatesList$",stateIndices[iSN], " <- innerList$new(n=", length(deps),", txt=deps)"))) 

}

print(dependentRatesList)

[David Pleydell]

Following up with a feature request. It would be great if numeric indexing of nimbleLists was possible. Currently only the first two of the following potential indexing options "work". 

dependentRatesList[["row3col3"]]        ## works

dependentRatesList[[stateIndices[9]]]  ## works 

dependentRatesList[[9]]                       ## fails

dependentRatesList[[3,3]]                    ## fails

dependentRatesList[3,3]                      ## fails

dependentRatesList[9]                         ## fails

 

[Perry de Valpine]

Hi David,

Thanks for this.

Right, one nimbleList can contain elements whose type is a different nimbleList.

I understand the need for numeric indexing, and more flexibly data structures in general.  Here is another option.  You can treat a nimbleFunction with setup code as simply a data structure, since it yields a class definition.  In the setup code, "setupOutputs(my_var, my_other_var)" will ensure that "my_var" and "my_other_var" are established as member data of the resulting class definition.  You can then make a nimbleFunctionList of this nimbleFunction type, giving you a numerically-indexed list of objects with whatever types you created in setup code.  The easiest way to get the types you want is to create my_var and my_other_var in setup code as objects of the types you want.

This is a quick reply but we can dive deeper if it would help.

Best,

Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/a16ef844-49b1-4b38-a00b-df16dba40ae0%40googlegroups.com.

[David Pleydell]

Here is another option.  You can treat a nimbleFunction with setup code as simply a data structure, since it yields a class definition.  In the setup code, "setupOutputs(my_var, my_other_var)" will ensure that "my_var" and "my_other_var" are established as member data of the resulting class definition.  You can then make a nimbleFunctionList of this nimbleFunction type, giving you a numerically-indexed list of objects with whatever types you created in setup code.  The easiest way to get the types you want is to create my_var and my_other_var in setup code as objects of the types you want.

This is a quick reply but we can dive deeper if it would help.

 

Many thanks Perry, 

That certainly sounds like an interesting option - it would be great to see an example of that make either the manual or the website in the future. For now I can make do with the solution I found, so I'll stick with that.

One thing that remains confusing to me when reading the manual on this stuff is whether or not in (compiled) run code I can replace a vector (or matrix or array) stored in a nimbleList with another one of different dimension.  In C this would require re-allocating memory, so would nimble handle that automatically, or would the user need to build a memory re-allocation step into their code ?

All the best

David

[Perry de Valpine]

Hi David,

We know the User Manual is so long that it can sometimes be hard to find what you need. (That's one reason we made more examples for the web site recently, and of course we wish had time to do more, and we do welcome suggestions (or contributions) even if we don't always have time to take them up quickly.)  However, it is in there!  It's in section 15.4.10, "User-defined data structures".

Regarding assignment to non-scalar objects: You can change the sizes, but not the number of indices.  (Unfortunately the word "dimension" gets used in a couple of different ways in this context.)  For example, if X was already created as a matrix and A and B are matrices and C is a 3D array (i.e. an object with 3 indices), then:

X <- A %*% B ## OK: complete replacement of X; nimble compiler will manage the memory for you.

X <- C ## Error: can't assign a 3D object (having 3 indices) two a 2D (matrix) object.

X <- C[1:5, , 3] ## OK: complete replacement of X from a 2D object created by indexing a 3D object; nimble compiler will manage the memory for you.

HTH

-Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/92e0f435-3ba9-484d-8235-f536594ffd41%40googlegroups.com.

[David Pleydell]

Many thanks Perry for this feedback  

A further useful feature would be the ability to obtain a matrix element with a single index. e.g. in R...

> (M <- diag(1:3))

     [,1] [,2] [,3]

[1,]    1    0    0

[2,]    0    2    0

[3,]    0    0    3

> M[9]

[1] 3

right now the nimble compiler rejects such indexing. 

Best regards

David

[David Pleydell]

I am unable to compile a nimble function that uses the toy example code I posted on 15 January. 

The following code gives "Error in thisKeywordInfo$processor(code, nfProc) :   object 'callerCode' not found" at the compilation step 

#################

## Toy Example ##

#################

rm(list=ls())

library(nimble)

toyCode <- nimbleCode({

    rate[1,1] <- state[1,1] 

    rate[2,1] <- state[1,1] + state[2,1] + state[3,1] 

    rate[1,2] <- state[1,1] + state[1,2] + state[2,2] + state[3,2] 

    rate[2,2] <- sum(state[,])

})

inits    <- list(state = matrix(1:6, nrow=3))

toyModel <- nimbleModel(toyCode, inits=inits)

toyModel[["state"]]

toyModel[["rate"]]

##########################################################

setupMyFunc <- nimbleFunction( 

    setup = function(model, rateNode="rate", stateNode="state") {

        ## Obtain stateNodes, nStateNodes and stateIndices

        (stateIndices <- as.vector(outer(1:dim(model[[stateNode]])[1], 1:dim(model[[stateNode]])[2], paste, sep=",")))

        (stateNodes   <- paste0(stateNode,"[",stateIndices,"]"))

        (nStateNodes  <- length(stateNodes))

        (stateIndices <- paste0("row",sub(",","col",stateIndices)))

        ## Obtain rateIndices, rateNodes and nRateNodes

        (rateIndices  <- as.vector(outer(1:dim(model[[rateNode]])[1], 1:dim(model[[rateNode]])[2], paste, sep=",")))

        (rateNodes    <- paste0(rateNode,"[",rateIndices,"]"))

        (nRateNodes   <- length(rateNodes))

        ## Define list structures

        (innerList <- nimbleList(n = integer(0), txt = character(1)))

        (outerDef  <- paste0("nimbleList(",paste0(stateIndices, "=innerList()", collapse=","),")"))

        (outerList <- eval(parse(text=outerDef)))

        ## Create outer list

        dependentRatesList <- outerList$new()

        ## Create inner list & fill outer list

        for (iSN in 1:nStateNodes) { ## iSN=2

            (deps <- model$getDependencies(stateNodes[iSN]))

            (deps <- deps[grep(rateNode, deps)])

            eval(parse(text=paste0("dependentRatesList$",stateIndices[iSN], " <- innerList$new(n=", length(deps),", txt=deps)"))) 

        }

        ## Make objects available in run 

        setupOutputs(nStateNodes, stateIndices, dependentRatesList)

    },

    run = function(){

        for (iSN in 1:nStateNodes) {

            n   <- dependentRatesList[[stateIndices[iSN]]]$n

        }

    }

)

myFunc <- setupMyFunc(model=toyModel)

myFunc$run()

cModel <- compileNimble(toyModel)

cFunc  <- compileNimble(myFunc, showCompilerOutput = TRUE)

[David Pleydell]

If I change the indexing in the run code from 

n  <- dependentRatesList[[stateIndices[iSN]]]$n

to 

 n   <- dependentRatesList[[stateIndices[iSN]]][["n"]]

the error code changes to  "Error in code$args[[2]] : subscript out of bounds"

Any thoughts?

[Perry de Valpine]

Hi David,

You're running into limitations of the nimbleList system.  It's not very general.

Typically, there is little to no dynamic run-time resolution of name-based indexing in compiled code.  Name-based indexing (e.g. model[[target]], where target is character) mostly works only if target is fixed in setup code and then baked in to the compiled code.  There are exceptions with vectors of node names, such as model$calculate(nodes[i]).  But your stateIndices is a character vector.  The reason something like " n   <- dependentRatesList[[stateIndices[iSN]]]$n" is not amenable to the compilation system is that the type of the result wouldn't be able to be determined at compile time, and there is no dynamic typing in compiled code.

Can you convey a more general picture of what you want to accomplish?  Do you want the "txt" entries to be used in model calculations and such, or for some other purpose?  My guess is that a nimbleFunctionList might work, or that a re-arrangement in what is managed in R vs. compiled nimbleFunctions could make sense, or possibly that what you want to do is beyond nimble's compilation system.

-Perry

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/546f6fb8-9413-4d61-9586-f6c10dd3895f%40googlegroups.com.

[David Pleydell]

Many thanks Perry for this valuable information. 

I am attempting to write a nimbleFunction that performs the Gillespie direct method algorithm for a dynamic model represented in BUGS code. So the list is to identify which rates require recalculation once one or two states have been changed (as opposed to redundantly recalculating all rates). 

You guessed right - I had some success yesterday with the nimbleFunctionsList approach that I hope to integrate into my code today. 

Best regards

David