Using the Negative Hypergeometric to model a discrete finite random variable

69 views
Skip to first unread message

David Christianson

unread,
Apr 3, 2023, 6:40:47 PM4/3/23
to nimble-users
Hello All, I have perused several of the conversations relating to custom distributions and found them very useful for demonstrating how to expand the flexibility of nimble models. However, I haven't found an example that mimics the problems I have trying to make distributions from the 'extraDistr' package available to my nimble model. In particular, I would like to include the negative hypergeometric distribution in my model (rnhyper,dnhyper,qnhyper,pnhyper).   Any advice on how to implement distributions from other packages would be very helpful.    Thank you. 

Chris Paciorek

unread,
Apr 3, 2023, 9:30:45 PM4/3/23
to David Christianson, nimble-users
hi David,

You could of course implement the distribution of interest by directly coding the mathematical calculations in user-defined nimbleFunctions, as you say.
If you want to be able to call out to code provided by `extraDistr`, you might want to look into using `nimbleRcall` or `nimbleExternalCall` to
be able to call out to R or C/C++ code from within a nimbleFunction and thereby avoiding having to write up the calculations yourself.
There is help information via R's help system: `help(nimbleRcall)` and `help(nimbleExternalCall)`.

Happy to discuss further.

-chris

On Mon, Apr 3, 2023 at 3:40 PM David Christianson <daveal...@gmail.com> wrote:
Hello All, I have perused several of the conversations relating to custom distributions and found them very useful for demonstrating how to expand the flexibility of nimble models. However, I haven't found an example that mimics the problems I have trying to make distributions from the 'extraDistr' package available to my nimble model. In particular, I would like to include the negative hypergeometric distribution in my model (rnhyper,dnhyper,qnhyper,pnhyper).   Any advice on how to implement distributions from other packages would be very helpful.    Thank you. 

--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/3aac93bb-26eb-40cb-9dbe-97e59d9992ebn%40googlegroups.com.
Message has been deleted
Message has been deleted

Chris Paciorek

unread,
Apr 5, 2023, 12:02:02 PM4/5/23
to David Christianson, nimble-users
Hi David,

As far as using nimbleRcall or nimbleExternalCall, the latter will generally be faster, as there is overhead in calling out to R. So if the density will be called many times (as it would be in many algorithms, including MCMC), it may be worthwhile to set up the C/C++ code for the distribution of interest. And in fact, since extraDistr::dnhyper (and the 'r','p','q' functions) calls directly out to C++, it should 'just' be a matter of using the underlying C++ code in the source code for the extraDistr package.

You definitely need the `x` argument. That doesn't appear directly in the density in a nimble model (it's actually the left-hand side of the declaration in the nimble model code), but behind the scenes the model code is transformed so that the first argument to the density is `x`.

A couple other comments about your nimbleRcall:
- in general we recommend just working with doubles rather than integers, as nimble model calculations are all done with doubles. There is no loss of accuracy. We mention this in the help info for `registerDistributions`, though we should probably also mention it in Section 12.2 of the User Manual.
- I presume your returnType should be of dimension 0 not dimension 1.

I'm not quite sure what you mean by "as the model develops".

-chris

On Tue, Apr 4, 2023 at 2:01 PM David Christianson <daveal...@gmail.com> wrote:
Thanks for the quick reply to this pretty basic question that I have not been able to work around.  To use nimbleRcall, would the code below be appropriate? Here dnhyper is a distribution in 'extraDistr' with x, n, m, r, k, and log arguments).  Would I include the 'x' argument here as it is not called by nimble when modelling a distribution?   I have wondered about the overlap between nimbleRcall and nimbleExternalCall.     It would seem that in many situations one could use either call. Functionally, will these operate the same way (in terms of speed or flexibility)? What about adaptability as a model develops?  Apologies, this is several questions that maybe don't all belong in one thread!
dnhyper2 <- nimbleRcall(function(x = integer(1), n = integer(0), m = integer(0), r = integer(0), log = integer(0) {}, Rfun = 'dnhyper', returnType = integer(1), envir = .GlobalEnv)

Chris Paciorek

unread,
Apr 5, 2023, 5:28:12 PM4/5/23
to David Christianson, nimble-users
Hi David,

I haven't looked closely but nimble expects the first argument of the 'r' function is named 'n' and gives the number of samples desired when generating from the distribution -- see Section 12.2 of the manual. So I think you'll need to rename the 'n' parameter of your user-defined distribution to be something different to avoid argument name conflicts.

-chris

On Wed, Apr 5, 2023 at 2:22 PM David Christianson <daveal...@gmail.com> wrote:
Thanks for the quick and helpful reply Chris.  I thought it might be something rather simple as that but I was unsure having not found an example where a 3rd party-package distribution gets called and registered as a nimble distriubtion.  I think I was on the right track but after trying a couple of different approraches. I keep getting a consistent error that suggest there are two "n" formal arguments. One is an argument in the dnhyper() distribution, but I can't discern where the other one could be.  
Here is some code that should demonstrate the issue. 


rm(list=ls())

library(nimble)
library(extraDistr)

nsites<-1000
testG<-rnhyper(nsites,80,80,5) # 'data'

data<-list(nsites=nsites,testG=testG)

dnhyper2<-nimbleRcall(function(x = double(0,default=1), n = integer(0, default = 1), m=integer(0, default = 1),r=integer(0, default = 1),log=integer(0)) {} ,Rfun='dnhyper',returnType=double(0))

nhyper_code <- nimbleCode({
 
  n~dunif(1,100)
  m~dunif(1,100)
  r~dunif(1,10)
   for (i in 1:nsites){
     testG[i]~dnhyper2(n,m,r)
   }
})

  # Inital values
  inits <- function() list(n=80, m=80,r=5)
  initsValues <- inits
 
  # Parameters to monitor
  params <- c("n", "m", "r")
 
  # MCMC
  ni <- 12000
  nb <- 2000
 
  ## run nimble
  nhype <- nimbleMCMC(
    code = nhyper_code,
    constants = data,
    inits = initsValues,
    monitors = params,
    nburnin = nb,
    niter = ni,
    samplesAsCodaMCMC = TRUE
  )
  



On Monday, April 3, 2023 at 7:30:45 PM UTC-6 Chris Paciorek wrote:

David Christianson

unread,
Apr 5, 2023, 9:33:01 PM4/5/23
to nimble-users
Great thanks, that makes sense.   I think I may have achieved that with the code below (using urn terminology for white balls and black balls) but it is producing some odd results. There is no variance in the posterior distribution for the neghyp parameters but the random sample generating seems to be working well.

rm(list=ls())

library(nimble)
library(extraDistr)


dnhype2<-nimbleRcall(function(x= double(0,default=1), n = double(0, default = 1), m = double(0, default = 1), r = double(0, default = 1), log=integer(0,default=0)){}, Rfun="dnhyper",returnType=double(0))

rnhype2<-nimbleRcall(function(nn= double(0,default=1), n = double(0, default = 1), m = double(0, default = 1), r = double(0, default = 1)){}, Rfun="rnhyper",returnType=double(0))

dnhype <- nimbleFunction(
  run = function(x= double(0), white = double(0, default = 1), black = double(0, default = 1), failures = double(0, default = 1),log = integer(0, default = 0)) {
    returnType(double(0))
    Prob<-dnhype2(x,white,black,failures)
  return(Prob)
  })

rnhype <- nimbleFunction(
  run = function(n= double(0,default=1), white = double(0, default = 1), black = double(0, default = 1), failures = double(0, default = 1)) {
    returnType(double(0))
    return(rnhype2(1,white,black,failures))
  })


nsites<-1000
testG<-rnhyper(nsites,50,50,3) # 'data'
testG[seq(2,nsites,by=2)]<-NA


data<-list(testG=testG)
constants<-list(nsites=nsites)
nhyper_code <- nimbleCode({
 
  # failurs cannot be smaller than white
  black~dunif(11,100)
  white~dunif(11,100)
  failures~dunif(1,10)

   for (i in 1:nsites){
     testG[i]~dnhype(black,white,failures)
   }
})

  # Inital values
  testGinit<-rep(NA,nsites)
  testGinit[is.na(testG)]<-11
 
  # best to initialize failures at 1, so that no inconsistencies arise with initial values
  inits <- function() list(black=50, white=50,failures=1,testG=testGinit)

  initsValues <- inits
 
  # Parameters to monitor
  params <- c("black","white","failures","testG")
 
  # MCMC
  ni <- 25000
  nb <- 5000

 
  ## run nimble
  nhype <- nimbleMCMC(
    code = nhyper_code,
    data=data,
    constants=constants,

    inits = initsValues,
    monitors = params,
    nburnin = nb,
    niter = ni,
    thin=100,
    samplesAsCodaMCMC = TRUE
  )
 
  library(coda)
  library(mcmcplots)
 
  mcmcplot(nhype)
 
  hist(nhype[,seq(2,nsites,by=2)])
  hist(testG)
  hist(nhype[,3])
  

Reply all
Reply to author
Forward
0 new messages