Negative binomial regression for truncated data

385 views
Skip to first unread message

Guido Biele

unread,
Dec 1, 2014, 5:23:50 PM12/1/14
to stan-...@googlegroups.com

Hello, I am trying to run a negative binomial regression and succeed until I try to implement truncation of the criterion variable.


I started with the following simple negative binomial regression:


data {

  int<lower=1> N;               // number of data points

  int<lower=0> y[N];            // observations

  int<lower=1> K;               // number parameters

  matrix[N,K] x;                // individual predictors

}


parameters {

  vector[K] beta;

  real<lower=0.01, upper=50> nb_beta; // beta (inv. scale) parameter of NB

}


model {

  for (k in 1:K){              

    beta[k] ~ normal(0,1);}    

  nb_beta ~ cauchy(0,1);


  for (n in 1:N) {

         y[n] ~ neg_binomial(exp(x[n] * beta)*nb_beta,nb_beta);

   }

}



This model works fine and converges in less than 10 seconds and within 250 samples for a data set with N = 1000 and K=15 predictors.


Next I tried to adapt the basic model for a truncated outcome variable, where all y >= 3, by changing only the sampling statement from

y[n] ~ neg_binomial(exp(x[n] * beta)*nb_beta,nb_beta); 

to

y[n] ~ neg_binomial(exp(x[n] * beta)*nb_beta,nb_beta) T[2,];



However, this led to a few of the following messages: 

Informational Message: The current Metropolis proposal is about to be rejected because of the following issue:

stan::prob::neg_binomial_log(N4stan5agrad3varE): Shape parameter is inf:0, but must be finite!

If this warning occurs sporadically, such as for highly constrained variable types like covariance matrices, then the sampler is fine,

but if this warning occurs often then your model may be either severely ill-conditioned or misspecified.


And stan does not complete 10% of the samples within 20+ minutes.

When I initialize the sampling with regression weights obtained from the full data set (i.e. outcomes y >= 0), I can fit the model with truncation.


My questions: 

- Is my general approach to modeling truncated outcome variable sound?

- Is it to be expected that truncation makes sampling so much harder, or could I improve my model to achieve convergence also without initialization?


Thanks in advance!

Guido

Bob Carpenter

unread,
Dec 3, 2014, 6:26:39 PM12/3/14
to stan-...@googlegroups.com
Sorry for the slow replies --- we've been hammered with visitors,
travel, and grant writing here at Stan HQ.

I'm not sure why adding the truncation would cause a problem
if all y[n] >= 3. It might just be slow because we have to
take derivatives of CDFs, which can be hairy.

A few warning messages of the kind you see are OK. Your negative
binomial shape parameter is wandering off to +infinity. One
way to fix that is to provide a thinner tailed prior than cauchy(0,1),
but I would've thought that would provide a fair bit of regularization
due to the concentration of mass around 0.

Where do the nb_beta parameters go to? Truncating those above at 50
is supsicious --- in the non-truncated fit, do the parameters brush
up against those boundaries? We generally prefer to use soft priors
than hard truncations to avoid issues like parameters running up against
boundaries.

- Bob
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Guido Biele

unread,
Dec 4, 2014, 5:07:02 PM12/4/14
to

Thanks for taking time to look at this Bob,

(and apologies for my longish reply)

The derivatives might indeed be the problem. When I run a model without truncation the Gradient evaluation takes 0.06 seconds, and when I run a model with truncation the time taken jumps to 0.45 seconds. These numbers are for a hierarchical regression model with N = 33 000, three groups, and K = 17 predictors. 
(with cmdstan--compiled with intel MKL, linux redhat,  2.6 GHz intel processors, 8 or 16GB RAM)

Given the large N and mostly only weakly correlated predictors, the model should not generally be hard to estimate, I assume.

The hard constraint on nb_beta (the beta parameter for the negative binomial) is likely not the problem, because the posterior mean is around 0.9.


I have following ideas to speed up the code, maybe you have an opinion about which is most likely to improve speed (the full model is below)?

(a)
Use more informative priors (i.e. normal(0,1) for betas and chauchy(0,1) for variances.
(I know that nb_beta and all all betas are below 1.) I hope this can address one factor Marc mentioned in his discussion of what influences speed, namely how fast the sampler towards the region of the parameter space with high posteriors.

(b)
Speed up calculations by vectorizing (1) (the second factor Marc mentioned). 
In particular, I think of splitting the input by group, i.e. I could submit all relevant variables split by group

Ngrp1, Ngrp2, Ngrp3, ygrp1, ygrp2, ygrp3, xgrp1, xgrp2, xgrp,

which would allow to calculate the means as e.g. 

mugrp1 <- beta[1] * xgrp1[1]

instead of looping through all N and do mu[n] <- beta[g[n]] * x[n]

(c)
Speed up by vectorizing (2): 
Inspired by Marco (Inacio’s) description of a zero-altered NB regression I thought of doing something along these line:
First increment_log_probs gobally as if there were no truncation:

y ~ neg_binomial(alpha, beta)

Then correct for truncation by incrementing again

increment_log_prob(sum(neg_binomial_cdf_log(SC-1, alpha,beta) * is_equalabove_SC))
increment_log_prob(sum(neg_binomial_ccdf_log(SC, alpha,beta) * is_below_SC))

whereby SC is the truncation point and is_equalabove_SC is a vector with zeros for criterion values below the cutoff SC and ones otherwise, so that the sum is calculated only over the relevant rows (the line below is then for below cutoff values). 

Somehow I fear that this is a naïve approach, but I think it should be possible to calculate the cdf and ccdf in vectorized form. But this requires that alpha and beta in neg_binomial_ccdf_log(SC, alpha,beta) can be vectors. I didn't try it, but given that the stan reference describes the function as "neg_binomial_cdf(ints n, reals alpha, reals beta)" I hope that the use of plural means that alpha and beta can be vectors.

Lastly, I found that the stan dev branch of includes functions for cdfs and ccdfs for neg_binomilal_2. I have now simply build stan 2.5 after replacing the original neg_binomial_2.hpp with the dev-version. Is this a safe thing to do?


What do you, or anybody else following this discussion, think?

Cheers – Guido


* without going into to many details: In each groups participants over and at/above the truncation/cutoff point are either over or under represented. Think of running a binomial regression to predict number of symptoms in a group that consists to 80% of cases that have more symptoms than the 90th percentile of the population, and 20% of controls that have fewer symptoms.


#################### Model ########################

data {

  int<lower=1> N;     // number of data points
  int<lower=0> y[N];  // observations

  int M;                      // number of groups
  int SC;                    // cut off value for cases (all cases >= cutoff)
  int g[N];                  // group for individual
  int<lower=1> K;     // number predictors (including intercept)
  int<lower=1> L;     // num group predictors

  matrix[N,K] x;        // individual predictors

  matrix[M,L] u;        // group predictors
}


parameters {
  matrix[K,M] z;
  cholesky_factor_corr[K] L_Omega;
  vector<lower=0>[K] tau;               // prior scale
  matrix[L,K] gamma;                      // group coeffs
  real<lower=0.01> nb_beta;          // beta (inv scale) parameter of negative binomial
}

transformed parameters {
  matrix[M,K] beta;
  beta <- u * gamma + (diag_pre_multiply(tau,L_Omega) * z)';
}


model {
  vector[N] mu;         // expected number of symptoms
  tau ~ cauchy(0,1); // orig scale: 2.5
  to_vector(z) ~ normal(0,1);
  L_Omega ~ lkj_corr_cholesky(2);
  to_vector(gamma) ~ normal(0,1); // orig scale: 5

  nb_beta ~ cauchy(0,1);

  for (n in 1:N) {

    mu[n] <- exp(dot_product(x[n],beta[g[n]]));


    if (y[n] < SC) 

        y[n] ~ neg_binomial(mu[n]*nb_beta,nb_beta) T[,SC];
    else
        y[n] ~ neg_binomial(mu[n]*nb_beta,nb_beta) T[SC-1,];
  }
}

Bob Carpenter

unread,
Dec 5, 2014, 1:45:42 PM12/5/14
to stan-...@googlegroups.com

> On Dec 4, 2014, at 5:07 PM, Guido Biele <guido...@gmail.com> wrote:
>
> Thanks for taking time to look at this Bob,
>
> (and apologies for my longish reply)
>
> The derivatives might indeed be the problem. When I run a model without truncation the Gradient evaluation takes 0.06 seconds, and when I run a model with truncation the time taken jumps to 0.45 seconds. These numbers are for a hierarchical regression model with N = 33 000, three groups, and K = 17 predictors.
> (with cmdstan--compiled with intel MKL, linux redhat, 2.6 GHz intel processors, 8 or 16GB RAM)
>
> Given the large N and mostly only weakly correlated predictors, the model should not generally be hard to estimate, I assume.
>
> The hard constraint on nb_beta (the beta parameter for the negative binomial) is likely not the problem, because the posterior mean is around 0.9.
>
>
>
> I have following ideas to speed up the code, maybe you have an opinion about which is most likely to improve speed (the full model is below)?
>
> (a)
> Use more informative priors (i.e. normal(0,1) for betas and chauchy(0,1) for variances.
> (I know that nb_beta and all all betas are below 1.) I hope this can address one factor Marc mentioned in his discussion of what influences speed, namely how fast the sampler towards the region of the parameter space with high posteriors

This can often help with the number of effective samples per iteration and
also improve the iteration times if you find the iterations in the tails
take longer (they can if you need more leapfrog steps to get a u-turn).

> (b)
> Speed up calculations by vectorizing (1) (the second factor Marc mentioned).
> In particular, I think of splitting the input by group, i.e. I could submit all relevant variables split by group
>
> Ngrp1, Ngrp2, Ngrp3, ygrp1, ygrp2, ygrp3, xgrp1, xgrp2, xgrp,
>
> which would allow to calculate the means as e.g.
>
> mugrp1 <- beta[1] * xgrp1[1]
>
> instead of looping through all N and do mu[n] <- beta[g[n]] * x[n]

This can help with speed. Anything that lets you call vector-based
log densities with parameters that are mixed vectors and non-vectors
will usually let you reuse computations.

> (c)
> Speed up by vectorizing (2):
> Inspired by Marco (Inacio’s) description of a zero-altered NB regression I thought of doing something along these line:
> First increment_log_probs gobally as if there were no truncation:
>
> y ~ neg_binomial(alpha, beta)
>
> Then correct for truncation by incrementing again
>
> increment_log_prob(na*neg_binomial_cdf_log(SC-1, alpha,beta))
> increment_log_prob(nb*neg_binomial_ccdf_log(SC, alpha,beta))
>
> whereby SC is the truncation point and na are the number of people with above the truncation point and nb the number of people below.*
>
> Somehow I fear that this is a naïve approach, but I think it should be possible to calculate the cdf and ccdf, or actually the entire experession “na*neg_binomial_cdf_log(SC-1, alpha,beta)”, only once. But maybe this is already happening under the hood?

This kind of thing can provide a big speedup. We don't currently reuse
the computation because we don't have a vector-based form of the truncated
densities, which is what we'd need to do it under the hood.

I didn't follow your algebra, but in general, you can do this. If you
N cases of the same density with truncation below at 2, then you just need

increment_log_prob(-N * neg_binomial_ccdf(2, alpha, beta));

If the parameters alpha and beta vary by individual, say with a regression,
then this won't work.

There's a discussion in the manual of what's inserted into the log
probability for the various truncations (left/lower, right/upper, both).

- Bob

Guido Biele

unread,
Dec 5, 2014, 4:16:07 PM12/5/14
to stan-...@googlegroups.com

thanks Bob,
in had updated my response to reflect what to do when parameters vary by individual. I see you replied to my original responses, but not my updated response (I'm can update response directly on the google groups web page, these updates are apparently not distributed via email).
I'll try out if the approach I describe there can be implemented/improves speed and let you know.
guido

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/QYBfP4n3N44/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

Guido Biele

unread,
Dec 15, 2014, 3:39:42 AM12/15/14
to stan-...@googlegroups.com
Hello,

I am posting the best I could come up with to vector as much as possible.

A brief reminder of what I am trying to model:
- I want to fit a hierarchical (zero altered) negative binomial regression model
- The groups I model consist of 2 groups with many individuals above a threshold and few individuals below a threshold, as well as a third group with the opposite distribution (see attached figure)
- To model these "funny" outcome distribution, I split outcomes (y) into those smaller than the threshold (y_a) and those equal to or larger than the threshold (y_b) and then model y_a and y_b with truncated distributions.

The complete model is below.
When I ran this model with on my full data set (N = 30000 for group 1 and 1500 for groups 2 and 3, respectively), it took around 88 hours to complete 375 iterations (i.e. around 15 minutes per iteration). A Gradient evaluation takes 0.56s.
So before I actually start such an analysis, it would be great if anybody could have a look at the model below and suggest improvements.

Best Guido

################## MODEL ############################
// Hierachical Zero altered negative binomial regression model for truncated data
data {
  // to allow better vectorization
  // data is split into .._a for right-truncated
  // and .._b for left_truncated outcomes
  int<lower=1> N_a;         // number of data points
  int<lower=0> y_a[N_a];    // observations
  int<lower=0> z_a[N_a];    // indicator: observation is not zero
  int g_a[N_a];             // group for individual
  int<lower=1> N_b;         // number of data points
  int<lower=0> y_b[N_b];    // observations
  int<lower=0> z_b[N_b];    // indicator: observation is not zero
  int g_b[N_b];             // group for individual
  int SC;    // truncation / cut off point
  int<lower=1> K;           // number parameters
  int<lower=1> L;           // number group predictors
  matrix[N_a,K] x_a;        // individual predictors
  matrix[N_b,K] x_b;        // individual predictors
  int M;                    // number of groups
  matrix[M,L] u;            // group predictors
}

parameters {
  // covariances for regression model parameters for mean of negative binomial
  // (from stan language reference)
  matrix[K,M] z;
  cholesky_factor_corr[K] L_Omega; 
  vector<lower=0>[K] tau;           // prior scale
  matrix[L,K] gamma;                // group coeffs
  
  // parameters for regression model for p(score = 0) (zero alteration)
  // no hierarchical estimation here because some groups have only a few zeros
  vector[K] alpha;                // group coeffs  
  
  // variance of negative binomial
  // I know from previous analysis that it should be around 10
  real<lower=0,upper=15> phi;
}

transformed parameters {
  // regression weights for mean of negative binomial model
  matrix[M,K] beta; 
  beta <- u * gamma + (diag_pre_multiply(tau,L_Omega) * z)';
}


model {
  // mean on negative binomial
  vector[N_a] eta_a;
  vector[N_b] eta_b;
  // zero alteration
  vector[N_a] psi_a;
  vector[N_b] psi_b;

  / sample parameters
  tau ~ cauchy(0,2.5); # orig scale = 2.5
  to_vector(z) ~ normal(0,1);
  L_Omega ~ lkj_corr_cholesky(2);
  to_vector(gamma) ~ normal(0,5); # orig scale = 5
  phi ~ cauchy(0,1);  
  alpha ~ normal(0,5); # orig scale = 5 
  

  // prepare and partially update for y_a < SC
  for (n in 1:N_a){
    eta_a[n] <- exp(dot_product(x_a[n],beta[g_a[n]]));
    psi_a[n] <- dot_product(x_a[n],alpha);
    // because this is a zero altered negative binomial
    // and y_a < SC, I model y_a as being truncated
    // at the right- and left-hand side
    increment_log_prob(-log_diff_exp(neg_binomial_2_cdf_log(SC-1,exp(eta_a[n]),phi),
                                     neg_binomial_2_cdf_log(0,exp(eta_a[n]),phi)));}
  
  // prepare for y_b >= SC
  for (n in 1:N_b){
    eta_b[n] <- exp(dot_product(x_b[n],beta[g_b[n]]));
    psi_b[n] <- dot_product(x_b[n],alpha);}
  
  // increment zero alteration
  z_a ~ bernoulli_logit(psi_a);
  z_b ~ bernoulli_logit(psi_b);
  
  // increment negative binomial model
  increment_log_prob(neg_binomial_2_log_log(y_a,eta_a,phi));
  increment_log_prob(neg_binomial_2_log_log(y_b,eta_b,phi));

  // increment for left hand side truncation of y_b
  increment_log_prob(-neg_binomial_2_ccdf_log(SC-1, exp(eta_b),phi));
y_desities.png

Bob Carpenter

unread,
Dec 15, 2014, 1:10:19 PM12/15/14
to stan-...@googlegroups.com
If you know phi should be around 10, then this is probably
not the right prior:

phi ~ cauchy(0,1);

You should be able to use something that doesn't put a hard
bound on phi (the negative binomial overdispersion).

You could make beta local variable in model and avoid writing the output.

I think the combination of this

> eta_b[n] <- exp(dot_product(x_b[n],beta[g_b[n]]));

and this

> increment_log_prob(neg_binomial_2_log_log(y_b,eta_b,phi));

may be a mistake, because it amounts to:

y_b ~ neg_binomial_2(y_b, exp(eta_b), phi);

which is equivalent to:

y_b ~ neg_binomial_2(y_b, exp(exp(dot_product(x_b[n],beta[g_b[n]]))), phi);

Our doc for negative binomial 2 should really be broken out to indicate
that neg_binomial_2_log is yet another parameterization.

You could simplify this a bit

> for (n in 1:N_b){
> eta_b[n] <- exp(dot_product(x_b[n],beta[g_b[n]]));
> psi_b[n] <- dot_product(x_b[n],alpha);}
>
> // increment zero alteration
> z_a ~ bernoulli_logit(psi_a);
> z_b ~ bernoulli_logit(psi_b);

by just writing this:

z_b ~ bernoulli_logit(x_b * alpha);

If you really want the loop, it's better to use an array of vectors
than a matrix if you're pulling out rows because there's no copying.

- Bob
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> <y_desities.png>

Guido Biele

unread,
Dec 17, 2014, 5:21:25 AM12/17/14
to stan-...@googlegroups.com
Thanks for the pointers Bob!

I think the doc for neg_binomial_2 is ok. The root of my error was that the eta-parameterization is available for neg_binomial_2_log in the form of neg_binomial_2_log_log, but there is no such thing as 
neg_binomial_2_cdf_log_log, so that I went back and forth about where to do the exponentiation and in the end screwed it up.
maybe a more intuitive naming scheme would be neg_binomial_.. (for the alpha, beta parameterization) , neg_binomial_mu_... (for the mu, phi parameterization) and neg_binomial_eta_... (for the eta, phi parameterization)?

When going through the code again it appeared to me that I incorrectly also incremented the log-probability for zero values in y_a and y_b with the likelihood from the negative_binomial.
I fixed this in the updated model by including an additional outcome-vector y0, which is zero or 1 for non-zero outcomes.

The updated model (attached) is still slow, but I know believe that this is mostly due to the large amount of data.

Cheers - Guido
HZAxNB2RegressionV.stan
Reply all
Reply to author
Forward
0 new messages