no variation in sigma

[Ross Boylan]

I tried multinomial logistic with clustering, 200 iterations and 1
chain. The results look plausible except that the only value for sigma
is 1.6.

Any thoughts about what is going on? I guess my next step is to put a
proper prior on sigma.

data {
int<lower=0> N; // number of partners
int<lower=2> K; // number of types of partner race
// K is the number of categories in the outcome pEthnic
int<lower=0> J; //number of covariates
int<lower=1,upper=K> pEthnic[N]; // partner race
int<lower=0> NCLUSTER; //number of respondents (1046)
int iCluster[N]; // cluster id
row_vector[J] x[N]; // covariates
}
parameters {
matrix[J, K] beta; // standard covariate coefficients
vector[K] alpha[NCLUSTER]; // cluster effects
real<lower=0> sigma;
}
model {
vector[K] beta_x;
for(ir in 1:J)
beta[ir] ~ normal(0, 5);
for(i in 1:NCLUSTER)
alpha[i] ~ normal(0, sigma);
for(n in 1:N) {
beta_x <- (x[n] * beta)'+alpha[iCluster[n]];
pEthnic[n] ~ categorical_logit(beta_x);
}
}

sigma shows an sd of 0, n_eff=3, Rhat=2.4.

BTW, without the prior on alpha the model took 900 seconds; with the prior it took 90.

[Ross Boylan]

On Fri, 2013-09-13 at 13:55 -0700, Ross Boylan wrote:
> I tried multinomial logistic with clustering, 200 iterations and 1
> chain. The results look plausible except that the only value for
> sigma
> is 1.6.
>
> Any thoughts about what is going on? I guess my next step is to put a
> proper prior on sigma.

I tried cauchy(0, 5). This induced a small amount of variation giving
1.5 with sd of 0.1. The previous estimate was 1.6.

[Bob Carpenter]

On 9/13/13 5:15 PM, Ross Boylan wrote:
> On Fri, 2013-09-13 at 13:55 -0700, Ross Boylan wrote:
>> I tried multinomial logistic with clustering, 200 iterations and 1
>> chain. The results look plausible except that the only value for
>> sigma
>> is 1.6.

Did you print out to more than two decimal places? Our default
print hides variation if it's small.

>> Any thoughts about what is going on? I guess my next step is to put a
>> proper prior on sigma.

> I tried cauchy(0, 5). This induced a small amount of variation giving
> 1.5 with sd of 0.1. The previous estimate was 1.6.

Compared to having no prior at all? I'd expect more variation
without a prior.

But having said all this, I need to go back in and check what's
going on with all the changes I made to categorical between 1.3 and
2.0 --- there's a (good) chance that there's something broken in
the basic distributions.

Sorry I haven't gotten to this sooner --- we'll definitely add lots
more tests and create models that work before pushing this out in
2.0.

- Bob

[Ross Boylan]

On Fri, Sep 13, 2013 at 06:29:08PM -0400, Bob Carpenter wrote:
>
>
> On 9/13/13 5:15 PM, Ross Boylan wrote:
>> On Fri, 2013-09-13 at 13:55 -0700, Ross Boylan wrote:
>>> I tried multinomial logistic with clustering, 200 iterations and 1
>>> chain. The results look plausible except that the only value for
>>> sigma
>>> is 1.6.
>
> Did you print out to more than two decimal places? Our default
> print hides variation if it's small.

I only got one decimal place. I thought from the docs there wasn't any way to control it.
The sd was 0.0.

>
>>> Any thoughts about what is going on? I guess my next step is to put a
>>> proper prior on sigma.
>
>> I tried cauchy(0, 5). This induced a small amount of variation giving
>> 1.5 with sd of 0.1. The previous estimate was 1.6.
>
> Compared to having no prior at all?

Yes. In other words, without the Cauchy sigma seems to take one
value, 1.6. As you note, there might be some small variation.

> I'd expect more variation
> without a prior.
>
> But having said all this, I need to go back in and check what's
> going on with all the changes I made to categorical between 1.3 and
> 2.0 --- there's a (good) chance that there's something broken in
> the basic distributions.
>

OK. Note that in my later report sigma had some variation, but the
estimate looked crude (elevated Rhat and small effective sample
sizes).

When I estimated related models (binary logistic or probit) using
other tehnologies, sigma was often the toughest term to get
convergence on. I gather from the Gelman et al textbook that's pretty
typical.

> Sorry I haven't gotten to this sooner --- we'll definitely add lots
> more tests and create models that work before pushing this out in
> 2.0.
>

I know software is slow.

Ross

[Bob Carpenter]

I'm responding to too many people, so I'm losing track of
their contexts. Digits can be controlled in bin/print
in the GitHub version of Stan. Digits can be controlled
in the usual way in R using digits=5, etc., in the print
statement, e.g., print(stan.fit, digits=5).

Loosening priors can indeed lead to lower R-hat values and
wider posterior intervals.

Which terms are hard to estimate totally depend on the problem.
In some cases, models are only weakly identified, meaning
it takes a lot of samples before the posterior means start to
look right. In linear regressions, sigma is usually dead easy
to fit.

Research software's usually slow because not enough people are working
on it in a directed enough way. We're bogged down because
we're trying to do too many things at once.

- Bob