ADVI... log-lik worse than with NuTS

[Sebastian Weber]

Hi!

I am working with a large/complex model which I would like to address with ADVI due to speed. However, it seems that the variance components are not estimated so well. That is, the random effects are often not moved away from the initial and most importantly the residual error sigma_y is estimated with nuts as 4.3-4.5 while ADVI gives me around 5.3-5.5, i.e. giving much less weight to the data.

I found that doing the automatic eta tuning does not work well for me, i.e. it tends to choose a too large value of 1 while 0.1 seems to be a safe choice and ADVI gives me a nice CONVERGED message.

Any ideas what one could try?

Sorry to not share the model which is just too big. From my view the model has a problem with it's parametrization (correlation of 0.9 of two parameters), but for reasons out of my control I don't want to change this.

Best,

Sebastian

[Alp Kucukelbir]

hi sebastian,

could you expand a bit on what you mean when you say "the variance components"? do you mean the variance components of the model, or the (marginal) variances of the posterior estimates?

i don't have much to go on here, but i might recommend decreasing the `tol_rel_obj` parameter, such that you run ADVI for more iterations before it converges.

another option is to try the fullrank version of ADVI, which is considerably slower (depending on the number of parameters in your model).

a downstream idea is to find a way to specify which parameters should be modeled independently (computationally cheap) and which ones should be modeled together (more accurate).

i hope this helps.

cheers,

alp

[Sebastian Weber]

Hi Alp,

I was referring to random effects, so variance components of the model. To be specific, I have a model about patients (the patients are observed over time, longitudinal data) which are grouped into treatment arms and then I have multiple treatment arms grouped into different studies. Some parameters vary by patients and some by study and by patient (nested!). While the patient specific parameters are well informed by the data, I observe quite some shrinkage towards the population mean for the study specific parameters when I use NuTS; still I think I need those.

Now when I run the model with ADVI, then the patient-level parameters receive a somewhat increased between-subject variability and the between-study parameters don't move away from the initial (both in comparison to the NuTS result). It seems that ADVI can't deal well with random effects which have some shrinkage on them.

I will try your suggestion of playing with the tolerance, sounds reasonable.

Thanks!

Sebastian

[Alp Kucukelbir]

hi sebastian,

thanks for the explanation.

if the model isn't too big, i would also be curious to see how the
"fullrank" version of the algorithm fares.

eager to hear if any of this helps the fit!

cheers,
alp

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Stan users mailing list" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/stan-users/HE87JtBYMeE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Sebastian Weber]

Hi Alp,

hmmm I can try, but I never had good experience with full rank ADVI (way too slow). Moreover, I have ~1000 patients and 15 parameters out of which 2 are patient specific, which leaves me with ~2000 parameters which are mostly glued together by a hierarchy. Is that big or small for ADVI? Or have there been updates to full rank ADVI recently?

On another note, I have essentially solved my performance problems by centering and rescaling to unit scale; quite amazing what the effect of that is, really!

Sebastian

[Alp Kucukelbir]

hi sebastian,

there have not been any updates to the fullrank version of ADVI
recently. i was just curious.

yes! i'm glad to hear that centering and rescaling helped. it can
sometimes be very powerful.

cheers,
alp

[Bob Carpenter]

Same for MCMC --- in all these cases you need to learn
what the scales are, which is hard if you don't already know
what they are. Especially if they vary around the posterior
(because I think all these algorithms only learn static
scalings [and/or rotations]).

- Bob

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Andrew Gelman]

Yes, unit scale is great.
A

> --

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Sebastian Weber]

To clarify: NUTS runs amazingly fast when centering and scaling... as a side note... this is the huge benefit of the Matt trick which automatically unit-scales all those hierarchical parameters.

BTW, I wrote my model in a way like

parameters {

  vector[15] phi_raw;

}

tansformed parameters {

   real parameter1;

   ....

   vector[15] phi;

  phi <- phi_location + phi_scale .* phi_raw;

  parameter1 <- phi[1];

}

model {

  phi ~ mvn_normal_cholesky(prior);

}

The phi_location and phi_scale are my guesses for the scale and location (for most applications the same as the mvn prior, but not in my application).

Wouldn't it make sense to allow this in the Stan language in a principled way? I mean, it would be great to have direct access to the unconstrained space for users.

Sebastian

On Friday, April 22, 2016 at 1:14:48 AM UTC+2, Andrew Gelman wrote:

Yes, unit scale is great.
A

> On Apr 21, 2016, at 4:34 PM, Sebastian Weber
>

[Bob Carpenter]

Do you have a specific proposal? Your code

- Bob

[Bob Carpenter]

Do you have a specific proposal? Your code

- Bob

[Bob Carpenter]

Oops --- Amtrak sent that last one.

What I meant is that the code uses a raw parameter.

Given people's confusions about Jacobians, we've been
very reluctant to literally expose the unconstrained
parameters.

- Bob

[Sebastian Weber]

This is kind of expert, yes; but a raw parameter is used all the time in about any hierarchical model which uses the Matt trick. I mean OK, you end up getting that Jacobian warning and need to know that you don't need to worry about it as the Jacobian from shift and scale is a constant, but that could be explained in the manual, no?

I think it would be very useful, I am using this as a sort of hack to not being able to specify the mass matrix myself - and in fact this unit scaling and centering is better than specifying the mass matrix since this will always be only a wild guess, but Stan's adaptation is there to really find out what the mass matrix should be.

Sebastian