Need help with improving speed performance

[UD1989]

Hi everyone

I am runninng the attached codes with rstan , but it's taking a LOT of time. Just running it on a sample of 1000 observations including 5 groups take 3.17 hours (my original daaset contains ~50k obs and has 514 groups, and takes many days to run) . Can someone run it in their systems and tell me if it takes the same amount of time? How can I improve the speed?

Any help would be appreciated!

Thanks and regards
UD

 

[Andrew Gelman]

Hi, here are a few tips:

- Make your Stan program a separate stand-alone .stan file, don’t put it as a character string in your R script.  This won’t speed up your model but it should help your workflow.

- normal(,100) priors on beta don’t look good to me.  Try something more stable such as normal(,1), assuming x and y have been roughly put on unit scale

- change to a ~ normal(0,1); and then in the expression for y_hat, replace “a[streetlist[n]]” by “sigma_a*a[streetlist[n]]”.  This is the so-called non-centered parameterization.

Good luck!

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<Hedon.csv><script.R><sample_sourcecode.R>

[UD1989]

Hi Professor Gelman

Thank you for your reply. I implemented the suggested changes ; unfortunately though theres no improvement in performance.

[Bob Carpenter]

You should replace the long-form arithmetic expression with a dot
product. Otherwise, besides tightening up the priors to be more
informative, I don't see much else to do. You could do some more
vectorization, but it's unlikely to help.

Are your predictors standardized?

You also have a model that's only weakly identified by priors:
because of this term

a[streetlist[n]] + beta[streetlist[n]][1] + ...

You could move to a centered parameterization if you have lots of
data, removing the a[streetlist[n]] term from y_hat sum and
instead putting it into the prior for beta[j].

beta[j] ~ normal(a[j] + u[j] * gamma, 100);

That 100 may be bad news if you don't expect beta to be on that
scale.

- Bob

> On Aug 10, 2015, at 12:59 PM, UD1989 <upasa...@gmail.com> wrote:
>