Feedback on paper about rstanarm's informative prior on R2

[Jonah Sol Gabry]

The stan_lm function in rstanarm implements a linear model with regularizing priors driven by prior beliefs about the R^2. This has been confusing to many users and pretty much everyone except for Ben (who had the idea for this cool prior). In light of the confusion, I've written a rough draft of a paper about the prior and I would very much appreciate some feedback. The latest version is attached as a pdf. You can also find the pdf as well as the raw .tex file and R code for the graphics on GitHub in my R2prior repository. 

The paper is pretty short (9 pages or so) so it should be a quick read.

Thanks, 

Jonah

[Andrew Gelman]

Hi, this is a great start!  I’d like to do more regarding the derivation, we can talk more at Stan mtg.  Also, channeling Andrew, here are some quick non-substantive comments:

- Graphs should not go below 0 on y-axis, and fig 1 should not go below 0 on x-axis either.

- On p.8, I would not call it the “frequentist estimator” as any estimator can be evaluated using frequency properties.  Just call it the least-squares estimate.  (Plz also call it least squares, not OLS.)

- Also for good practice on that page please use display() to summarize the fitted model.

A

[Jonah Gabry]

Thanks for the comments. That extra space below 0 is the ggplot default, which is kind of annoying. But it will be easy to fix.

Let's definitely talk more about it this coming week. We should talk with Ben too as he knows more about this prior than anyone.

[Aki Vehtari]

The draft made R^2 prior more clear for me. Here are few quick comments

- I just remembered that one of our papers http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0029115
has a discussion on prior beliefs about the R^2 using similar (but different) approach as the approach by Guan and Stephens (2011). The real data experiment had n=3 895, p=1 051 811.

- Could you describe more about the properties of the implied marginal prior? Now you write that it gets more concentrated, but I'm interested how the tail shape behaves or how would the implied marginal prior on the amount of shrinkage look like (cf. horseshoe prior paper)

- In the prior for R^2, the first parameter is K/2 and the second is eta. Information about the location of R^2 is used to choose eta. How is the amount of prior information controlled (weakly informative vs strong information)? If you think it is not needed, it would helpfull to comment anyway, why there is no another parameter for controlling how strong the prior information is about specific location of R^2.

- In the experiment you write "R^2 should be fairly small". How did you decide 0.2 is good value for fairly small? Why not 0.1 or 0.3? Is the result sensitive to this?

Aki

[Ben Goodrich]

On Monday, March 28, 2016 at 3:27:02 PM UTC-4, Aki Vehtari wrote:

- Could you describe more about the properties of the implied marginal prior? Now you write that it gets more concentrated, but I'm interested how the tail shape behaves or how would the implied marginal prior on the amount of shrinkage look like (cf. horseshoe prior paper)

For an unknown correlation matrix among the K -1 predictors (the paper / vignette redefines this as K predictors), the 1-dimensional PDF of any standardized coefficient is "known" to be


which is not very intuitive. The characteristic function is also known and real (so the odd moments are zero if they exist) so


We also know the characteristic function (but not the density) of the full-conditional distribution, which may provide some intuition


 

- In the prior for R^2, the first parameter is K/2 and the second is eta. Information about the location of R^2 is used to choose eta. How is the amount of prior information controlled (weakly informative vs strong information)? If you think it is not needed, it would helpfull to comment anyway, why there is no another parameter for controlling how strong the prior information is about specific location of R^2.

Yeah, I started out thinking about the traditional case where the number of observations was large relative to a fixed number of predictors, and you get this Beta(K/2, eta) prior on the R^2. But more generally K could be the rank of the design matrix in the case where there are more predictors than observations or K could even be the expected number of non-zero coefficients. We haven't worked out the case yet where there are more predictors than observations.

- In the experiment you write "R^2 should be fairly small". How did you decide 0.2 is good value for fairly small? Why not 0.1 or 0.3? Is the result sensitive to this?

The coefficients are a bit sensitive but mostly because there are something like 25 observations and 12 parameters. I don't think the estimate of the average treatment effect is very sensitive to this. In general, the data are informative about the R^2.

Thanks,
Ben

[Aki Vehtari]

On Monday, March 28, 2016 at 11:03:00 PM UTC+3, Ben Goodrich wrote:
> For an unknown correlation matrix among the K -1 predictors (the paper / vignette redefines this as K predictors), the 1-dimensional PDF of any standardized coefficient is "known" to be

...

> which is not very intuitive.

But useful to mention in the appendix, and you could use this to plot the corresponding prior on amount of shrinkage.

Aki

[Jonah Sol Gabry]

On Mon, Mar 28, 2016 at 4:02 PM, Ben Goodrich <goodri...@gmail.com> wrote:

For an unknown correlation matrix among the K -1 predictors (the paper / vignette redefines this as K predictors), the 1-dimensional PDF of any standardized coefficient is "known" to be


which is not very intuitive.

Is the derivation for this worked out somewhere? 

[Ben Goodrich]

Yeah, I wrote a Mathematica file that did an inverse Fourier transform of the characteristic function and about an hour later it came up with that PDF. Then my Mathematica subscription expired.

Ben
 

[Jonah Gabry]

Can you share the Mathematica script if you have it? I think I can probably run it.