"Matt trick" vs. "non-centered parameterization"

[Bob Carpenter]

I see we have a whole meetup in NY dedicated to the "Matt trick".
As much as I like Matt and think he's brilliant, this "trick"
had a name in the literature that we should be using. And as
Matt pointed out, it was only a Matt trick --- he knows several,
including other variants of the non-centered parameterizations.

I also don't like things called "trick" --- they imply that you
need to trick the software into doing something, which has
seriously negative connotations (for me, at least)

OK, this must be my morning for ranting.

- Bob

[Andrew Gelman]

+1 on avoiding the word “trick.”

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

For my QMSS class, I have taken to calling things like the Matt trick an "implicit prior", which is a prior you put explicitly on a primitive in order to imply the prior you want on a transformation thereof in order to get computational benefits. It should have a name though. It seems as fundamental to HMC as data augmentation (which is a poor name) has been to Gibbs sampling.

[Andrew Gelman]

I like that.  It’s related to the general idea that the parameterization is part of the model.  A point that can be confusing to non-Bayesians because the least squares estimate toes not change under linear transformations, and the MLE does not change under arbitrary one-to-one transformations.

[Jiqiang Guo]

Do we just call it induced prior?

Jiqiang

--

[Bob Carpenter]

I like "induced" better than "implicit", but why not
just call it a "transformed prior"?

Because sampling is based on geometry in all of
Metropolis, Gibbs, and HMC, it seems quite important
everywhere for MCMC. Of course, different transformation
horses for different sampling courses.

- Bob

[Daniel Lee]

Does anyone want me to change the title of the meetup?

I put the 'aka the "Matt trick"' because I assume enough people read Andrew's blog.

[Andrew Gelman]

I prefer “implicit” or possibly “induced” but not “transformed” (i.e., my preference order is the reverse of Bob’s). The trouble with “transformed” is that it sounds too mathematical: you have a prior, then it gets transformed by a change of variables. That’s mathematically correct but it hides the fact that, in many ways, the transformation is part of the prior (this is related to that 2004 paper of mine).

[Bob Carpenter]

Definitely take Andrew's opinion on this over mine! That's
why we call him The Nominator (see sense 5 of
http://www.thefreedictionary.com/nominate which is related
to "nominal", one of Michael's favorite words judging by
frequency ).

- Bob

[Andrew Gelman]

Perhaps this is a strategic concession to get me to stop talking about the Stanbox…

[Bob Carpenter]

Not at all --- on matters of statistical terminology I'm happy
to defer to you. (Well, maybe I'll keep saying "bias" even if
you'd rather I just say non-zero expected error, and I sneak the
odd "MAP" in when you're not listening. I have pretty much
broken myself of the habit of referring to a sample as being made
up of multiple samples --- a sample is a sequence of draws, end of
story.)

And that's not just so you'll defer to me on computational naming,
which is much less of a quagmire and which has a lot more wiggle room
for naming without confusing people. But I'll stand by Stanbox being
confusing to anyone who knows what an Xbox is. Now we could grab
a raspberry pi, give it to Ben, and get a Stanbox Mini back, or we
could do the same by jailbreaking a first generation Apple TV;
even better would be a case mod for a raspberry pi to produce a
StanWallWart.

I think I've been in e-mail too long today.

- Bob

[Jonah]

For what it's worth I like 'implicit', or 'implied' maybe. I think it gives the right intuition (as well as being an appropriate description mathematically).

And if these sorts of computationally motivated transformations are going to remain important (which they probably will) I like the idea of teaching them as just being part of the model. In some sense there's a gap between the theoretical model and the model as programmed, but in many ways they're inseparable in any practical context.

[Bob Carpenter]

The way I think of these, there are two models in play:

1. The posterior we actually care about.

2. The density over which HMC is actually performed.

If there are constrained parameters, Stan already provides
a built-in (implicit?) transform to some (2). For example,
positive-constrained parameters are log transformed (with
Jacobian) to unconstrained ones.

Then we sometimes add additional (explicit?) transforms to ease
computation. For example, applying the non-centered parameterization
to a hierarchical prior.

I take it the implicit or implied prior is the one we place
on (2) given that we want to be able to inverse transform
it back to (1). Isn't that just going to be the original prior
times the absolute Jacobian determinant of the inverse transform?

I'm really not arguing about the terminology here, I just want
to understand it.

- Bob

> On Mar 7, 2015, at 5:29 PM, Jonah <jga...@gmail.com> wrote:
>
> For what it's worth I like 'implicit', or 'implied' maybe. I think it gives the right intuition (as well as being an appropriate description mathematically).
>
> And if these sorts of computationally motivated transformations are going to remain important (which they probably will) I like the idea of teaching them as just being part of the model. In some sense there's a gap between the theoretical model and the model as programmed, but in many ways they're inseparable in any practical context.
>

[Michael Betancourt]

This is basically the correct perspective. Remember, that the posterior
is a probability _measure_, which is a parameterization-invariant, abstract
mathematical object. The Stan Modeling Language defines a
probability _density_, which is a non-unique, parameterization-dependent
function from R^{n} -> R. What this means is that a given model can be
represented different ways in Stan, and different representations have
different computational performances.

There’s nothing implicit or explicit or whatnot going on. It’s just the fundamental
ambiguity of trying to define measures via densities.

[Andrew Gelman]

My point from my 2004 paper (also the talk that Mike saw me give in London) is that, when the parameterization changes, the model can change too, because we tend to use certain natural classes of prior distributions. Thus, it’s not _just_ that we have a fixed distribution that we want to sample from, with reparameterizations being computational aids. In addition, once we reparameterize and add prior information, the model itself typically changes, often in useful ways. That is, the Pinocchio Principle.

[Bob Carpenter]

You guys make my life very easy --- I can just shovel
e-mails into the doc. And it's instructive for me, to
boot! Thanks.

- Bob

[Bob Carpenter]

And I get to cite both the Pinocchio Principle and the
Folk Theorem in a footnote, with a link to the lexicon. :-)

- Bob

[Robert Trangucci]

I hope you all don't mind if I integrate a lot of this into my meetup talk this week (with the appropriate citation, of course). Very useful to me too.

Rob