double exponential

[Sebastian Weber]

Hi!

Stan has the Laplace distribution built-in. I am about to throw this on a very large model and I was wondering how problematic this density is, given that the distribution has a discontinuous derivative at the mean. Does Stan do some tricks to smooth this out or is there another technique which gives me essentially the same, but with some trick to avoid the to-be-expected trouble due to the discontinuity.

Thanks!

Best,
Sebastian

[andre.p...@googlemail.com]

Hi Sebastian, 

here you find the Tricks Stan does: 

https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal/prob/double_exponential_lpdf.hpp

fabs is used, which is correct.

Upps, find this already:

https://groups.google.com/forum/#!topic/stan-users/HMtsTEpOg-w

Well if  you have a large model and using Laplace will get you also more run-time.

You could try to implement this cosh(x*alpha)*x and hope the autodiff will be faster.

I hope to specify my gradients by myself, since the time criticals I know or can figure

out. The rest could be handled by Stan autodiff.

Stan is the best MCMC around, so no complains, but I have to dig myself into this C++

type system again, which is a pain compared to other languages, like OCAML or Haskell

or live with it. 

Andre

[Bob Carpenter]

> On Feb 21, 2017, at 10:16 AM, andre.pfeuffer via Stan users mailing list <stan-...@googlegroups.com> wrote:
>
> Hi Sebastian,
>
> here you find the Tricks Stan does:
> https://github.com/stan-dev/math/blob/develop/stan/math/prim/scal/prob/double_exponential_lpdf.hpp
> fabs is used, which is correct.
> Upps, find this already:
> https://groups.google.com/forum/#!topic/stan-users/HMtsTEpOg-w
>
> Well if you have a large model and using Laplace will get you also more run-time.
> You could try to implement this cosh(x*alpha)*x and hope the autodiff will be faster.
> I hope to specify my gradients by myself, since the time criticals I know or can figure
> out. The rest could be handled by Stan autodiff.

People like Aki have been recommending the horshoe prior(s?) to concentrate mass
around zero and have long-ish tails.

> Stan is the best MCMC around, so no complains, but I have to dig myself into this C++
> type system again, which is a pain compared to other languages, like OCAML or Haskell
> or live with it.

Why did you need to dig into the type system? The function's documented.
The discontinuity problem is only worse than other sharp curves in that it
gets sharper the closer you get to zero, whereas something smooth will eventually
level off.

I love functional languages, but the upsides of C++ outweigh the benefit:

* matrix library Eigen
* math library Boost
* native(-ish) back-end for Python and R
* much larger developer base among statisticians and scientists

We're going through and adding doc and cleaning up a lot of the metaprogramming.
We were learning as we went the first time through, so it's a bit rough in
places.

- Bob

[Ben Goodrich]

On Tuesday, February 21, 2017 at 7:31:40 AM UTC-5, Sebastian Weber wrote:

Stan has the Laplace distribution built-in. I am about to throw this on a very large model and I was wondering how problematic this density is, given that the distribution has a discontinuous derivative at the mean. Does Stan do some tricks to smooth this out or is there another technique which gives me essentially the same, but with some trick to avoid the to-be-expected trouble due to the discontinuity.

The rstanarm package now does Laplace priors as a mixture of normals, in which case there is no fabs().

 

[Sebastian Weber]

Hi!

Hmm.. sounds as if I just give it a shot with the vanilla implementation (I would be surprised if there is something much smarter).

I do want L1 norm instead of the usual L2 which is why I want the Laplace. So I don't think that horshoe is a good idea.

Sebastian

[Sebastian Weber]

Hi Ben!

Thanks! I was looking for that as I remembered this was mentioned somewhere, but I did not find the right "google words".

Would it make sense to use what rstanarm has done also for the double exponential distribution definition in stan::math?

Sebastian

[Ben Goodrich]

On Tuesday, February 21, 2017 at 11:12:23 AM UTC-5, Sebastian Weber wrote:

Thanks! I was looking for that as I remembered this was mentioned somewhere, but I did not find the right "google words".

Would it make sense to use what rstanarm has done also for the double exponential distribution definition in stan::math?

I don't think so. The function in the Stan Math library does what it is supposed to do, but when used in conjunction with NUTS, it can lead to poor MCMC estimates when the leapfrogger leaps over the discontinuity at zero. In a Stan program, you can introduce the scale-mixture parameter in the parameters block, define the thing that is supposed have a Laplace prior in transformed parameters, and then ignore the margins of the output that you do not care about.

Ben

[andre.p...@googlemail.com]

So that means we could do something likes this in Stan? Great!

http://math.stackexchange.com/questions/1175076/how-to-prove-laplace-distribution-is-scale-mixture-of-gaussians

Andre

[andre.p...@googlemail.com]

I love functional languages, but the upsides of C++ outweigh the benefit:
 
* matrix library Eigen
* math library Boost
* native(-ish) back-end for Python and R
* much larger developer base among statisticians and scientists

We're going through and adding doc and cleaning up a lot of the metaprogramming.
We were learning as we went the first time through, so it's a bit rough in
places.

That sounds great. I can only blame myself. I invested a few hours in math library 

in Stan and couldn't figure out very much. I would already have had written

some lines in of code in C++ to extend Stan. On the other hand, not many people do it

either. One math professor said once, math problems are only difficult, if you don't

understand them. Same true for Stan maybe.

Andre

[Bob Carpenter]

Right --- we can't do sampling inside our function definition!

We could, though, define variables with built-in priors:

normal_real<mu, sigma> y;

and have that be equivalent to putting this in the parameters block

normal y;

and

y ~ normal(mu, sigma);

in the model block. Then all of a sudden our models would look
a lot more like other probabilistic programs. Like Michael, I'm
not convinced this kind of thing is a good idea.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Bob Carpenter]

> On Feb 21, 2017, at 11:56 AM, andre.pfeuffer via Stan users mailing list <stan-...@googlegroups.com> wrote:
>
> ... One math professor said once, math problems are only difficult, if you don't

> understand them. Same true for Stan maybe.

The usual CS analogy is unix man pages. They only make sense
if you already know unix and just need to lookup names for
things, argument orders, edge conditions, etc.

And while there's some truth to all this, I have to confess tha
we made our type system more difficult than it needed to be.
Sean and Mitzi are working on different parts of it in different
pull requests to add tests, doc, and to simplify everything.



- Bob