stan_lasso

[Andrew Gelman]

So, we’ve been having some discussion about lasso in Stan and I wanted to give some perspective.

Lasso is, I would say, three things:
(1) Autotuned hierarchical regression that shrinks most coefficients close to 0.
(2) Regularized regression that shrinks most coefficients all the way to 0.
(3) Bayesian regression with a double-exponential prior (or, equivalently, L1-penalized likelihood).

I was saying we should have stan_lasso because lasso is a useful tool. We should have something just as easy to use, that a user can just grab and run, something named stan_lasso (because the name is part of the game).

Our stan_lasso will be better and worse than standard lasso. It will be better in two ways: (a) full Bayes, will automatically account for uncertainty, (b) flexibility, it can be put into any Stan model, a user can add mesurement-error models, hierarchical structures, mixtures, etc etc etc. It will be worse in two ways: (a) slower, I assume, since Stan’s not optimized for this particular class of problems, (b) full Bayes version doesn’t shrink estimates all the way to zero (we might like that, but for most users this will be a drawback).

One question, as always with these Stan extensions, is how much we want to mimic the existing behavior of lasso and how much do we want to do our own thing? A pure lasso emulator would be nearly pointless—but not completely, as it could be used as a component in larger Stan models.

With that all said, I can imagine different options:

- A stan_lasso function that fits a hierarchical model with double-exponential model and with other models being allowed also. It’s now set up so we can do full Bayes. Once we have GMO, we could have it do something very close to what is done by the official “lasso” function.

- Full Bayes with an option to shrink to 0 during post-processing. I’m not quite sure how we would do this but I don’t think it would be hard to come up with something.

I think it’s ok if our default “lasso” function uses Aki’s preferred prior, if we have some option that pulls point estimates all the way to 0. The point is that (1) the function should usable as a routine tool for regression with lots of predictors, (2) users can get that shrinkage-all-the-way-to-zero if that’s what they want, and (3) the double-exponential prior should be an option.

In any case, I do want a stan_lasso function, I don’t just want to send people to stan_glmer and have them navigate the options.
A

[Ben Goodrich]

On Thursday, March 3, 2016 at 9:13:29 AM UTC-5, Andrew Gelman wrote:

I was saying we should have stan_lasso because lasso is a useful tool.  We should have something just as easy to use, that a user can just grab and run, something named stan_lasso (because the name is part of the game).

I think that gets to a broader question of "Who is the target audience of a high-level interface such as rstanarm (or StataARM, etc.)?"

• Your answer to that question has usually been "people who don't use R or are frustrated by R and just want a simple(r) way to do modeling who may not be overtly Bayesian". From that perspective your suggestions to have R functions like stan_logit or stan_lasso in rstanarm make some sense.
• My answer to that question has always been "experienced R users who are interested in or at least curious about Bayesian inference", which is why for any model fitting function stan_foo(), foo() is an existing R function that experienced R users are already familiar with. From that perspective, there shouldn't be a function named stan_lasso() because there is no lasso() function in R. Trevor Hastie's package called lars has a lars() function that can be called with type = "lasso", which is the default, but there are other penalization schemes. Hastie et al.'s glmnet package has a glmnet() function that takes a family argument and does lasso penalization by default. Conceivably, we could have a stan_glmnet() function that just calls stan_glm() with Laplace priors.
  

So, it is more that I am unconvinced by your answer to this broader question than I am by the specific application of it to lasso. But specifically

• I don't think there are so many machine learning people who are itching to do Bayesian inference
• I think machine learning people tend to prefer Matlab, Python, Julia, etc. over R

If there are a non-negligible number of machine learning people using R who are interested in or at least curious about Bayesian inference, then it is critically important to make sure that crowd understands

1. The penalty functions that they are familiar with are usually kernels of some prior.
   
2. The popular penalty functions are chosen to make finding the mode of the penalized likelihood function an easy optimization problem, but if mode-finding is not the goal, then you are free to choose a penalty / prior that makes more sense.

To reinforce those concepts, I think it makes sense to have the penalty / prior be a function argument that goes with a variety of likelihood functions rather than having the penalty / prior be part of the function name. That is why most of the models in rstanarm have an option for prior = hs() or prior = hs_plus() for hierarchical shrinkage, which works even if your outcome is binary, count, etc. Suppose someone wants to do L1-regularization when the outcome is binary, would you have them call stan_logit_lasso()?

 

With that all said, I can imagine different options:

- A stan_lasso function that fits a hierarchical model with double-exponential model and with other models being allowed also.  It’s now set up so we can do full Bayes.  Once we have GMO, we could have it do something very close to what is done by the official “lasso” function.

- Full Bayes with an option to shrink to 0 during post-processing.  I’m not quite sure how we would do this but I don’t think it would be hard to come up with something.

I think it’s ok if our default “lasso” function uses Aki’s preferred prior, if we have some option that pulls point estimates all the way to 0.  The point is that (1) the function should usable as a routine tool for regression with lots of predictors, (2) users can get that shrinkage-all-the-way-to-zero if that’s what they want, and (3) the double-exponential prior should be an option.

I think the word "lasso" is so associated with "L1-penalization" rather than "regularization" that it would be confusing to reassociate "lasso" with "regularization" generally or "hiearchical shrinkage priors" specifically. I would like to implement the projection method for selecting a subspace of a model, but to me, that would be a separate post-estimation function that works on anything estimated in rstanarm rather than calling stan_lasso(*, shrunk = TRUE).

Ben

[Aki Vehtari]

On Thursday, March 3, 2016 at 6:46:12 PM UTC+2, Ben Goodrich wrote:
> I think the word "lasso" is so associated with "L1-penalization" rather than "regularization" that it would be confusing to reassociate "lasso" with "regularization" generally or "hiearchical shrinkage priors" specifically. I would like to implement the projection method for selecting a subspace of a model, but to me, that would be a separate post-estimation function that works on anything estimated in rstanarm rather than calling stan_lasso(*, shrunk = TRUE).

+1

Aki

[Bob Carpenter]

I think this would be a terrible way to spend our limited
developer resources. And I really hate the idea of trying
to co-opt the stupid name "lasso".

- Bob

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

[Michael Betancourt]

I agree with Bob — why are we trying to leech onto the popularity
of the LASSO when it is fundamentally frequentist? Any appeal
we make to that community can only be accepted with so much
success as we’re ultimately trying to solve different problems. If
you think there are secret (or potential) Bayesians hiding in those
ranks then I don’t think this will be all that helpful, either. It would
be better to communicate the weaknesses of the LASSO and the
benefits of alternatives, rather than trying to bait and switch.

I also agree with Ben that we shouldn’t convolve everything the
LASSO actually does with just “regularization”. If model projection
is desired it’s best to make that a post processing step that can be
done for anything rather than a separate R package for a few
models that will drain waaaaaaaay too much developer time.

[Dustin Tran]

Although I like the identification of the core idea and its clear separation for further improvements (as I generally do with Andrew’s redefinitions), I agree with everyone else in not misusing the term “lasso”. This is the same complaint I had with the EP paper. I also agree with Ben’s summary of target audiences. Most likely the ones this would appeal to would be statisticians and machine learning R users who are curious about Bayesian inference.

But I think that it does make sense to have some standalone function, stan_something(), which by default (note: a minimal # of arguments need to be written) does a similar behavior as glmnet/lasso but “better” and just as fast. Right now, the user has to be pretty familiar with the behavior of various priors to use rstanarm.

> I agree with Bob — why are we trying to leech onto the popularity
> of the LASSO when it is fundamentally frequentist?

I think we can always be purists about things but practically we often need to make compromises about computation, whether it be to excite other communities or just to apply as Bayesians ourselves. That’s part of why we have ADVI, Laplace, and soon GMO for the “efficiency frontier” in inference, and also rstanarm for modeling convenience. For something like rstanarm, people may as well have just gone and defined Stan models but they chose to use R formulas instead; therefore it makes sense to also make it easy to specify sparse shrinkage-inducing type of parameter estimation.

Dustin

[Michael Betancourt]

I’m not going to harp on the importance of quantifying the accuracy of
all of these approximations, and the prematurity of trying to shove
them onto users, again, but that is not the most important point at
issue here which Bob has been consistently making.

The problem is that all of these features cost huge amounts of time
to build, test, and _maintain_. They are not free and they are not
one off costs. We are barely keeping up with what we have, let
alone getting to the long list of features we already want to implement
that have been motivated by our practical work.

This is as much a question of priorities as it is validity.

[Andrew Gelman]

Hi, if it’s too much to maintain, I can just write such functions myself with the help of Jonah, Aki, etc. I think there is a real need for a function that has lasso-like behavior and I’d like it to be written using Stan so that it plays well with larger, more complicated models.

One approach would be to start not with an R function but with a paper with some title such as Lasso-like Functionality Using Stan, that has example code demonstrating everything that I’d like to be doing. The advantage of writing a paper with sample code is (a) it’s a paper, not a program, so it doesn’t have to be maintained, (b) people can take the sample code and adapt to their own problems, so we’re targeting the early adopters and power-users.

A

[Bob Carpenter]

It's not that a single banana is too heavy to carry, it's
that a fruit basket with half a dozen varieties of banana
in bunches, a bushel of corn, a peck of tomatoes, two stone
of potatoes, five of blueberries, and a case of grapes in
wine form is too much to carry for one or two people.

So we either buy a donkey or get choosy about which fruit
we tote around.

- Bob

[Daniel Lee]

At least we'd have a lasso to catch the donkey. I don't know about the fruit basket, though. That's tougher to wrangle with a lasso.

[Andrew Gelman]

Understood. I will start with the paper with example code and see how things go from there. This is the path we took with Waic and LOO and it’s worked ok so far.
A

[Jonah Sol Gabry]

On Thursday, March 3, 2016, Bob Carpenter <ca...@alias-i.com> wrote:

It's not that a single banana is too heavy to carry, it's
that a fruit basket with half a dozen varieties of banana
in bunches, a bushel of corn, a peck of tomatoes, two stone
of potatoes, five of blueberries, and a case of grapes in
wine form is too much to carry for one or two people.

So we either buy a donkey or get choosy about which fruit
we tote around.

- Bob

Nice  

[Michael Betancourt]

If I may reiterate — the overall problem here is that developing ideas and
even algorithms is fast, but designing interfaces for those algorithms,
implementing them, and then testing and maintaining them is much slower.
So assume that every new feature will take 6-12 months, even if it took just
a few days to invent.

Now the problem comes about when ideas are being proposed at every
meeting. We spend one or two weeks trying to figure out how to implement
the old feature and then all of the sudden it becomes deprecated by the
new feature. Those weeks are then wasted, we have to start thinking about
something else, and nothing ever gets done. It’s demotivating and frankly
demoralizing. Software development, the kind that makes packages like
Stan _relatively_ pleasant experiences of users, evolves way more slowly
that the typical NIPS-style publication cycle.

WAIC and LOO were much more successful because in having to write the
paper you had to build and document prototypes that suggested interfaces
that could readily be turned into production code. Also prototypes came
with lots of examples that were used for demonstration in the paper but
could be used at tests in the code.

In summary:

“I thought of this new algorithm, it’s neat. Let me tell you about it so we
can discuss it. I’m going to built a prototype, document a bunch of examples
of it working and maybe failing in a manuscript draft, and then let’s consider
adding it Stan.” is awesome and one of the most fun parts of working on
Stan.

“I thought of this new algorithm idea and haven’t prototyped it at all but it
should be in Stan. Why isn’t it in Stan? I _really_ want this in Stan now
guys. Oh, hey, look at this new thing! Let’s forget that old idea and focus
on this.” is where things becomes demotivating and demoralizing.

[Andrew Gelman]

Mike:
Not to worry. I’ll write the paper first and get help from people individually as I need it. I still think it was good to have the stan_lasso discussion on stan-dev because it was a way to get perspective on this from people I respect and trust.
A