Optimizing - Doubt

[Vasco Conceição]

Hi,

Sorry for the newbie question... (And please correct me if what I am writing below doesn't make much sense...)

I have just started to look into the output of the optimizing function and it seems to give estimates slightly different than the ones that I had previously obtained through maximum likelihood estimation (using MATLAB). Can you please tell me what is the penatly that the LBFGS algorithm imposes? 

In addition... is there any way to perform "optimizing" for maximum likelihood estimation and not penalized maximum likelihood estimation?

Thanks in advance!

[Krzysztof Sakrejda]

The Stan estimates should be ML estimates, not penalized.  The penalized likelihood option is not available yet.  If you want a better answer about why your estimates are different you might want to present your estimates here (preferably from a couple of starting values for each algorithm), the likeliest answer is that the differences are numerical and not practically significant.  For starters the MATLAB and Stan code might use different tolerances.

Krzysztof

[Bob Carpenter]

> On Sep 22, 2015, at 2:43 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> The Stan estimates should be ML estimates, not penalized. The penalized likelihood option is not available yet.

Yes it is! Check out the maximum likelihood chapter of the manual
for examples on how to write models for maximum likelihood and how
to add penalty terms (e.g., L1, L2, elastic net or user-defined).

You'll get penalized maximum likelihood if you add a penalty term.
For instance, in a linear regression, the vectorized form of the
likelihood is this:

y ~ normal(x * beta, sigma);

If you just use that in the model, you get maximum likelihood.

If you want a penalty function on beta, say unit L2 normalized with
shrinkage to zero, you'd add this to the model

increment_log_prob(-0.5 * beta' * beta);

or equivalently,

beta ~ normal(0, 1);

You can similarly penalize sigma, though that's less common in
my limited experience with penalized MLE.

> If you want a better answer about why your estimates are different you might want to present your estimates here (preferably from a couple of starting values for each algorithm), the likeliest answer is that the differences are numerical and not practically significant. For starters the MATLAB and Stan code might use different tolerances.

Exactly --- the differences should be within the tolerances of both
algorithms.

- Bob

[Vasco Conceição]

Thanks for the quick answer!

The error was very small, but bigger than what the tolerance could account for, but I have just noticed that I had indeed a small difference between models.

You were entirely correct! I am getting exactly the same values now.

[Krzysztof Sakrejda]

On Tuesday, September 22, 2015 at 3:03:57 PM UTC-4, Bob Carpenter wrote:

> On Sep 22, 2015, at 2:43 PM, Krzysztof Sakrejda <krzysztof...@gmail.com> wrote:
>
> The Stan estimates should be ML estimates, not penalized.  The penalized likelihood option is not available yet.

Yes it is!  Check out the maximum likelihood chapter of the manual
for examples on how to write models for maximum likelihood and how
to add penalty terms (e.g., L1, L2, elastic net or user-defined).

Wow, cool! I can't believe I missed that! :)