MLE error: Warning: Inverting hessian failed: Maybe I cant use 'matrix' containers?

[mike w]

New to Py and very rusty with MLE.
I am parroting a statsModels example that fits neg-binomial:  http://statsmodels.sourceforge.net/stable/examples/generated/example_gmle.html
with 2 exceptions:  I'm fitting a beta-binomial AND I am passing endog and exog as 'matrix' containers instead of
endog being pandas.core.series.Series and
exog being pandas.core.frame.DataFrame

Q:  I fought tons of containerization and data-type issues to get my call to LL-function to work.  So - is below error driven by yet more data-type problems (should I abandon use of 'matrix' as my input types),
OR is it a genuine econometric-error.  If econometric-error, any clues on the Hessian bit ?

python-3.3.5.amd64\lib\site-packages\statsmodels\base\model.py:375: Warning: Inverting hessian failed, no bse or cov_params available
  warn(warndoc, Warning)

Thank you !

[josef...@gmail.com]

In general it is not recommended to use matrices or to mix matrices with ndarrays because it's difficult to keep track of the differences in behavior.

"Inverting hessian failed," means that the calculated Hessian, second derivative of the loglikelihood, is not positive definite.

This can be either a numerical problem when the calculations are not precise enough or when the optimization hasn't found a minimum, or it can be that the data or model is such that some parameters are not identified, or a bug if it's not fully tested code.

One possibility is to check the optimization, try different starting values and different optimization methods, and see whether it's possible to converge to a solution with positive hessian.

Also check `model.score(results.params)` which should be close to zero (or smaller than 1e-5 ? in absolute value) to see whether the condition for optimization are achieved.

If you post your code, then I can look into it. (in two, three days since I'm travelling again.)

Also if you have some ideas about improving the documentation or examples for this, then this feedback would be helpful to the next users who try this.

Josef

[Nathaniel Smith]

On Fri, Jul 25, 2014 at 6:56 PM, mike w <mikepy...@gmail.com> wrote:
> Q: I fought tons of containerization and data-type issues to get my call to
> LL-function to work. So - is below error driven by yet more data-type
> problems (should I abandon use of 'matrix' as my input types),

I can't speak to the specific problem you're running into, but with my
numpy dev hat on, I strongly recommend that you not use matrix. It
pretty much only exists at all at this point because of the need to
support legacy code that uses it.

-n

--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

[mike w]

I really appreciate the pointers.  Order of attack:

This can be either a numerical problem when the calculations are not precise enough

- I vaguely recall from school, a trick to 'exponentiate' terms within the LL-calculation, to avoid overflow errors - so
I will try this first.

Re dont mix matrix and array types:  I've been with Python 1 wk - and I get the comment.
I'll re-cast/revise all to np-arrays.

[mike w]

Resolved.:
Several causes/issues, worst being that I had LL spanning 2 lines, so LL wasn't 'seeing' the last few factors (nor was I getting any error from this):
    ll = gammaln(yy + alpha) + gammaln(XX - yy + betta) - gammaln(alpha + XX + betta) - gammaln(alpha) - gammaln(betta) + np.log(factorial(XX))
    - np.log(factorial(XX)) - np.log(factorial(XX - yy)) 

Apparently, there is a syntax to continue across lines that I do not know yet.

The problem is still poorly scaled or something, as after I succeed
Optimization terminated successfully.
I still get
 "Warning: Inverting hessian failed, no bse or ..."

Separate bug from the original bug.

Thanks again!

On Friday, July 25, 2014 1:56:22 PM UTC-4, mike w wrote:

[Sturla Molden]

<josef...@gmail.com> wrote:

> "Inverting hessian failed," means that the calculated Hessian, second
> derivative of the loglikelihood, is not positive definite.

> One possibility is to check the optimization, try different starting values
> and different optimization methods, and see whether it's possible to
> converge to a solution with positive hessian.

You should use an optimizer that does not require an exact or positive
definite Hessian. Trust-region optimization and quasi-Newton methods (e.g.
BFGS) are good alternatives to Newton-Raphson. Even good old Nelder-Mead
simplex method might be preferable (yes it is slower, but the fact that is
nearly always succeeds more than makes up for the speed penalty).

Sturla

[josef...@gmail.com]

The optimization finished in this case, but as the error message says, we don't have a positive Hessian to get the standard errors of the parameters. (*)

I think our default is bfgs for GenericLikelihoodModel, sometimes I use Nelder-Mead to get away from bad starting values and use bfgs or newton to finish up and get more precise convergence in the neighborhood of the optimum.

However, in many cases where we saw a Hessian that is not positive definite, either the model or the data were "fishy" (overparameterized, badly scaled, ...).

(*) we can also use the outerproduct of the gradient as an estimate of the covariance matrix of the parameter estimate, which should be positive definite (if the model is "reasonable")

Josef

 

Sturla

[Sturla Molden]

<josef...@gmail.com> wrote:

> The optimization finished in this case, but as the error message says, we
> don't have a positive Hessian to get the standard errors of the parameters.

Hm, I think there are situations where the Fisher Information can still be
inverted, even if the Hessian is singular with the finite machine
precision.

Sturla

[Brent Pedersen]

On Friday, 25 July 2014 17:09:41 UTC-6, mike w wrote:

Resolved.:
Several causes/issues, worst being that I had LL spanning 2 lines, so LL wasn't 'seeing' the last few factors (nor was I getting any error from this):
    ll = gammaln(yy + alpha) + gammaln(XX - yy + betta) - gammaln(alpha + XX + betta) - gammaln(alpha) - gammaln(betta) + np.log(factorial(XX))
    - np.log(factorial(XX)) - np.log(factorial(XX - yy)) 

Apparently, there is a syntax to continue across lines that I do not know yet.

The problem is still poorly scaled or something, as after I succeed
Optimization terminated successfully.
I still get
 "Warning: Inverting hessian failed, no bse or ..."

Separate bug from the original bug.

Thanks again!

Would you mind sharing your beta-binomial code? 

thanks,

-Brent