ANN: statsmodels release candidate 0.8.0rc1

[josef...@gmail.com]

We are very pleased to announce the release candidate for 0.8.0 of statsmodels.

Statsmodels 0.8.0 combines the work of two years of statsmodels
development with many new features given that version 0.7 was never
released. See the release notes below for details.

The source distribution and Windows wheels compiled with Microsoft
compilers are available on pypi.

The documentation for this release candidate is currently at
http://www.statsmodels.org/0.8.0/
The release notes are at
http://www.statsmodels.org/0.8.0/release/version0.8.html and
http://www.statsmodels.org/0.8.0/release/version0.7.html


Please report any problems on the github issues
https://github.com/statsmodels/statsmodels/issues or on the mailing
list http://groups.google.com/group/pystatsmodels .


The release team on behalf of statsmodels developers

--------------------------------

about statsmodels:

Statsmodels is a Python package that provides estimation and inference
for statistical and econometric models and descriptive and inferential
statistics.


Release 0.8.0
=============

See also changes in the unreleased 0.7

Release summary
---------------

The main features of this release are several new time series models based
on the statespace framework, multiple imputation using MICE as well as many
other enhancements. The codebase also has been updated to be compatible with
recent numpy and pandas releases.

Statsmodels is using now github to store the updated documentation which
is available under
http://www.statsmodels.org/stable for the last release, and
http://www.statsmodels.org/dev/ for the development version.

This is the last release that supports Python 2.6.


**Warning**

API stability is not guaranteed for new features, although even in this case
changes will be made in a backwards compatible way if possible. The stability
of a new feature depends on how much time it was already in statsmodels master
and how much usage it has already seen.
If there are specific known problems or limitations, then they are mentioned
in the docstrings.


The following major new features appear in this version.

Statespace Models
-----------------

Building on the statespace framework and models added in 0.7, this release
includes additional models that build on it.
Authored by Chad Fulton largely during GSOC 2015

Kalman Smoother
^^^^^^^^^^^^^^^

The Kalman smoother (introduced in #2434) allows making inference on the
unobserved state vector at each point in time using data from the entire
sample. In addition to this improved inference, the Kalman smoother is required
for future improvements such as simulation smoothing and the expectation
maximization (EM) algorithm.

As a result of this improvement, all state space models now inherit a `smooth`
method for producing results with smoothed state estimates. In addition, the
`fit` method will return results with smoothed estimates at the maximum
likelihood estimates.

Postestimation
^^^^^^^^^^^^^^

Improved post-estimation output is now available to all state space models
(introduced in #2566). This includes the new methods `get_prediction` and
`get_forecast`, providing standard errors and confidence intervals as well
as point estimates, `simulate`, providing simulation of time series following
the given state space process, and `impulse_responses`, allowing computation
of impulse responses due to innovations to the state vector.

Diagnostics
^^^^^^^^^^^

A number of general diagnostic tests on the residuals from state space
estimation are now available to all state space models (introduced in #2431).
These include:

* `test_normality` implements the Jarque-Bera test for normality of residuals
* `test_heteroskedasticity` implements a test for homoskedasticity of
residuals similar to the Goldfeld-Quandt test
* `test_serial_correlation` implements the Ljung-Box (or Box-Pierce) test for
serial correlation of residuals

These test statistics are also now included in the `summary` method output. In
addition, a `plot_diagnostics` method is available which provides four plots
to visually assess model fit.

Unobserved Components
^^^^^^^^^^^^^^^^^^^^^

The class of univariate Unobserved Components models (also known as structural
time series models) are now available (introduced in #2432). This includes as
special cases the local level model and local linear trend model. Generically
it allows decomposing a time series into trend, cycle, seasonal, and
irregular components, optionally with exogenous regressors and / or
autoregressive errors.

Multivariate Models
^^^^^^^^^^^^^^^^^^^

Two standard multivariate econometric models - vector autoregressive
moving-average model with exogenous regressors (VARMAX) and Dynamic Factors
models - are now available (introduced in #2563). The first is a popular
reduced form method of exploring the covariance in several time series, and the
second is a popular reduced form method of extracting a small number of common
factors from a large dataset of observed series.

Recursive least squares
^^^^^^^^^^^^^^^^^^^^^^^

A model for recursive least squares, also known as expanding-window OLS, is
now available in `statsmodels.regression` (introduced in #2830).

Miscellaneous
^^^^^^^^^^^^^

Other improvements to the state space framework include:

* Improved missing data handling #2770, #2809
* Ongoing refactoring and bug fixes in fringes and corner cases


New functionality in statistics
-------------------------------

Contingency Tables #2418 (Kerby Shedden)

Local FDR, multiple testing #2297 (Kerby Shedden)

Mediation Analysis #2352 (Kerby Shedden)

other:

* weighted quantiles in DescrStatsW #2707 (Kerby Shedden)


Duration
--------

Kaplan Meier Survival Function #2614 (Kerby Shedden)

Cumulative incidence rate function #3016 (Kerby Shedden)

other:

* frequency weights in Kaplan-Meier #2992 (Kerby Shedden)


Imputation
----------

new subpackage in `statsmodels.imputation`

MICE #2076 (Frank Cheng GSOC 2014 and Kerby Shedden)

Imputation by regression on Order Statistic #3019 (Paul Hobson)


Time Series Analysis
--------------------

Markov Switching Models
^^^^^^^^^^^^^^^^^^^^^^^

Markov switching dynamic regression and autoregression models are now
available (introduced in #2980 by Chad Fulton). These models allow regression
effects and / or autoregressive dynamics to differ depending on an unobserved
"regime"; in Markov switching models, the regimes are assumed to transition
according to a Markov process.

Statistics
^^^^^^^^^^

* KPSS stationarity, unit root test #2775 (N-Wouda)
* The Brock Dechert Scheinkman (BDS) test for nonlinear dependence is now
available (introduced in #934 by Chad Fulton)


Penalized Estimation
--------------------

Elastic net: fit_regularized with L1/L2 penalization has been added to
OLS, GLM and PHReg (Kerby Shedden)


GLM
---

Tweedie is now available as new family #2872 (Peter Quackenbush, Josef Perktold)

other:

* frequency weights for GLM (currently without full support) #
* more flexible convergence options #2803 (Peter Quackenbush)


Multivariate
------------

new subpackage that currently contains PCA

PCA was added in 0.7 to statsmodels.tools and is now in statsmodels.multivariate


Documentation
-------------

New doc build with latest jupyter and Python 3 compatibility (Tom Augspurger)


Other important improvements
----------------------------

several existing functions have received improvements


* seasonal_decompose: improved periodicity handling #2987 (ssktotoro ?)
* tools add_constant, add_trend: refactoring and pandas compatibility
#2240 (Kevin Sheppard)
* acf, pacf, acovf: option for missing handling #3020 (joesnacks ?)
* acf, pacf plots: allow array of lags #2989 (Kevin Sheppard)
* io SimpleTable (summary): allow names with special characters #3015
(tvanessa ?)
* tsa tools lagmat, lagmat2ds: pandas support #2310 #3042 (Kevin Sheppard)
* CompareMeans: from_data, summary methods #2754 (Valery Tyumen)



Major Bugs fixed
----------------

* see github issues

Backwards incompatible changes and deprecations
-----------------------------------------------

* ???
* predict now returns a pandas Series if the exog argument is a DataFrame
* PCA moved to multivariate compared to 0.7


Development summary and credits
-------------------------------

Besides receiving contributions for new and improved features and for bugfixes,
important contributions to general maintenance came from

* Kevin Sheppard
* Pierre Barbier de Reuille
* Tom Augsburger

and the general maintainer and code reviewer

* Josef Perktold

Additionally, many users contributed by participation in github issues and
providing feedback.

Thanks to all of the contributors for the 0.8 release:

.. note::

* Ashish
* Brendan
* Brendan Condon
* BrianLondon
* Chad Fulton
* Chris Fonnesbeck
* Christoph T. Weidemann
* James Kerns
* Josef Perktold
* Kerby Shedden
* Kevin Sheppard
* Leoyzen
* Matthew Brett
* Niels Wouda
* Paul Hobson
* Pierre Barbier de Reuille
* Pietro Battiston
* Ralf Gommers
* Roman Ring
* Skipper Seabold
* Soren Fuglede Jorgensen
* Thomas Cokelaer
* Tom Augspurger
* ValeryTyumen
* Vanessa
* Yaroslav Halchenko
* joesnacks
* kokes
* matiumerca
* rlan
* ssktotoro
* thequackdaddy
* vegcev

Thanks to all of the contributors for the 0.7 release:

.. note::

* Alex Griffing
* Antony Lee
* Chad Fulton
* Christoph Deil
* Daniel Sullivan
* Hans-Martin von Gaudecker
* Jan Schulz
* Joey Stockermans
* Josef Perktold
* Kerby Shedden
* Kevin Sheppard
* Kiyoto Tamura
* Louis-Philippe Lemieux Perreault
* Padarn Wilson
* Ralf Gommers
* Saket Choudhary
* Skipper Seabold
* Tom Augspurger
* Trent Hauck
* Vincent Arel-Bundock
* chebee7i
* donbeo
* gliptak
* hlin117
* jerry dumblauskas
* jonahwilliams
* kiyoto
* neilsummers
* waynenilsen

These lists of names are automatically generated based on git log, and
may not be
complete.

[Matthew Brett]

Hi,

On Tue, Jun 21, 2016 at 1:21 PM, <josef...@gmail.com> wrote:
> We are very pleased to announce the release candidate for 0.8.0 of statsmodels.
>
> Statsmodels 0.8.0 combines the work of two years of statsmodels
> development with many new features given that version 0.7 was never
> released. See the release notes below for details.
>
> The source distribution and Windows wheels compiled with Microsoft
> compilers are available on pypi.
>
> The documentation for this release candidate is currently at
> http://www.statsmodels.org/0.8.0/
> The release notes are at
> http://www.statsmodels.org/0.8.0/release/version0.8.html and
> http://www.statsmodels.org/0.8.0/release/version0.7.html
>
>
> Please report any problems on the github issues
> https://github.com/statsmodels/statsmodels/issues or on the mailing
> list http://groups.google.com/group/pystatsmodels .

I am just building OSX and manylinux1 wheels - here :

https://travis-ci.org/MacPython/statsmodels-wheels/builds/139376876

I found a precision failure on 32-bit :

https://github.com/statsmodels/statsmodels/issues/3067

Great to see the release on its way,

Cheers,

Matthew

[josef...@gmail.com]

On Wed, Jun 22, 2016 at 3:29 AM, Matthew Brett <matthe...@gmail.com> wrote:
> Hi,
>
> On Tue, Jun 21, 2016 at 1:21 PM, <josef...@gmail.com> wrote:
>> We are very pleased to announce the release candidate for 0.8.0 of statsmodels.
>>
>> Statsmodels 0.8.0 combines the work of two years of statsmodels
>> development with many new features given that version 0.7 was never
>> released. See the release notes below for details.
>>
>> The source distribution and Windows wheels compiled with Microsoft
>> compilers are available on pypi.
>>
>> The documentation for this release candidate is currently at
>> http://www.statsmodels.org/0.8.0/
>> The release notes are at
>> http://www.statsmodels.org/0.8.0/release/version0.8.html and
>> http://www.statsmodels.org/0.8.0/release/version0.7.html
>>
>>
>> Please report any problems on the github issues
>> https://github.com/statsmodels/statsmodels/issues or on the mailing
>> list http://groups.google.com/group/pystatsmodels .
>
> I am just building OSX and manylinux1 wheels - here :
>
> https://travis-ci.org/MacPython/statsmodels-wheels/builds/139376876

Thanks, you still have the permission for upload on pypi

>
> I found a precision failure on 32-bit :
>
> https://github.com/statsmodels/statsmodels/issues/3067

There are some numerical problems on 32-bit in edge cases, AFAICS.
They are intermittent on Windows and showed up every few runs, but no
problems on Ubuntu 32-bit.


Josef

[Matthew Brett]

Hi,

I'm testing on Ubuntu 14.04 32-bit. One difference might also be that
I'm using the numpy OpenBLAS wheels from pypi.

Cheers,

Matthew

[josef...@gmail.com]

I was looking at python-xy nightly
https://code.launchpad.net/~pythonxy/+recipe/statsmodels-daily-current

It was green yesterday, but I didn't check what the recipe is using.

Josef



>
> Cheers,
>
> Matthew

[josef...@gmail.com]

installing with pip from pipy requires the `--pre` flag because recent pip have changed the default.

https://pip.pypa.io/en/stable/reference/pip_install/#pre-release-versions

(I didn't know that pip changed the behavior. Based on a comment on stackoverflow.)

Josef

[Matthew Brett]

Hi,

While I remember - OSX and manylinux wheel build / upload instructions at :

https://github.com/MacPython/statsmodels-wheels

Cheers,

Matthew